[jira] [Commented] (SPARK-22560) Must create spark session directly to connect to hive

2017-11-22 Thread Ran Mingxuan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262118#comment-16262118 ] Ran Mingxuan commented on SPARK-22560: -- In my opinion, options of spark session shou

[jira] [Commented] (SPARK-22578) CSV with quoted line breaks not correctly parsed

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262119#comment-16262119 ] Hyukjin Kwon commented on SPARK-22578: -- Can you enable {{multiLine}} option? It's di

[jira] [Commented] (SPARK-22579) BlockManager.getRemoteValues and BlockManager.getRemoteBytes should be implemented using streaming

2017-11-22 Thread Eyal Farago (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262120#comment-16262120 ] Eyal Farago commented on SPARK-22579: - CC: [~hvanhovell] (we've discussed this privat

[jira] [Resolved] (SPARK-22560) Must create spark session directly to connect to hive

2017-11-22 Thread Ran Mingxuan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ran Mingxuan resolved SPARK-22560. -- Resolution: Works for Me Fix Version/s: 2.2.0 Target Version/s: 2.2.0 I add on

[jira] [Resolved] (SPARK-22578) CSV with quoted line breaks not correctly parsed

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22578. -- Resolution: Invalid I just double checked with your input: {code} scala> spark.read.option("mu

[jira] [Updated] (SPARK-22574) Wrong request causing Spark Dispatcher going inactive

2017-11-22 Thread German Schiavon Matteo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] German Schiavon Matteo updated SPARK-22574: --- Description: When submitting a wrong _CreateSubmissionRequest_ to Spark Dispa

[jira] [Commented] (SPARK-22516) CSV Read breaks: When "multiLine" = "true", if "comment" option is set as last line's first character

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262189#comment-16262189 ] Hyukjin Kwon commented on SPARK-22516: -- This can be reproduced by: {code} spark.rea

[jira] [Commented] (SPARK-20101) Use OffHeapColumnVector when "spark.sql.columnVector.offheap.enable" is set to "true"

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262190#comment-16262190 ] Apache Spark commented on SPARK-20101: -- User 'kiszk' has created a pull request for

[jira] [Assigned] (SPARK-21168) KafkaRDD should always set kafka clientId.

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21168: Assignee: Apache Spark > KafkaRDD should always set kafka clientId. >

[jira] [Assigned] (SPARK-21168) KafkaRDD should always set kafka clientId.

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21168: Assignee: (was: Apache Spark) > KafkaRDD should always set kafka clientId. > -

[jira] [Commented] (SPARK-22516) CSV Read breaks: When "multiLine" = "true", if "comment" option is set as last line's first character

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262334#comment-16262334 ] Hyukjin Kwon commented on SPARK-22516: -- Seems fixed in 2.5.9. We could probably bump

[jira] [Commented] (SPARK-22575) Making Spark Thrift Server clean up its cache

2017-11-22 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262395#comment-16262395 ] Marco Gaido commented on SPARK-22575: - You can use `UNCACHE TABLE` to remove them fro

[jira] [Created] (SPARK-22580) Count after filtering uncached CSV for isnull(columnNameOfCorruptRecord) always 0

2017-11-22 Thread Florian Kaspar (JIRA)
Florian Kaspar created SPARK-22580: -- Summary: Count after filtering uncached CSV for isnull(columnNameOfCorruptRecord) always 0 Key: SPARK-22580 URL: https://issues.apache.org/jira/browse/SPARK-22580

[jira] [Created] (SPARK-22581) Catalog api does not allow to specify partitioning columns with create(external)table

2017-11-22 Thread Tim Van Wassenhove (JIRA)
Tim Van Wassenhove created SPARK-22581: -- Summary: Catalog api does not allow to specify partitioning columns with create(external)table Key: SPARK-22581 URL: https://issues.apache.org/jira/browse/SPARK-22581

[jira] [Resolved] (SPARK-22572) spark-shell does not re-initialize on :replay

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22572. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19791 [https://git

[jira] [Assigned] (SPARK-22572) spark-shell does not re-initialize on :replay

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22572: Assignee: Mark Petruska > spark-shell does not re-initialize on :replay >

[jira] [Commented] (SPARK-17920) HiveWriterContainer passes null configuration to serde.initialize, causing NullPointerException in AvroSerde when using avro.schema.url

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262483#comment-16262483 ] Apache Spark commented on SPARK-17920: -- User 'vinodkc' has created a pull request fo

[jira] [Commented] (SPARK-19580) Support for avro.schema.url while writing to hive table

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262484#comment-16262484 ] Apache Spark commented on SPARK-19580: -- User 'vinodkc' has created a pull request fo

[jira] [Commented] (SPARK-19878) Add hive configuration when initialize hive serde in InsertIntoHiveTable.scala

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262485#comment-16262485 ] Apache Spark commented on SPARK-19878: -- User 'vinodkc' has created a pull request fo

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-22 Thread mohamed imran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262489#comment-16262489 ] mohamed imran commented on SPARK-22526: --- [~ste...@apache.org] Thanks for your sugge

[jira] [Commented] (SPARK-22581) Catalog api does not allow to specify partitioning columns with create(external)table

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262497#comment-16262497 ] Apache Spark commented on SPARK-22581: -- User 'timvw' has created a pull request for

[jira] [Assigned] (SPARK-22581) Catalog api does not allow to specify partitioning columns with create(external)table

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22581: Assignee: Apache Spark > Catalog api does not allow to specify partitioning columns with

[jira] [Assigned] (SPARK-22581) Catalog api does not allow to specify partitioning columns with create(external)table

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22581: Assignee: (was: Apache Spark) > Catalog api does not allow to specify partitioning col

[jira] [Commented] (SPARK-22516) CSV Read breaks: When "multiLine" = "true", if "comment" option is set as last line's first character

2017-11-22 Thread Sandor Murakozi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262516#comment-16262516 ] Sandor Murakozi commented on SPARK-22516: - I'm a newbie, I would be happy to happ

[jira] [Updated] (SPARK-22560) Must create spark session directly to connect to hive

2017-11-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22560: -- Target Version/s: (was: 2.2.0) Fix Version/s: (was: 2.2.0) > Must create spark session dir

[jira] [Resolved] (SPARK-19417) spark.files.overwrite is ignored

2017-11-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19417. --- Resolution: Won't Fix I think this behavior is on purpose, as these resources are effectively immuta

[jira] [Updated] (SPARK-22570) Cast may create a lot of UTF8String.IntWrapper or UTF8String.longWrapper instances

2017-11-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22570: - Issue Type: Sub-task (was: Bug) Parent: SPARK-22510 > Cast may create a lot of U

[jira] [Updated] (SPARK-22579) BlockManager.getRemoteValues and BlockManager.getRemoteBytes should be implemented using streaming

2017-11-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22579: -- Issue Type: Improvement (was: Bug) You have to read the data either way. I don't know the code but you

[jira] [Commented] (SPARK-22575) Making Spark Thrift Server clean up its cache

2017-11-22 Thread Oz Ben-Ami (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262724#comment-16262724 ] Oz Ben-Ami commented on SPARK-22575: Thanks [~mgaido], we are not caching anything wi

[jira] [Commented] (SPARK-22580) Count after filtering uncached CSV for isnull(columnNameOfCorruptRecord) always 0

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262725#comment-16262725 ] Hyukjin Kwon commented on SPARK-22580: -- There was a limitation and discussion about

[jira] [Resolved] (SPARK-22580) Count after filtering uncached CSV for isnull(columnNameOfCorruptRecord) always 0

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22580. -- Resolution: Duplicate > Count after filtering uncached CSV for isnull(columnNameOfCorruptRecord

[jira] [Commented] (SPARK-22516) CSV Read breaks: When "multiLine" = "true", if "comment" option is set as last line's first character

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262738#comment-16262738 ] Hyukjin Kwon commented on SPARK-22516: -- Sure. Please go ahead. Probably, you could r

[jira] [Resolved] (SPARK-22564) csv reader no longer logs errors

2017-11-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22564. --- Resolution: Won't Fix > csv reader no longer logs errors > > >

[jira] [Commented] (SPARK-22267) Spark SQL incorrectly reads ORC file when column order is different

2017-11-22 Thread Mark Petruska (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262850#comment-16262850 ] Mark Petruska commented on SPARK-22267: --- Hi [~dongjoon], I see that both issues men

[jira] [Created] (SPARK-22582) Spark SQL round throws error with negative precision

2017-11-22 Thread Yuxin Cao (JIRA)
Yuxin Cao created SPARK-22582: - Summary: Spark SQL round throws error with negative precision Key: SPARK-22582 URL: https://issues.apache.org/jira/browse/SPARK-22582 Project: Spark Issue Type: Bu

[jira] [Updated] (SPARK-22570) Create a lot of global variables to reuse an object in generated code

2017-11-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22570: - Summary: Create a lot of global variables to reuse an object in generated code (was: Cre

[jira] [Updated] (SPARK-22570) Create a lot of global variables to reuse an object

2017-11-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22570: - Summary: Create a lot of global variables to reuse an object (was: Cast may create a lot

[jira] [Updated] (SPARK-22570) Create a lot of global variables to reuse an object in generated code

2017-11-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22570: - Description: Generated code for several operations such as {{Cast}}, {{RegExpReplace}}, a

[jira] [Assigned] (SPARK-22570) Create a lot of global variables to reuse an object in generated code

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22570: Assignee: (was: Apache Spark) > Create a lot of global variables to reuse an object in

[jira] [Commented] (SPARK-22570) Create a lot of global variables to reuse an object in generated code

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262922#comment-16262922 ] Apache Spark commented on SPARK-22570: -- User 'kiszk' has created a pull request for

[jira] [Assigned] (SPARK-22570) Create a lot of global variables to reuse an object in generated code

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22570: Assignee: Apache Spark > Create a lot of global variables to reuse an object in generated

[jira] [Assigned] (SPARK-19580) Support for avro.schema.url while writing to hive table

2017-11-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-19580: --- Assignee: Vinod KC > Support for avro.schema.url while writing to hive table > -

[jira] [Resolved] (SPARK-17920) HiveWriterContainer passes null configuration to serde.initialize, causing NullPointerException in AvroSerde when using avro.schema.url

2017-11-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-17920. - Resolution: Fixed Assignee: Vinod KC Fix Version/s: 2.3.0 2.2.1 > Hive

[jira] [Assigned] (SPARK-19878) Add hive configuration when initialize hive serde in InsertIntoHiveTable.scala

2017-11-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-19878: --- Assignee: Vinod KC > Add hive configuration when initialize hive serde in InsertIntoHiveTable.scala

[jira] [Resolved] (SPARK-19878) Add hive configuration when initialize hive serde in InsertIntoHiveTable.scala

2017-11-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19878. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 > Add hive configuration when init

[jira] [Resolved] (SPARK-19580) Support for avro.schema.url while writing to hive table

2017-11-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19580. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 > Support for avro.schema.url whil

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-22 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263021#comment-16263021 ] Steve Loughran commented on SPARK-22526: If the input stream doesn't get closed,

[jira] [Resolved] (SPARK-22543) fix java 64kb compile error for deeply nested expressions

2017-11-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22543. - Resolution: Fixed Fix Version/s: 2.3.0 > fix java 64kb compile error for deeply nested expressions

[jira] [Created] (SPARK-22583) First delegation token renewal time is not 75% of renewal time in Mesos

2017-11-22 Thread Kalvin Chau (JIRA)
Kalvin Chau created SPARK-22583: --- Summary: First delegation token renewal time is not 75% of renewal time in Mesos Key: SPARK-22583 URL: https://issues.apache.org/jira/browse/SPARK-22583 Project: Spark

[jira] [Commented] (SPARK-22583) First delegation token renewal time is not 75% of renewal time in Mesos

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263068#comment-16263068 ] Apache Spark commented on SPARK-22583: -- User 'kalvinnchau' has created a pull reques

[jira] [Assigned] (SPARK-22583) First delegation token renewal time is not 75% of renewal time in Mesos

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22583: Assignee: Apache Spark > First delegation token renewal time is not 75% of renewal time in

[jira] [Assigned] (SPARK-22583) First delegation token renewal time is not 75% of renewal time in Mesos

2017-11-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22583: Assignee: (was: Apache Spark) > First delegation token renewal time is not 75% of rene

[jira] [Updated] (SPARK-22395) Fix the behavior of timestamp values for Pandas to respect session timezone

2017-11-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22395: Labels: release-notes (was: ) > Fix the behavior of timestamp values for Pandas to respect session timezon

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-22 Thread mohamed imran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263097#comment-16263097 ] mohamed imran commented on SPARK-22526: --- [~ste...@apache.org] Yes. Thats sounds sen

[jira] [Commented] (SPARK-22578) CSV with quoted line breaks not correctly parsed

2017-11-22 Thread Carlos Barahona (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263192#comment-16263192 ] Carlos Barahona commented on SPARK-22578: - I didn't realize this was multiLine wa

[jira] [Comment Edited] (SPARK-22578) CSV with quoted line breaks not correctly parsed

2017-11-22 Thread Carlos Barahona (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263192#comment-16263192 ] Carlos Barahona edited comment on SPARK-22578 at 11/22/17 7:20 PM:

[jira] [Created] (SPARK-22584) dataframe write partitionBy out of disk/java heap issues

2017-11-22 Thread Derek M Miller (JIRA)
Derek M Miller created SPARK-22584: -- Summary: dataframe write partitionBy out of disk/java heap issues Key: SPARK-22584 URL: https://issues.apache.org/jira/browse/SPARK-22584 Project: Spark

[jira] [Updated] (SPARK-22584) dataframe write partitionBy out of disk/java heap issues

2017-11-22 Thread Derek M Miller (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek M Miller updated SPARK-22584: --- Description: I have been seeing some issues with partitionBy for the dataframe writer. I cur

[jira] [Created] (SPARK-22585) Url encoding of jar path expected?

2017-11-22 Thread Jakub Dubovsky (JIRA)
Jakub Dubovsky created SPARK-22585: -- Summary: Url encoding of jar path expected? Key: SPARK-22585 URL: https://issues.apache.org/jira/browse/SPARK-22585 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-22586) Feature selection

2017-11-22 Thread Jorge Gonzalez Lopez (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jorge Gonzalez Lopez updated SPARK-22586: - Description: Hello everyone, I would like to know if there are plans to add dif

[jira] [Created] (SPARK-22586) Feature selection

2017-11-22 Thread Jorge Gonzalez Lopez (JIRA)
Jorge Gonzalez Lopez created SPARK-22586: Summary: Feature selection Key: SPARK-22586 URL: https://issues.apache.org/jira/browse/SPARK-22586 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-22586) Feature selection

2017-11-22 Thread Jorge Gonzalez Lopez (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jorge Gonzalez Lopez updated SPARK-22586: - Environment: (was: Hello everyone, I would like to know if there are plans t

[jira] [Resolved] (SPARK-22586) Feature selection

2017-11-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22586. --- Resolution: Invalid This should be a question for the mailing list. > Feature selection > -

[jira] [Resolved] (SPARK-22584) dataframe write partitionBy out of disk/java heap issues

2017-11-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22584. --- Resolution: Not A Problem This doesn't sound like a bug. Running out of memory is 'normal' in that yo

[jira] [Commented] (SPARK-22586) Feature selection

2017-11-22 Thread Jorge Gonzalez Lopez (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263498#comment-16263498 ] Jorge Gonzalez Lopez commented on SPARK-22586: -- Sorry, for posting on the wr

[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?

2017-11-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263512#comment-16263512 ] Sean Owen commented on SPARK-22585: --- Does the real file name contain "%3A443" or ":443"

[jira] [Commented] (SPARK-22267) Spark SQL incorrectly reads ORC file when column order is different

2017-11-22 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263548#comment-16263548 ] Dongjoon Hyun commented on SPARK-22267: --- [~mpetruska]. If you have a patch, please

[jira] [Resolved] (SPARK-21866) SPIP: Image support in Spark

2017-11-22 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-21866. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19439 [h

[jira] [Assigned] (SPARK-21866) SPIP: Image support in Spark

2017-11-22 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-21866: - Assignee: Ilya Matiach > SPIP: Image support in Spark >

[jira] [Commented] (SPARK-21866) SPIP: Image support in Spark

2017-11-22 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263582#comment-16263582 ] Joseph K. Bradley commented on SPARK-21866: --- As far as I know, it shouldn't be

[jira] [Commented] (SPARK-21866) SPIP: Image support in Spark

2017-11-22 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263589#comment-16263589 ] Reynold Xin commented on SPARK-21866: - Why not just declare an image function that lo

[jira] [Commented] (SPARK-22505) toDF() / createDataFrame() type inference doesn't work as expected

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263684#comment-16263684 ] Hyukjin Kwon commented on SPARK-22505: -- Ah, BTW for RDD one, Spark has {{def csv(csv

[jira] [Commented] (SPARK-22165) Type conflicts between dates, timestamps and date in partition column

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263686#comment-16263686 ] Hyukjin Kwon commented on SPARK-22165: -- [~cloud_fan], BTW, should we maybe leave a r

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263690#comment-16263690 ] Hyukjin Kwon commented on SPARK-22240: -- Sure, sounds good ! > S3 CSV number of part

[jira] [Commented] (SPARK-22551) Fix 64kb compile error for common expression types

2017-11-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263696#comment-16263696 ] Liang-Chi Hsieh commented on SPARK-22551: - After SPARK-22543 is merged, I can't r

[jira] [Resolved] (SPARK-22551) Fix 64kb compile error for common expression types

2017-11-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh resolved SPARK-22551. - Resolution: Not A Problem > Fix 64kb compile error for common expression types >

[jira] [Resolved] (SPARK-22552) Cannot Union multiple kafka streams

2017-11-22 Thread sachin malhotra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sachin malhotra resolved SPARK-22552. - Resolution: Not A Problem > Cannot Union multiple kafka streams > ---

[jira] [Commented] (SPARK-22553) Drop FROM in nonReserved

2017-11-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263712#comment-16263712 ] Xiao Li commented on SPARK-22553: - We need a COMPATIBILITY config like https://www.ibm.c

[jira] [Reopened] (SPARK-22560) Must create spark session directly to connect to hive

2017-11-22 Thread Ran Mingxuan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ran Mingxuan reopened SPARK-22560: -- My method not working. Need support. > Must create spark session directly to connect to hive > ---

[jira] [Commented] (SPARK-22505) toDF() / createDataFrame() type inference doesn't work as expected

2017-11-22 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263733#comment-16263733 ] Ruslan Dautkhanov commented on SPARK-22505: --- that's great. thank you [~hyukjin.

[jira] [Commented] (SPARK-22573) SQL Planner is including unnecessary columns in the projection

2017-11-22 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263745#comment-16263745 ] Yuming Wang commented on SPARK-22573: - It caused by https://github.com/apache/spark/p

[jira] [Updated] (SPARK-22495) Fix setup of SPARK_HOME variable on Windows

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-22495: - Fix Version/s: 2.3.0 > Fix setup of SPARK_HOME variable on Windows >

[jira] [Resolved] (SPARK-22495) Fix setup of SPARK_HOME variable on Windows

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22495. -- Resolution: Fixed Fixed in https://github.com/apache/spark/pull/19370 and needs a manual backp

[jira] [Created] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

2017-11-22 Thread Prabhu Joseph (JIRA)
Prabhu Joseph created SPARK-22587: - Summary: Spark job fails if fs.defaultFS and application jar are different url Key: SPARK-22587 URL: https://issues.apache.org/jira/browse/SPARK-22587 Project: Spar

[jira] [Commented] (SPARK-10848) Applied JSON Schema Works for json RDD but not when loading json file

2017-11-22 Thread Amit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263867#comment-16263867 ] Amit commented on SPARK-10848: -- This issue is still persistent in Spark 2.1.0. I tried below

[jira] [Comment Edited] (SPARK-10848) Applied JSON Schema Works for json RDD but not when loading json file

2017-11-22 Thread Amit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263867#comment-16263867 ] Amit edited comment on SPARK-10848 at 11/23/17 6:21 AM: This issu

[jira] [Comment Edited] (SPARK-10848) Applied JSON Schema Works for json RDD but not when loading json file

2017-11-22 Thread Amit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263867#comment-16263867 ] Amit edited comment on SPARK-10848 at 11/23/17 6:22 AM: This issu

[jira] [Created] (SPARK-22588) SPARK: Load Data from Dataframe or RDD to DynamoDB / dealing with null values

2017-11-22 Thread Saanvi Sharma (JIRA)
Saanvi Sharma created SPARK-22588: - Summary: SPARK: Load Data from Dataframe or RDD to DynamoDB / dealing with null values Key: SPARK-22588 URL: https://issues.apache.org/jira/browse/SPARK-22588 Proje

[jira] [Updated] (SPARK-22588) SPARK: Load Data from Dataframe or RDD to DynamoDB / dealing with null values

2017-11-22 Thread Saanvi Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saanvi Sharma updated SPARK-22588: -- Description: I am using spark 2.1 on EMR and i have a dataframe like this: ClientNum | Value

[jira] [Updated] (SPARK-22588) SPARK: Load Data from Dataframe or RDD to DynamoDB / dealing with null values

2017-11-22 Thread Saanvi Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saanvi Sharma updated SPARK-22588: -- Description: I am using spark 2.1 on EMR and i have a dataframe like this: ClientNum | Value

[jira] [Updated] (SPARK-22588) SPARK: Load Data from Dataframe or RDD to DynamoDB / dealing with null values

2017-11-22 Thread Saanvi Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saanvi Sharma updated SPARK-22588: -- Description: I am using spark 2.1 on EMR and i have a dataframe like this: ClientNum | Value

[jira] [Updated] (SPARK-22431) Creating Permanent view with illegal type

2017-11-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22431: Target Version/s: 2.3.0 > Creating Permanent view with illegal type > -