[jira] [Commented] (SPARK-22579) BlockManager.getRemoteValues and BlockManager.getRemoteBytes should be implemented using streaming

2017-11-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265021#comment-16265021 ] Saisai Shao commented on SPARK-22579: - I think this issue should have already been fixed by

[jira] [Assigned] (SPARK-22599) Avoid extra reading for cached table

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22599: Assignee: (was: Apache Spark) > Avoid extra reading for cached table >

[jira] [Assigned] (SPARK-22599) Avoid extra reading for cached table

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22599: Assignee: Apache Spark > Avoid extra reading for cached table >

[jira] [Commented] (SPARK-22599) Avoid extra reading for cached table

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265014#comment-16265014 ] Apache Spark commented on SPARK-22599: -- User 'CodingCat' has created a pull request for this issue:

[jira] [Updated] (SPARK-22599) Avoid extra reading for cached table

2017-11-23 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nan Zhu updated SPARK-22599: Description: In the current implementation of Spark, InMemoryTableExec read all data in a cached table,

[jira] [Created] (SPARK-22599) Avoid extra reading for cached table

2017-11-23 Thread Nan Zhu (JIRA)
Nan Zhu created SPARK-22599: --- Summary: Avoid extra reading for cached table Key: SPARK-22599 URL: https://issues.apache.org/jira/browse/SPARK-22599 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-22598) ExecutorAllocationManager hang when executor fail and numExecutorsTarget has not change

2017-11-23 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-22598: - Summary: ExecutorAllocationManager hang when executor fail and numExecutorsTarget has not change Key: SPARK-22598 URL: https://issues.apache.org/jira/browse/SPARK-22598

[jira] [Commented] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

2017-11-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264986#comment-16264986 ] Saisai Shao commented on SPARK-22587: - [~Prabhu Joseph], I think it is because the logic of comparing

[jira] [Commented] (SPARK-17920) HiveWriterContainer passes null configuration to serde.initialize, causing NullPointerException in AvroSerde when using avro.schema.url

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264965#comment-16264965 ] Apache Spark commented on SPARK-17920: -- User 'vinodkc' has created a pull request for this issue:

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-23 Thread mohamed imran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264931#comment-16264931 ] mohamed imran commented on SPARK-22526: --- [~ste...@apache.org] Thanks. But i have referred some of

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

2017-11-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264904#comment-16264904 ] Saisai Shao commented on SPARK-2926: [~XuanYuan], would you please use spark-perf's micro benchmark

[jira] [Reopened] (SPARK-22561) Dynamically update topics list for spark kafka consumer

2017-11-23 Thread Arun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun reopened SPARK-22561: -- Thanks [~c...@koeninger.org] - The SubscribePattern allows you to use a regex to specify topics of interest. Note

[jira] [Assigned] (SPARK-22597) Add spark-sql script for Windows users

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22597: Assignee: Apache Spark > Add spark-sql script for Windows users >

[jira] [Commented] (SPARK-22597) Add spark-sql script for Windows users

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264894#comment-16264894 ] Apache Spark commented on SPARK-22597: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-22597) Add spark-sql script for Windows users

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22597: Assignee: (was: Apache Spark) > Add spark-sql script for Windows users >

[jira] [Created] (SPARK-22597) Add spark-sql script for Windows users

2017-11-23 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-22597: Summary: Add spark-sql script for Windows users Key: SPARK-22597 URL: https://issues.apache.org/jira/browse/SPARK-22597 Project: Spark Issue Type:

[jira] [Commented] (SPARK-18084) write.partitionBy() does not recognize nested columns that select() can access

2017-11-23 Thread Yassine Labidi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264864#comment-16264864 ] Yassine Labidi commented on SPARK-18084: What's the status on this issue ? >

[jira] [Resolved] (SPARK-22592) cleanup filter converting for hive

2017-11-23 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22592. - Resolution: Fixed Fix Version/s: 2.3.0 > cleanup filter converting for hive >

[jira] [Commented] (SPARK-22495) Fix setup of SPARK_HOME variable on Windows

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264727#comment-16264727 ] Apache Spark commented on SPARK-22495: -- User 'jsnowacki' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22595) flaky test: CastSuite.SPARK-22500: cast for struct should not generate codes beyond 64KB

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22595: Assignee: Kazuaki Ishizaki (was: Apache Spark) > flaky test: CastSuite.SPARK-22500: cast

[jira] [Commented] (SPARK-22500) 64KB JVM bytecode limit problem with cast

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264726#comment-16264726 ] Apache Spark commented on SPARK-22500: -- User 'kiszk' has created a pull request for this issue:

[jira] [Commented] (SPARK-22595) flaky test: CastSuite.SPARK-22500: cast for struct should not generate codes beyond 64KB

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264725#comment-16264725 ] Apache Spark commented on SPARK-22595: -- User 'kiszk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22595) flaky test: CastSuite.SPARK-22500: cast for struct should not generate codes beyond 64KB

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22595: Assignee: Apache Spark (was: Kazuaki Ishizaki) > flaky test: CastSuite.SPARK-22500: cast

[jira] [Commented] (SPARK-22595) flaky test: CastSuite.SPARK-22500: cast for struct should not generate codes beyond 64KB

2017-11-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264663#comment-16264663 ] Kazuaki Ishizaki commented on SPARK-22595: -- I see. I will look at this. > flaky test:

[jira] [Resolved] (SPARK-22561) Dynamically update topics list for spark kafka consumer

2017-11-23 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger resolved SPARK-22561. Resolution: Not A Problem > Dynamically update topics list for spark kafka consumer >

[jira] [Commented] (SPARK-22561) Dynamically update topics list for spark kafka consumer

2017-11-23 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264643#comment-16264643 ] Cody Koeninger commented on SPARK-22561: See SubscribePattern

[jira] [Assigned] (SPARK-22573) SQL Planner is including unnecessary columns in the projection

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22573: Assignee: (was: Apache Spark) > SQL Planner is including unnecessary columns in the

[jira] [Commented] (SPARK-22573) SQL Planner is including unnecessary columns in the projection

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264617#comment-16264617 ] Apache Spark commented on SPARK-22573: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22573) SQL Planner is including unnecessary columns in the projection

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22573: Assignee: Apache Spark > SQL Planner is including unnecessary columns in the projection >

[jira] [Assigned] (SPARK-22596) set ctx.currentVars in CodegenSupport.consume

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22596: Assignee: Apache Spark (was: Wenchen Fan) > set ctx.currentVars in

[jira] [Assigned] (SPARK-22596) set ctx.currentVars in CodegenSupport.consume

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22596: Assignee: Wenchen Fan (was: Apache Spark) > set ctx.currentVars in

[jira] [Commented] (SPARK-22596) set ctx.currentVars in CodegenSupport.consume

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264615#comment-16264615 ] Apache Spark commented on SPARK-22596: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Created] (SPARK-22596) set ctx.currentVars in CodegenSupport.consume

2017-11-23 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-22596: --- Summary: set ctx.currentVars in CodegenSupport.consume Key: SPARK-22596 URL: https://issues.apache.org/jira/browse/SPARK-22596 Project: Spark Issue Type:

[jira] [Commented] (SPARK-22594) Handling spark-submit and master version mismatch

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264597#comment-16264597 ] Apache Spark commented on SPARK-22594: -- User 'Jiri-Kremser' has created a pull request for this

[jira] [Assigned] (SPARK-22594) Handling spark-submit and master version mismatch

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22594: Assignee: (was: Apache Spark) > Handling spark-submit and master version mismatch >

[jira] [Assigned] (SPARK-22594) Handling spark-submit and master version mismatch

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22594: Assignee: Apache Spark > Handling spark-submit and master version mismatch >

[jira] [Commented] (SPARK-22594) Handling spark-submit and master version mismatch

2017-11-23 Thread Jiri Kremser (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264596#comment-16264596 ] Jiri Kremser commented on SPARK-22594: -- PR here: https://github.com/apache/spark/pull/19802 it's

[jira] [Commented] (SPARK-22595) flaky test: CastSuite.SPARK-22500: cast for struct should not generate codes beyond 64KB

2017-11-23 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264589#comment-16264589 ] Wenchen Fan commented on SPARK-22595: - cc [~kiszk] can you take a look? We may need to reduce the

[jira] [Created] (SPARK-22595) flaky test: CastSuite.SPARK-22500: cast for struct should not generate codes beyond 64KB

2017-11-23 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-22595: --- Summary: flaky test: CastSuite.SPARK-22500: cast for struct should not generate codes beyond 64KB Key: SPARK-22595 URL: https://issues.apache.org/jira/browse/SPARK-22595

[jira] [Created] (SPARK-22594) Handling spark-submit and master version mismatch

2017-11-23 Thread Jiri Kremser (JIRA)
Jiri Kremser created SPARK-22594: Summary: Handling spark-submit and master version mismatch Key: SPARK-22594 URL: https://issues.apache.org/jira/browse/SPARK-22594 Project: Spark Issue

[jira] [Commented] (SPARK-22582) Spark SQL round throws error with negative precision

2017-11-23 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264582#comment-16264582 ] Marco Gaido commented on SPARK-22582: - I tried to run {code} spark.sql("select round(100.1 , 1) as

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-23 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264496#comment-16264496 ] Steve Loughran commented on SPARK-22526: I'm not giving a permanent fix. It's a bug in your code

[jira] [Comment Edited] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-23 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264095#comment-16264095 ] Steve Loughran edited comment on SPARK-22526 at 11/23/17 3:47 PM: -- # Fix

[jira] [Commented] (SPARK-22584) dataframe write partitionBy out of disk/java heap issues

2017-11-23 Thread Derek M Miller (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264475#comment-16264475 ] Derek M Miller commented on SPARK-22584: So, in this case, the executor and driver were both

[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?

2017-11-23 Thread Jakub Dubovsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264456#comment-16264456 ] Jakub Dubovsky commented on SPARK-22585: I just tried how URI behaves on some examples and

[jira] [Commented] (SPARK-22584) dataframe write partitionBy out of disk/java heap issues

2017-11-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264448#comment-16264448 ] Sean Owen commented on SPARK-22584: --- It depends on too many things: what did you transform the data

[jira] [Commented] (SPARK-22584) dataframe write partitionBy out of disk/java heap issues

2017-11-23 Thread Derek M Miller (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264447#comment-16264447 ] Derek M Miller commented on SPARK-22584: I disagree. I should not be running out of memory for a

[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?

2017-11-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264392#comment-16264392 ] Sean Owen commented on SPARK-22585: --- Yes, I mean escaping the path only; the host and scheme and so on

[jira] [Commented] (SPARK-7721) Generate test coverage report from Python

2017-11-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264349#comment-16264349 ] Hyukjin Kwon commented on SPARK-7721: - I knew the similar way but was't sure if this was the only way

[jira] [Created] (SPARK-22593) submitMissingTask in DagScheduler will call partitions function many times whch may be time consuming

2017-11-23 Thread tomzhu (JIRA)
tomzhu created SPARK-22593: -- Summary: submitMissingTask in DagScheduler will call partitions function many times whch may be time consuming Key: SPARK-22593 URL: https://issues.apache.org/jira/browse/SPARK-22593

[jira] [Commented] (SPARK-22592) cleanup filter converting for hive

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264266#comment-16264266 ] Apache Spark commented on SPARK-22592: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22592) cleanup filter converting for hive

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22592: Assignee: Wenchen Fan (was: Apache Spark) > cleanup filter converting for hive >

[jira] [Assigned] (SPARK-22592) cleanup filter converting for hive

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22592: Assignee: Apache Spark (was: Wenchen Fan) > cleanup filter converting for hive >

[jira] [Created] (SPARK-22592) cleanup filter converting for hive

2017-11-23 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-22592: --- Summary: cleanup filter converting for hive Key: SPARK-22592 URL: https://issues.apache.org/jira/browse/SPARK-22592 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-22591) GenerateOrdering shouldn't change ctx.INPUT_ROW

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22591: Assignee: (was: Apache Spark) > GenerateOrdering shouldn't change ctx.INPUT_ROW >

[jira] [Assigned] (SPARK-22591) GenerateOrdering shouldn't change ctx.INPUT_ROW

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22591: Assignee: Apache Spark > GenerateOrdering shouldn't change ctx.INPUT_ROW >

[jira] [Commented] (SPARK-22591) GenerateOrdering shouldn't change ctx.INPUT_ROW

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264254#comment-16264254 ] Apache Spark commented on SPARK-22591: -- User 'viirya' has created a pull request for this issue:

[jira] [Created] (SPARK-22591) GenerateOrdering shouldn't change ctx.INPUT_ROW

2017-11-23 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-22591: --- Summary: GenerateOrdering shouldn't change ctx.INPUT_ROW Key: SPARK-22591 URL: https://issues.apache.org/jira/browse/SPARK-22591 Project: Spark Issue

[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?

2017-11-23 Thread Jakub Dubovsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264234#comment-16264234 ] Jakub Dubovsky commented on SPARK-22585: I am not sure about adding encoding step into

[jira] [Commented] (SPARK-17920) HiveWriterContainer passes null configuration to serde.initialize, causing NullPointerException in AvroSerde when using avro.schema.url

2017-11-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264223#comment-16264223 ] Apache Spark commented on SPARK-17920: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-23 Thread mohamed imran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264208#comment-16264208 ] mohamed imran commented on SPARK-22526: --- [~ste...@apache.org] Thanks steve. I will modify my code

[jira] [Updated] (SPARK-22590) SparkContext's local properties missing from TaskContext properties

2017-11-23 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajith S updated SPARK-22590: Attachment: TestProps.scala > SparkContext's local properties missing from TaskContext properties >

[jira] [Created] (SPARK-22590) SparkContext's local properties missing from TaskContext properties

2017-11-23 Thread Ajith S (JIRA)
Ajith S created SPARK-22590: --- Summary: SparkContext's local properties missing from TaskContext properties Key: SPARK-22590 URL: https://issues.apache.org/jira/browse/SPARK-22590 Project: Spark

[jira] [Updated] (SPARK-22574) Wrong request causing Spark Dispatcher going inactive

2017-11-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22574: -- Target Version/s: (was: 2.0.0, 2.1.0, 2.2.0) Fix Version/s: (was: 2.2.0)

[jira] [Updated] (SPARK-22589) Subscribe to multiple roles in Mesos

2017-11-23 Thread Fabiano Francesconi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabiano Francesconi updated SPARK-22589: Description: Mesos offers the capability of [subscribing to multiple

[jira] [Resolved] (SPARK-22560) Must create spark session directly to connect to hive

2017-11-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22560. --- Resolution: Not A Problem Your second code snippet is correct. There is no problem here. In the

[jira] [Created] (SPARK-22589) Subscribe to multiple roles in Mesos

2017-11-23 Thread Fabiano Francesconi (JIRA)
Fabiano Francesconi created SPARK-22589: --- Summary: Subscribe to multiple roles in Mesos Key: SPARK-22589 URL: https://issues.apache.org/jira/browse/SPARK-22589 Project: Spark Issue

[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?

2017-11-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264129#comment-16264129 ] Sean Owen commented on SPARK-22585: --- Hm, I think the issue is rather than the path needs to be encoded

[jira] [Commented] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

2017-11-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264118#comment-16264118 ] Sean Owen commented on SPARK-22587: --- Hm, but if the src and dest FS are different, it overwrites

[jira] [Updated] (SPARK-22165) Type conflicts between dates, timestamps and date in partition column

2017-11-23 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-22165: Labels: release-notes (was: ) > Type conflicts between dates, timestamps and date in partition

[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-11-23 Thread Mark Petruska (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264102#comment-16264102 ] Mark Petruska commented on SPARK-22393: --- The problem is also reproducible with non-spark related

[jira] [Resolved] (SPARK-22588) SPARK: Load Data from Dataframe or RDD to DynamoDB / dealing with null values

2017-11-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22588. --- Resolution: Invalid A question should go to the mailing list. Here it's not clear that it's not

[jira] [Commented] (SPARK-22526) Spark hangs while reading binary files from S3

2017-11-23 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264095#comment-16264095 ] Steve Loughran commented on SPARK-22526: # Fix the code you invoke #. wrap the code you invoke

[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?

2017-11-23 Thread Jakub Dubovsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263979#comment-16263979 ] Jakub Dubovsky commented on SPARK-22585: Real file path is exactly the same as one I am passing

[jira] [Commented] (SPARK-7721) Generate test coverage report from Python

2017-11-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263944#comment-16263944 ] Hyukjin Kwon commented on SPARK-7721: - [~joshrosen], ahh, I happened to duplicate the efforts here