[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23206: --- Attachment: (was: ExecutorTab2.png) > Additional Memory Tuning Metrics >

[jira] [Comment Edited] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357844#comment-16357844 ] Lantao Jin edited comment on SPARK-23206 at 2/9/18 7:56 AM: Yes, we have done

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23206: --- Attachment: ExecutorsTab2.png > Additional Memory Tuning Metrics >

[jira] [Commented] (SPARK-23333) SparkML VectorAssembler.transform slow when needing to invoke .first() on sorted DataFrame

2018-02-08 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358048#comment-16358048 ] Liang-Chi Hsieh commented on SPARK-2: - Currently I think we don't have API in Dataset to just

[jira] [Created] (SPARK-23369) HiveClientSuites fails with unresolved dependency

2018-02-08 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-23369: --- Summary: HiveClientSuites fails with unresolved dependency Key: SPARK-23369 URL: https://issues.apache.org/jira/browse/SPARK-23369 Project: Spark Issue Type:

[jira] [Commented] (SPARK-11222) Add style checker rules to validate doc tests aren't included in docs

2018-02-08 Thread Rekha Joshi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358002#comment-16358002 ] Rekha Joshi commented on SPARK-11222: - For doctest style raised issue -

[jira] [Created] (SPARK-23368) OutputOrdering and OutputPartitioning in ProjectExec should reflect the projected columns

2018-02-08 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-23368: --- Summary: OutputOrdering and OutputPartitioning in ProjectExec should reflect the projected columns Key: SPARK-23368 URL: https://issues.apache.org/jira/browse/SPARK-23368

[jira] [Reopened] (SPARK-23363) Fix spark-sql bug or improvement

2018-02-08 Thread guoxiaolongzte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] guoxiaolongzte reopened SPARK-23363: > Fix spark-sql bug or improvement > > > Key:

[jira] [Commented] (SPARK-23354) spark jdbc does not maintain length of data type when I move data from MS sql server to Oracle using spark jdbc

2018-02-08 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357966#comment-16357966 ] Yuming Wang commented on SPARK-23354: - Do you mean custom column type? you can find more details

[jira] [Commented] (SPARK-23364) 'desc table' command in spark-sql add column head display

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357965#comment-16357965 ] Apache Spark commented on SPARK-23364: -- User 'guoxiaolongzte' has created a pull request for this

[jira] [Assigned] (SPARK-23364) 'desc table' command in spark-sql add column head display

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23364: Assignee: (was: Apache Spark) > 'desc table' command in spark-sql add column head

[jira] [Assigned] (SPARK-23364) 'desc table' command in spark-sql add column head display

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23364: Assignee: Apache Spark > 'desc table' command in spark-sql add column head display >

[jira] [Assigned] (SPARK-23367) Include python document style checking

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23367: Assignee: (was: Apache Spark) > Include python document style checking >

[jira] [Commented] (SPARK-23367) Include python document style checking

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357958#comment-16357958 ] Apache Spark commented on SPARK-23367: -- User 'rekhajoshm' has created a pull request for this issue:

[jira] [Updated] (SPARK-23364) 'desc table' command in spark-sql add column head display

2018-02-08 Thread guoxiaolongzte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] guoxiaolongzte updated SPARK-23364: --- Summary: 'desc table' command in spark-sql add column head display (was: desc table add

[jira] [Assigned] (SPARK-23367) Include python document style checking

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23367: Assignee: Apache Spark > Include python document style checking >

[jira] [Reopened] (SPARK-23364) desc table add column head display

2018-02-08 Thread guoxiaolongzte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] guoxiaolongzte reopened SPARK-23364: > desc table add column head display > -- > >

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357953#comment-16357953 ] Imran Rashid commented on SPARK-23206: -- +1 on all the ideas discussed here so far. One thing

[jira] [Created] (SPARK-23367) Include python document style checking

2018-02-08 Thread Rekha Joshi (JIRA)
Rekha Joshi created SPARK-23367: --- Summary: Include python document style checking Key: SPARK-23367 URL: https://issues.apache.org/jira/browse/SPARK-23367 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

2018-02-08 Thread huangtengfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357952#comment-16357952 ] huangtengfei commented on SPARK-23053: -- the following is a repro case, for clarity  {code:java} /**

[jira] [Commented] (SPARK-23235) Add executor Threaddump to api

2018-02-08 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357949#comment-16357949 ] Imran Rashid commented on SPARK-23235: -- [~jerryshao] thanks for pointing me at SPARK-23206, looks

[jira] [Commented] (SPARK-23364) desc table add column head display

2018-02-08 Thread guoxiaolongzte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357946#comment-16357946 ] guoxiaolongzte commented on SPARK-23364: I will PR to solve this matter, thank you. > desc table

[jira] [Commented] (SPARK-19870) Repeatable deadlock on BlockInfoManager and TorrentBroadcast

2018-02-08 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357930#comment-16357930 ] Imran Rashid commented on SPARK-19870: -- [~eyalfa] I see that warning from every task in stage 23. 

[jira] [Resolved] (SPARK-23186) Initialize DriverManager first before loading Drivers

2018-02-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23186. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20359

[jira] [Assigned] (SPARK-23186) Initialize DriverManager first before loading Drivers

2018-02-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23186: --- Assignee: Dongjoon Hyun > Initialize DriverManager first before loading Drivers >

[jira] [Resolved] (SPARK-23364) desc table add column head display

2018-02-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23364. --- Resolution: Invalid It's not clear what you are trying to communicate here. This is a common

[jira] [Resolved] (SPARK-23363) Fix spark-sql bug or improvement

2018-02-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23363. --- Resolution: Invalid > Fix spark-sql bug or improvement > > >

[jira] [Assigned] (SPARK-23366) Improve hot reading path in ReadAheadInputStream

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23366: Assignee: Apache Spark > Improve hot reading path in ReadAheadInputStream >

[jira] [Assigned] (SPARK-23366) Improve hot reading path in ReadAheadInputStream

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23366: Assignee: (was: Apache Spark) > Improve hot reading path in ReadAheadInputStream >

[jira] [Commented] (SPARK-23310) Perf regression introduced by SPARK-21113

2018-02-08 Thread Juliusz Sompolski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357890#comment-16357890 ] Juliusz Sompolski commented on SPARK-23310: --- [~kiszk] I raised SPARK-23366 and submitted

[jira] [Commented] (SPARK-23366) Improve hot reading path in ReadAheadInputStream

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357889#comment-16357889 ] Apache Spark commented on SPARK-23366: -- User 'juliuszsompolski' has created a pull request for this

[jira] [Updated] (SPARK-23364) desc table add column head display

2018-02-08 Thread guoxiaolongzte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] guoxiaolongzte updated SPARK-23364: --- Description: fix before: !2.png! fix after: !1.png! > desc table add column head

[jira] [Updated] (SPARK-23364) desc table add column head display

2018-02-08 Thread guoxiaolongzte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] guoxiaolongzte updated SPARK-23364: --- Attachment: 2.png 1.png > desc table add column head display >

[jira] [Created] (SPARK-23366) Improve hot reading path in ReadAheadInputStream

2018-02-08 Thread Juliusz Sompolski (JIRA)
Juliusz Sompolski created SPARK-23366: - Summary: Improve hot reading path in ReadAheadInputStream Key: SPARK-23366 URL: https://issues.apache.org/jira/browse/SPARK-23366 Project: Spark

[jira] [Updated] (SPARK-23364) desc table add column head display

2018-02-08 Thread guoxiaolongzte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] guoxiaolongzte updated SPARK-23364: --- Priority: Minor (was: Major) > desc table add column head display >

[jira] [Created] (SPARK-23365) DynamicAllocation with failure in straggler task can lead to a hung spark job

2018-02-08 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-23365: Summary: DynamicAllocation with failure in straggler task can lead to a hung spark job Key: SPARK-23365 URL: https://issues.apache.org/jira/browse/SPARK-23365

[jira] [Created] (SPARK-23364) desc table add column head display

2018-02-08 Thread guoxiaolongzte (JIRA)
guoxiaolongzte created SPARK-23364: -- Summary: desc table add column head display Key: SPARK-23364 URL: https://issues.apache.org/jira/browse/SPARK-23364 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-23363) Fix spark-sql bug or improvement

2018-02-08 Thread guoxiaolongzte (JIRA)
guoxiaolongzte created SPARK-23363: -- Summary: Fix spark-sql bug or improvement Key: SPARK-23363 URL: https://issues.apache.org/jira/browse/SPARK-23363 Project: Spark Issue Type: Task

[jira] [Comment Edited] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357846#comment-16357846 ] Edwina Lu edited comment on SPARK-23206 at 2/9/18 2:30 AM: --- [~cltlfcjin],

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357846#comment-16357846 ] Edwina Lu commented on SPARK-23206: --- [~cltlfcjin], thanks for uploading the screenshot – these would be

[jira] [Comment Edited] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357844#comment-16357844 ] Lantao Jin edited comment on SPARK-23206 at 2/9/18 2:28 AM: Yes, we have done

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23206: --- Attachment: ExecutorTab2.png > Additional Memory Tuning Metrics > >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23206: --- Attachment: (was: Screen Shot 2018-02-09 at 10.21.19.png) > Additional Memory Tuning Metrics >

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357844#comment-16357844 ] Lantao Jin commented on SPARK-23206: Yes, we have done the similar things: !Screen Shot 2018-02-09

[jira] [Updated] (SPARK-23362) Migrate Kafka microbatch source to v2

2018-02-08 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23362: -- Summary: Migrate Kafka microbatch source to v2 (was: Migrate Kafka Microbatch source to v2)

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-23206: --- Attachment: Screen Shot 2018-02-09 at 10.21.19.png > Additional Memory Tuning Metrics >

[jira] [Resolved] (SPARK-23013) Migrate MemoryStream to V2

2018-02-08 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-23013. --- Resolution: Duplicate > Migrate MemoryStream to V2 > -- > >

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357841#comment-16357841 ] Edwina Lu commented on SPARK-23206: --- [~jerryshao], thanks for your help and advice. [~cltlfcjin], I'll

[jira] [Updated] (SPARK-23098) Migrate Kafka batch source to v2

2018-02-08 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23098: -- Summary: Migrate Kafka batch source to v2 (was: Migrate kafka batch source) > Migrate Kafka

[jira] [Updated] (SPARK-23097) Migrate text socket source to v2

2018-02-08 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23097: -- Summary: Migrate text socket source to v2 (was: Migrate text socket source) > Migrate text

[jira] [Assigned] (SPARK-23097) Migrate text socket source

2018-02-08 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-23097: - Assignee: Saisai Shao > Migrate text socket source > -- > >

[jira] [Updated] (SPARK-23362) Migrate Kafka Microbatch source to v2

2018-02-08 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23362: -- Issue Type: Sub-task (was: Improvement) Parent: SPARK-22911 > Migrate Kafka

[jira] [Updated] (SPARK-23098) Migrate kafka batch source

2018-02-08 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23098: -- Summary: Migrate kafka batch source (was: Migrate kafka source) > Migrate kafka batch source

[jira] [Assigned] (SPARK-23362) Migrate Kafka Microbatch source to v2

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23362: Assignee: Apache Spark (was: Tathagata Das) > Migrate Kafka Microbatch source to v2 >

[jira] [Commented] (SPARK-23362) Migrate Kafka Microbatch source to v2

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357827#comment-16357827 ] Apache Spark commented on SPARK-23362: -- User 'tdas' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23362) Migrate Kafka Microbatch source to v2

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23362: Assignee: Tathagata Das (was: Apache Spark) > Migrate Kafka Microbatch source to v2 >

[jira] [Created] (SPARK-23362) Migrate Kafka Microbatch source to v2

2018-02-08 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-23362: - Summary: Migrate Kafka Microbatch source to v2 Key: SPARK-23362 URL: https://issues.apache.org/jira/browse/SPARK-23362 Project: Spark Issue Type:

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357824#comment-16357824 ] Saisai Shao commented on SPARK-23206: - [~cltlfcjin] from Ebay also plans to do similar things, I

[jira] [Commented] (SPARK-23285) Allow spark.executor.cores to be fractional

2018-02-08 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357755#comment-16357755 ] Felix Cheung commented on SPARK-23285: -- Aounds reasonable to me > Allow spark.executor.cores to

[jira] [Updated] (SPARK-22156) Word2Vec: incorrect learning rate update equation when numIterations > 1

2018-02-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-22156: -- Summary: Word2Vec: incorrect learning rate update equation when numIterations > 1

[jira] [Commented] (SPARK-23285) Allow spark.executor.cores to be fractional

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357665#comment-16357665 ] Apache Spark commented on SPARK-23285: -- User 'liyinan926' has created a pull request for this issue:

[jira] [Commented] (SPARK-23360) SparkSession.createDataFrame results in correct results with non-Arrow codepath

2018-02-08 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357645#comment-16357645 ] Li Jin commented on SPARK-23360: cc [~bryanc] as well. I tried to fix this but didn't succeed. I don't

[jira] [Commented] (SPARK-23351) checkpoint corruption in long running application

2018-02-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357632#comment-16357632 ] Shixiong Zhu commented on SPARK-23351: -- I believe this should be resolved in

[jira] [Commented] (SPARK-23351) checkpoint corruption in long running application

2018-02-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357629#comment-16357629 ] Shixiong Zhu commented on SPARK-23351: -- What's your file system? HDFS? > checkpoint corruption in

[jira] [Created] (SPARK-23361) Driver restart fails if it happens after 7 days from app submission

2018-02-08 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-23361: -- Summary: Driver restart fails if it happens after 7 days from app submission Key: SPARK-23361 URL: https://issues.apache.org/jira/browse/SPARK-23361 Project:

[jira] [Commented] (SPARK-23360) SparkSession.createDataFrame results in correct results with non-Arrow codepath

2018-02-08 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357573#comment-16357573 ] Li Jin commented on SPARK-23360: Also this works fine in Arrow path. > SparkSession.createDataFrame

[jira] [Commented] (SPARK-23360) SparkSession.createDataFrame results in correct results with non-Arrow codepath

2018-02-08 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357572#comment-16357572 ] Li Jin commented on SPARK-23360: cc [~cloud_fan] [~ueshin] [~hyukjin.kwon] This is a regression from

[jira] [Created] (SPARK-23360) SparkSession.createDataFrame results in correct results with non-Arrow codepath

2018-02-08 Thread Li Jin (JIRA)
Li Jin created SPARK-23360: -- Summary: SparkSession.createDataFrame results in correct results with non-Arrow codepath Key: SPARK-23360 URL: https://issues.apache.org/jira/browse/SPARK-23360 Project: Spark

[jira] [Assigned] (SPARK-23099) Migrate foreach sink

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23099: Assignee: Apache Spark > Migrate foreach sink > > >

[jira] [Assigned] (SPARK-23099) Migrate foreach sink

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23099: Assignee: (was: Apache Spark) > Migrate foreach sink > > >

[jira] [Commented] (SPARK-23099) Migrate foreach sink

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357525#comment-16357525 ] Apache Spark commented on SPARK-23099: -- User 'jose-torres' has created a pull request for this

[jira] [Comment Edited] (SPARK-23285) Allow spark.executor.cores to be fractional

2018-02-08 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357500#comment-16357500 ] Yinan Li edited comment on SPARK-23285 at 2/8/18 8:22 PM: -- Given the complexity

[jira] [Comment Edited] (SPARK-23244) Incorrect handling of default values when deserializing python wrappers of scala transformers

2018-02-08 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357504#comment-16357504 ] Bryan Cutler edited comment on SPARK-23244 at 2/8/18 8:08 PM: -- This is the

[jira] [Commented] (SPARK-23244) Incorrect handling of default values when deserializing python wrappers of scala transformers

2018-02-08 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357504#comment-16357504 ] Bryan Cutler commented on SPARK-23244: -- This is same issue as SPARK-21685 caused by pyspark not

[jira] [Commented] (SPARK-23285) Allow spark.executor.cores to be fractional

2018-02-08 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357500#comment-16357500 ] Yinan Li commented on SPARK-23285: -- Given the complexity and significant impact of the changes proposed

[jira] [Commented] (SPARK-23318) FP-growth: WARN FPGrowth: Input data is not cached

2018-02-08 Thread Arseniy Tashoyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357482#comment-16357482 ] Arseniy Tashoyan commented on SPARK-23318: -- I want. But I'm short on time now. Will do. >

[jira] [Commented] (SPARK-23244) Incorrect handling of default values when deserializing python wrappers of scala transformers

2018-02-08 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357473#comment-16357473 ] Marco Gaido commented on SPARK-23244: - The change is related because your problem is caused by the

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

2018-02-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357441#comment-16357441 ] Kazuaki Ishizaki commented on SPARK-23309: -- When there is a repro, I am happy to investigate the

[jira] [Assigned] (SPARK-23336) Upgrade snappy-java to 1.1.7.1

2018-02-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-23336: - Assignee: Yuming Wang > Upgrade snappy-java to 1.1.7.1 > -- > >

[jira] [Resolved] (SPARK-23336) Upgrade snappy-java to 1.1.7.1

2018-02-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23336. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20510

[jira] [Commented] (SPARK-23271) Parquet output contains only "_SUCCESS" file after empty DataFrame saving

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357399#comment-16357399 ] Apache Spark commented on SPARK-23271: -- User 'dilipbiswal' has created a pull request for this

[jira] [Updated] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2018-02-08 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-21187: - Description: This is to track adding the remaining type support in Arrow Converters. Currently,

[jira] [Updated] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2018-02-08 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-21187: - Description: This is to track adding the remaining type support in Arrow Converters. Currently,

[jira] [Updated] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2018-02-08 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-21187: - Description: This is to track adding the remaining type support in Arrow Converters. Currently,

[jira] [Commented] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

2018-02-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357387#comment-16357387 ] Steve Loughran commented on SPARK-23308: HADOOP-15216 covers S3A handling this failure with

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

2018-02-08 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357363#comment-16357363 ] Sameer Agarwal commented on SPARK-23309: Thanks, I'll then go ahead and downgrade the priority

[jira] [Updated] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

2018-02-08 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sameer Agarwal updated SPARK-23309: --- Priority: Major (was: Blocker) > Spark 2.3 cached query performance 20-30% worse then spark

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

2018-02-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357346#comment-16357346 ] Thomas Graves commented on SPARK-23309: --- sorry I haven't had time to make a query/dataset to

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

2018-02-08 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357344#comment-16357344 ] Sameer Agarwal commented on SPARK-23309: [~tgraves] [~smilegator] [~cloud_fan] – any advice here? 

[jira] [Reopened] (SPARK-23244) Incorrect handling of default values when deserializing python wrappers of scala transformers

2018-02-08 Thread Tomas Nykodym (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomas Nykodym reopened SPARK-23244: --- I might be wrong but I don't think this is a duplicate. There is an overlap between the two in

[jira] [Commented] (SPARK-18844) Add more binary classification metrics to BinaryClassificationMetrics

2018-02-08 Thread Sandeep Kumar Choudhary (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357281#comment-16357281 ] Sandeep Kumar Choudhary commented on SPARK-18844: - I have submitted the patch. It is now

[jira] [Commented] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

2018-02-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357266#comment-16357266 ] Steve Loughran commented on SPARK-23308: bq. Other option would be creating a special exception

[jira] [Commented] (SPARK-23244) Incorrect handling of default values when deserializing python wrappers of scala transformers

2018-02-08 Thread Tomas Nykodym (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357268#comment-16357268 ] Tomas Nykodym commented on SPARK-23244: --- I might be wrong but I don't think this is a duplicate.

[jira] [Assigned] (SPARK-18844) Add more binary classification metrics to BinaryClassificationMetrics

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18844: Assignee: (was: Apache Spark) > Add more binary classification metrics to

[jira] [Assigned] (SPARK-18844) Add more binary classification metrics to BinaryClassificationMetrics

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18844: Assignee: Apache Spark > Add more binary classification metrics to

[jira] [Commented] (SPARK-18844) Add more binary classification metrics to BinaryClassificationMetrics

2018-02-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357256#comment-16357256 ] Apache Spark commented on SPARK-18844: -- User 'sandecho' has created a pull request for this issue:

[jira] [Updated] (SPARK-11334) numRunningTasks can't be less than 0, or it will affect executor allocation

2018-02-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-11334: --- Fix Version/s: 2.3.0 > numRunningTasks can't be less than 0, or it will affect executor

[jira] [Commented] (SPARK-20327) Add CLI support for YARN custom resources, like GPUs

2018-02-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357107#comment-16357107 ] Marcelo Vanzin commented on SPARK-20327: We cannot add a feature that requires Hadoop 3.0. If you

[jira] [Updated] (SPARK-20327) Add CLI support for YARN custom resources, like GPUs

2018-02-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-20327: --- Shepherd: (was: Mark Grover) > Add CLI support for YARN custom resources, like GPUs >

[jira] [Resolved] (SPARK-21860) Improve memory reuse for heap memory in `HeapMemoryAllocator`

2018-02-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21860. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 19077

  1   2   >