[jira] [Comment Edited] (SPARK-21476) RandomForest classification model not using broadcast in transform

2017-07-24 Thread Peng Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099593#comment-16099593 ] Peng Meng edited comment on SPARK-21476 at 7/25/17 6:55 AM: H

[jira] [Commented] (SPARK-21476) RandomForest classification model not using broadcast in transform

2017-07-24 Thread Peng Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099593#comment-16099593 ] Peng Meng commented on SPARK-21476: --- Hi @Suarabh, I am profiling RF transform performan

[jira] [Commented] (SPARK-14887) Generated SpecificUnsafeProjection Exceeds JVM Code Size Limits

2017-07-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099584#comment-16099584 ] Andrew Ash commented on SPARK-14887: [~fang fang chen] have you seen this in the late

[jira] [Updated] (SPARK-21524) ValidatorParamsSuiteHelpers generates wrong temp files

2017-07-24 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21524: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) > ValidatorParamsSuiteHelpers g

[jira] [Resolved] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-24 Thread Devaraj K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K resolved SPARK-15142. --- Resolution: Duplicate > Spark Mesos dispatcher becomes unusable when the Mesos master restarts >

[jira] [Commented] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-24 Thread Devaraj K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099535#comment-16099535 ] Devaraj K commented on SPARK-15142: --- [~skonto] Thanks for showing interest on this. I h

[jira] [Commented] (SPARK-16784) Configurable log4j settings

2017-07-24 Thread HanCheol Cho (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099477#comment-16099477 ] HanCheol Cho commented on SPARK-16784: -- Hi, I used the following options that allo

[jira] [Commented] (SPARK-21517) Fetch local data via block manager cause oom

2017-07-24 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099412#comment-16099412 ] zhoukang commented on SPARK-21517: -- Can any one help verify the patch related to this is

[jira] [Commented] (SPARK-21517) Fetch local data via block manager cause oom

2017-07-24 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099411#comment-16099411 ] zhoukang commented on SPARK-21517: -- Can any one help verify the patch related to this is

[jira] [Commented] (SPARK-21521) History service requires user is in any group

2017-07-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099392#comment-16099392 ] Saisai Shao commented on SPARK-21521: - [~vanzin], I guess so, in the current logics o

[jira] [Commented] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-24 Thread Iurii Antykhovych (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099390#comment-16099390 ] Iurii Antykhovych commented on SPARK-21491: --- Done, could you please re-check.

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

2017-07-24 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099351#comment-16099351 ] Kazuaki Ishizaki commented on SPARK-21501: -- I see. I misunderstood the descripti

[jira] [Commented] (SPARK-21526) Add support to ML LogisticRegression for setting initial model

2017-07-24 Thread John Brock (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099331#comment-16099331 ] John Brock commented on SPARK-21526: Related to SPARK-21386. > Add support to ML Log

[jira] [Created] (SPARK-21526) Add support to ML LogisticRegression for setting initial model

2017-07-24 Thread John Brock (JIRA)
John Brock created SPARK-21526: -- Summary: Add support to ML LogisticRegression for setting initial model Key: SPARK-21526 URL: https://issues.apache.org/jira/browse/SPARK-21526 Project: Spark I

[jira] [Commented] (SPARK-21524) ValidatorParamsSuiteHelpers generates wrong temp files

2017-07-24 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099313#comment-16099313 ] yuhao yang commented on SPARK-21524: https://github.com/apache/spark/pull/18728 > Va

[jira] [Created] (SPARK-21525) ReceiverSupervisorImpl seems to ignore the error code when writing to the WAL

2017-07-24 Thread Mark Grover (JIRA)
Mark Grover created SPARK-21525: --- Summary: ReceiverSupervisorImpl seems to ignore the error code when writing to the WAL Key: SPARK-21525 URL: https://issues.apache.org/jira/browse/SPARK-21525 Project:

[jira] [Commented] (SPARK-21523) Fix bug of strong wolfe linesearch `init` parameter lose effectiveness

2017-07-24 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099229#comment-16099229 ] Joseph K. Bradley commented on SPARK-21523: --- CC [~yanboliang] [~yuhaoyan] [~dbt

[jira] [Updated] (SPARK-21523) Fix bug of strong wolfe linesearch `init` parameter lose effectiveness

2017-07-24 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-21523: -- Description: We need merge this breeze bugfix into spark because it influence a series

[jira] [Created] (SPARK-21524) ValidatorParamsSuiteHelpers generates wrong temp files

2017-07-24 Thread yuhao yang (JIRA)
yuhao yang created SPARK-21524: -- Summary: ValidatorParamsSuiteHelpers generates wrong temp files Key: SPARK-21524 URL: https://issues.apache.org/jira/browse/SPARK-21524 Project: Spark Issue Type

[jira] [Commented] (SPARK-21523) Fix bug of strong wolfe linesearch `init` parameter lose effectiveness

2017-07-24 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099225#comment-16099225 ] Weichen Xu commented on SPARK-21523: I will work on this once the breeze cut a new ve

[jira] [Updated] (SPARK-21523) Fix bug of strong wolfe linesearch `init` parameter lose effectiveness

2017-07-24 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-21523: --- Priority: Minor (was: Major) > Fix bug of strong wolfe linesearch `init` parameter lose effectivenes

[jira] [Created] (SPARK-21523) Fix bug of strong wolfe linesearch `init` parameter lose effectiveness

2017-07-24 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-21523: -- Summary: Fix bug of strong wolfe linesearch `init` parameter lose effectiveness Key: SPARK-21523 URL: https://issues.apache.org/jira/browse/SPARK-21523 Project: Spark

[jira] [Comment Edited] (SPARK-21521) History service requires user is in any group

2017-07-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099114#comment-16099114 ] Marcelo Vanzin edited comment on SPARK-21521 at 7/24/17 8:54 PM: --

[jira] [Commented] (SPARK-21521) History service requires user is in any group

2017-07-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099114#comment-16099114 ] Marcelo Vanzin commented on SPARK-21521: Just do double-correct myself, it should

[jira] [Commented] (SPARK-21521) History service requires user is in any group

2017-07-24 Thread Adrian Bridgett (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099111#comment-16099111 ] Adrian Bridgett commented on SPARK-21521: - Thanks Marcelo - good idea regarding t

[jira] [Commented] (SPARK-21521) History service requires user is in any group

2017-07-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099100#comment-16099100 ] Marcelo Vanzin commented on SPARK-21521: Hmm, if you're using a local FS it shoul

[jira] [Comment Edited] (SPARK-21521) History service requires user is in any group

2017-07-24 Thread Adrian Bridgett (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099090#comment-16099090 ] Adrian Bridgett edited comment on SPARK-21521 at 7/24/17 8:40 PM: -

[jira] [Commented] (SPARK-21521) History service requires user is in any group

2017-07-24 Thread Adrian Bridgett (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099090#comment-16099090 ] Adrian Bridgett commented on SPARK-21521: - Hmm, I didn't check that (it's actuall

[jira] [Commented] (SPARK-21521) History service requires user is in any group

2017-07-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099078#comment-16099078 ] Marcelo Vanzin commented on SPARK-21521: This smells of a configuration issue...

[jira] [Updated] (SPARK-21522) Flaky test: LauncherServerSuite.testStreamFiltering

2017-07-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-21522: --- Summary: Flaky test: LauncherServerSuite.testStreamFiltering (was: Flay test:) > Flaky test

[jira] [Created] (SPARK-21522) Flay test:

2017-07-24 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-21522: -- Summary: Flay test: Key: SPARK-21522 URL: https://issues.apache.org/jira/browse/SPARK-21522 Project: Spark Issue Type: Bug Components: Tests

[jira] [Comment Edited] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-07-24 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099030#comment-16099030 ] Mitesh edited comment on SPARK-20112 at 7/24/17 7:37 PM: - Still s

[jira] [Commented] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-07-24 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099030#comment-16099030 ] Mitesh commented on SPARK-20112: Still seeing this on 2.1.0, attached new err file > SIG

[jira] [Commented] (SPARK-14239) Add load for LDAModel that supports both local and distributedModel

2017-07-24 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098948#comment-16098948 ] yuhao yang commented on SPARK-14239: Close overlooked stale jira. > Add load for LDA

[jira] [Resolved] (SPARK-14239) Add load for LDAModel that supports both local and distributedModel

2017-07-24 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuhao yang resolved SPARK-14239. Resolution: Won't Do > Add load for LDAModel that supports both local and distributedModel > --

[jira] [Commented] (SPARK-12875) Add Weight of Evidence and Information value to Spark.ml as a feature transformer

2017-07-24 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098946#comment-16098946 ] yuhao yang commented on SPARK-12875: Close stale jira. > Add Weight of Evidence and

[jira] [Resolved] (SPARK-12875) Add Weight of Evidence and Information value to Spark.ml as a feature transformer

2017-07-24 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuhao yang resolved SPARK-12875. Resolution: Won't Do > Add Weight of Evidence and Information value to Spark.ml as a feature > tra

[jira] [Comment Edited] (SPARK-14760) Feature transformers should always invoke transformSchema in transform or fit

2017-07-24 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098940#comment-16098940 ] yuhao yang edited comment on SPARK-14760 at 7/24/17 6:23 PM: -

[jira] [Commented] (SPARK-14760) Feature transformers should always invoke transformSchema in transform or fit

2017-07-24 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098940#comment-16098940 ] yuhao yang commented on SPARK-14760: Close it since it's been overlooked for some tim

[jira] [Resolved] (SPARK-13223) Add stratified sampling to ML feature engineering

2017-07-24 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuhao yang resolved SPARK-13223. Resolution: Not A Problem > Add stratified sampling to ML feature engineering > ---

[jira] [Commented] (SPARK-13223) Add stratified sampling to ML feature engineering

2017-07-24 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098933#comment-16098933 ] yuhao yang commented on SPARK-13223: Close it since it's been overlooked for some tim

[jira] [Resolved] (SPARK-21502) --supervise causing frameworkId conflicts in mesos cluster mode

2017-07-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21502. Resolution: Fixed Assignee: Stavros Kontopoulos Fix Version/s: 2.3.0 > --su

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

2017-07-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098907#comment-16098907 ] Thomas Graves commented on SPARK-21501: --- The issue was actually introduced with SPA

[jira] [Updated] (SPARK-21501) Spark shuffle index cache size should be memory based

2017-07-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21501: -- Affects Version/s: (was: 2.0.0) 2.1.0 > Spark shuffle index cache si

[jira] [Commented] (SPARK-20392) Slow performance when calling fit on ML pipeline for dataset with many columns but few rows

2017-07-24 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098612#comment-16098612 ] Maciej BryƄski commented on SPARK-20392: Is it safe to merge it to 2.2 ? I'm trac

[jira] [Closed] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

2017-07-24 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki closed SPARK-21387. Resolution: Cannot Reproduce > org.apache.spark.memory.TaskMemoryManager.allocatePage cause

[jira] [Closed] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

2017-07-24 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki closed SPARK-21387. Resolution: Fixed > org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM > ---

[jira] [Reopened] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

2017-07-24 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki reopened SPARK-21387: -- > org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM > -

[jira] [Commented] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

2017-07-24 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098545#comment-16098545 ] Kazuaki Ishizaki commented on SPARK-21387: -- While I got OOM in my unit test, I h

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-24 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098239#comment-16098239 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/24/17 3:01 PM: -

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-24 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098239#comment-16098239 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/24/17 3:01 PM: -

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-24 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098239#comment-16098239 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/24/17 3:00 PM: -

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

2017-07-24 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098531#comment-16098531 ] Kazuaki Ishizaki commented on SPARK-21501: -- I guess that to use Spark 2.1 or lat

[jira] [Commented] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2017-07-24 Thread Leif Walsh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098524#comment-16098524 ] Leif Walsh commented on SPARK-21187: Also, if you're unfamiliar, {{object}} columns a

[jira] [Resolved] (SPARK-19214) Inconsistencies between DataFrame and Dataset APIs

2017-07-24 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19214. --- Resolution: Won't Fix > Inconsistencies between DataFrame and Dataset APIs >

[jira] [Commented] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2017-07-24 Thread Leif Walsh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098522#comment-16098522 ] Leif Walsh commented on SPARK-21187: [~rxin] [~bryanc], pandas does support array and

[jira] [Commented] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098344#comment-16098344 ] Liang-Chi Hsieh commented on SPARK-21177: - [~hyukjin.kwon] I ran spark-shell and

[jira] [Commented] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098305#comment-16098305 ] Hyukjin Kwon commented on SPARK-21177: -- [~viirya], I assume you did in the way I did

[jira] [Reopened] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-21177: -- I am reopening this as I can reproduce: {code} def printTimeTaken(str: String, f: () => Unit) {

[jira] [Commented] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098262#comment-16098262 ] Hyukjin Kwon commented on SPARK-21177: -- Oh, no. I created a Hive table and then inse

[jira] [Commented] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098255#comment-16098255 ] Prashant Sharma commented on SPARK-21177: - Yes. In fact, hive should not be setup

[jira] [Updated] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-21177: Description: In short, please use the following shell transcript for the reproducer. {cod

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-24 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098239#comment-16098239 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/24/17 11:52 AM:

[jira] [Commented] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-24 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098239#comment-16098239 ] Stavros Kontopoulos commented on SPARK-15142: - I will give it a shot. > Spar

[jira] [Updated] (SPARK-21519) Add an option to the JDBC data source to initialize the environment of the remote database session

2017-07-24 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-21519: Description: This proposes an option to the JDBC datasource, tentatively called " sessionInitState

[jira] [Updated] (SPARK-21519) Add an option to the JDBC data source to initialize the environment of the remote database session

2017-07-24 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-21519: Description: This proposes an option to the JDBC datasource, tentatively called " sessionInitState

[jira] [Commented] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098227#comment-16098227 ] Hyukjin Kwon commented on SPARK-21177: -- Wait ... is the code itself a self-contained

[jira] [Commented] (SPARK-10872) Derby error (XSDB6) when creating new HiveContext after restarting SparkContext

2017-07-24 Thread Mathieu Rossignol (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098199#comment-16098199 ] Mathieu Rossignol commented on SPARK-10872: --- Using Spark 2.1.0, still having th

[jira] [Created] (SPARK-21521) History service requires user is in any group

2017-07-24 Thread Adrian Bridgett (JIRA)
Adrian Bridgett created SPARK-21521: --- Summary: History service requires user is in any group Key: SPARK-21521 URL: https://issues.apache.org/jira/browse/SPARK-21521 Project: Spark Issue Typ

[jira] [Commented] (SPARK-21520) Hivetable scan for all the columns the SQL statement contains the 'rand'

2017-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098190#comment-16098190 ] Apache Spark commented on SPARK-21520: -- User 'heary-cao' has created a pull request

[jira] [Assigned] (SPARK-21520) Hivetable scan for all the columns the SQL statement contains the 'rand'

2017-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21520: Assignee: Apache Spark > Hivetable scan for all the columns the SQL statement contains the

[jira] [Assigned] (SPARK-21520) Hivetable scan for all the columns the SQL statement contains the 'rand'

2017-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21520: Assignee: (was: Apache Spark) > Hivetable scan for all the columns the SQL statement c

[jira] [Updated] (SPARK-21520) Hivetable scan for all the columns the SQL statement contains the 'rand'

2017-07-24 Thread caoxuewen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caoxuewen updated SPARK-21520: -- Description: Currently, when the rand function is present in the SQL statement, hivetable searches all

[jira] [Created] (SPARK-21520) Hivetable scan for all the columns the SQL statement contains the 'rand'

2017-07-24 Thread caoxuewen (JIRA)
caoxuewen created SPARK-21520: - Summary: Hivetable scan for all the columns the SQL statement contains the 'rand' Key: SPARK-21520 URL: https://issues.apache.org/jira/browse/SPARK-21520 Project: Spark

[jira] [Updated] (SPARK-21519) Add an option to the JDBC data source to initialize the environment of the remote database session

2017-07-24 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-21519: Description: This proposes an option to the JDBC datasource, tentatively called " sessionInitState

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2017-07-24 Thread Manoj Mahalingam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098185#comment-16098185 ] Manoj Mahalingam commented on SPARK-18016: -- Is there a workaround we can do when

[jira] [Commented] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098149#comment-16098149 ] Prashant Sharma commented on SPARK-21177: - Well, I have tried it on three differe

[jira] [Commented] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098148#comment-16098148 ] Hyukjin Kwon commented on SPARK-21177: -- Meanwhile let me give another try. > df.sav

[jira] [Comment Edited] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098143#comment-16098143 ] Hyukjin Kwon edited comment on SPARK-21177 at 7/24/17 9:35 AM:

[jira] [Commented] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098143#comment-16098143 ] Hyukjin Kwon commented on SPARK-21177: -- Yea, I did but I could not. Could you descri

[jira] [Updated] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-21177: Description: In short, please use the following shell transcript for the reproducer. {cod

[jira] [Commented] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098139#comment-16098139 ] Prashant Sharma commented on SPARK-21177: - It is pretty easy to reproduce, you ne

[jira] [Updated] (SPARK-21517) Fetch local data via block manager cause oom

2017-07-24 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-21517: - Description: In our production cluster,oom happens when NettyBlockRpcServer receive OpenBlocks message.T

[jira] [Updated] (SPARK-21519) Add an option to the JDBC data source to initialize the environment of the remote database session

2017-07-24 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-21519: Description: This proposes an option to the JDBC datasource, tentatively called " sessionInitState

[jira] [Updated] (SPARK-21519) Add an option to the JDBC data source to initialize the environment of the remote database session

2017-07-24 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-21519: Description: This proposes an option to the JDBC datasource, tentatively called " sessionInitState

[jira] [Updated] (SPARK-21519) Add an option to the JDBC data source to initialize the environment of the remote database session

2017-07-24 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-21519: Description: This proposes an option to the JDBC datasource, tentatively called " sessionInitState

[jira] [Updated] (SPARK-21519) Add an option to the JDBC data source to initialize the environment of the remote database session

2017-07-24 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-21519: Description: This proposes an option to the JDBC datasource, tentatively called " sessionInitState

[jira] [Assigned] (SPARK-21519) Add an option to the JDBC data source to initialize the environment of the remote database session

2017-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21519: Assignee: (was: Apache Spark) > Add an option to the JDBC data source to initialize th

[jira] [Commented] (SPARK-21519) Add an option to the JDBC data source to initialize the environment of the remote database session

2017-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098105#comment-16098105 ] Apache Spark commented on SPARK-21519: -- User 'LucaCanali' has created a pull request

[jira] [Assigned] (SPARK-21519) Add an option to the JDBC data source to initialize the environment of the remote database session

2017-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21519: Assignee: Apache Spark > Add an option to the JDBC data source to initialize the environme

[jira] [Created] (SPARK-21519) Add an option to the JDBC data source to initialize the environment of the remote database session

2017-07-24 Thread Luca Canali (JIRA)
Luca Canali created SPARK-21519: --- Summary: Add an option to the JDBC data source to initialize the environment of the remote database session Key: SPARK-21519 URL: https://issues.apache.org/jira/browse/SPARK-21519

[jira] [Created] (SPARK-21518) Warnings if spark.mesos.task.labels is unset

2017-07-24 Thread Adrian Bridgett (JIRA)
Adrian Bridgett created SPARK-21518: --- Summary: Warnings if spark.mesos.task.labels is unset Key: SPARK-21518 URL: https://issues.apache.org/jira/browse/SPARK-21518 Project: Spark Issue Type

[jira] [Updated] (SPARK-21517) Fetch local data via block manager cause oom

2017-07-24 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-21517: - Description: In our production cluster,oom happens when NettyBlockRpcServer receive OpenBlocks message.T

[jira] [Updated] (SPARK-21517) Fetch local data via block manager cause oom

2017-07-24 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-21517: - Description: In our production cluster,oom happens when NettyBlockRpcServer receive OpenBlocks message.T

[jira] [Assigned] (SPARK-21517) Fetch local data via block manager cause oom

2017-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21517: Assignee: Apache Spark > Fetch local data via block manager cause oom > --

[jira] [Assigned] (SPARK-21517) Fetch local data via block manager cause oom

2017-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21517: Assignee: (was: Apache Spark) > Fetch local data via block manager cause oom > ---

[jira] [Commented] (SPARK-21517) Fetch local data via block manager cause oom

2017-07-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098074#comment-16098074 ] Apache Spark commented on SPARK-21517: -- User 'caneGuy' has created a pull request fo

[jira] [Created] (SPARK-21517) Fetch local data via block manager cause oom

2017-07-24 Thread zhoukang (JIRA)
zhoukang created SPARK-21517: Summary: Fetch local data via block manager cause oom Key: SPARK-21517 URL: https://issues.apache.org/jira/browse/SPARK-21517 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-24 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098060#comment-16098060 ] Sean Owen commented on SPARK-21491: --- Looks reasonable. I'd say just add a very brief no

[jira] [Commented] (SPARK-21508) Documentation on 'Spark Streaming Custom Receivers' has error in example code

2017-07-24 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098057#comment-16098057 ] Sean Owen commented on SPARK-21508: --- You don't need it assigned, not until you resolve

  1   2   >