[jira] [Created] (SPARK-20981) Add --repositories equivalent configuration for Spark

2017-06-04 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-20981: --- Summary: Add --repositories equivalent configuration for Spark Key: SPARK-20981 URL: https://issues.apache.org/jira/browse/SPARK-20981 Project: Spark Issue

[jira] [Commented] (SPARK-20943) Correct BypassMergeSortShuffleWriter's comment

2017-06-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034296#comment-16034296 ] Saisai Shao commented on SPARK-20943: - I think the original purpose of comment i

[jira] [Commented] (SPARK-20943) Correct BypassMergeSortShuffleWriter's comment

2017-06-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033016#comment-16033016 ] Saisai Shao commented on SPARK-20943: - I don't think the previous c

Re: [VOTE] Livy to enter Apache Incubator

2017-05-31 Thread Saisai Shao
+1 (non-binding) - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org

[jira] [Commented] (SPARK-20898) spark.blacklist.killBlacklistedExecutors doesn't work in YARN

2017-05-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030710#comment-16030710 ] Saisai Shao commented on SPARK-20898: - [~tgraves], I addressed this issue in h

[jira] [Commented] (SPARK-19569) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

2017-05-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017066#comment-16017066 ] Saisai Shao commented on SPARK-19569: - [~ouyangxc.zte] In you above code

[jira] [Commented] (SPARK-20772) Add support for query parameters in redirects on Yarn

2017-05-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013431#comment-16013431 ] Saisai Shao commented on SPARK-20772: - I'm guessing if it is an

Re: Review Request 59305: Fix Livy service check and alerts script with SSL enabled

2017-05-16 Thread Saisai Shao
ly, visit: https://reviews.apache.org/r/59305/#review175106 ------- On May 16, 2017, 5:58 a.m., Saisai Shao wrote: > > --- > This is an automatically generated e-mail. To re

[jira] [Assigned] (AMBARI-21012) Livy service check fails with wire encryption setup

2017-05-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-21012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned AMBARI-21012: Assignee: Saisai Shao > Livy service check fails with wire encryption se

[jira] [Commented] (AMBARI-21012) Livy service check fails with wire encryption setup

2017-05-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/AMBARI-21012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011798#comment-16011798 ] Saisai Shao commented on AMBARI-21012: -- [~sumitmohanty], please help to re

Review Request 59305: Fix Livy service check and alerts script with SSL enabled

2017-05-15 Thread Saisai Shao
services/SPARK2/2.0.0/package/scripts/service_check.py 8e7a766 Diff: https://reviews.apache.org/r/59305/diff/1/ Testing --- Local verification is done. Thanks, Saisai Shao

Re: How about the fetch the shuffle data in one same machine?

2017-05-10 Thread Saisai Shao
There is a JIRA about this thing ( https://issues.apache.org/jira/browse/SPARK-6521). In the current Spark shuffle fetch still leverages Netty even two executors are on the same node, but according to the test on the JIRA, the performance is close whether to bypass network or not. From my understan

[jira] [Commented] (SPARK-20658) spark.yarn.am.attemptFailuresValidityInterval doesn't seem to have an effect

2017-05-08 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001917#comment-16001917 ] Saisai Shao commented on SPARK-20658: - It is mainly depends on YARN to measure

[jira] [Created] (SPARK-20605) Deprecate not used AM and executor port configuration

2017-05-04 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-20605: --- Summary: Deprecate not used AM and executor port configuration Key: SPARK-20605 URL: https://issues.apache.org/jira/browse/SPARK-20605 Project: Spark Issue

Re: Kerberos impersonation of a Spark Context at runtime

2017-05-04 Thread Saisai Shao
Current Spark doesn't support impersonate different users at run-time. Current Spark's proxy user is application level, which means when setting through --proxy-user the whole application will be running with that user. On Thu, May 4, 2017 at 5:13 PM, matd wrote: > Hi folks, > > I have a Spark a

[jira] [Commented] (SPARK-20421) Mark JobProgressListener (and related classes) as deprecated

2017-05-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996301#comment-15996301 ] Saisai Shao commented on SPARK-20421: - [~vanzin], what will you d

[jira] [Created] (SPARK-20517) Download link in history server UI is not correct

2017-04-27 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-20517: --- Summary: Download link in history server UI is not correct Key: SPARK-20517 URL: https://issues.apache.org/jira/browse/SPARK-20517 Project: Spark Issue Type

[jira] [Updated] (SPARK-20517) Download link in history server UI is not correct

2017-04-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20517: Description: The download link in history server UI is concatenated with: {code} Download

[jira] [Commented] (SPARK-19688) Spark on Yarn Credentials File set to different application directory

2017-04-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986546#comment-15986546 ] Saisai Shao commented on SPARK-19688: - Does this issue exist in the latest ma

Re: Off heap memory settings and Tungsten

2017-04-24 Thread Saisai Shao
AFAIK, I don't think the off-heap memory settings is enabled automatically, there're two configurations control the tungsten off-heap memory usage: 1. spark.memory.offHeap.enabled. 2. spark.memory.offHeap.size. On Sat, Apr 22, 2017 at 7:44 PM, geoHeil wrote: > Hi, > I wonder when to enable sp

[jira] [Commented] (SPARK-20426) OneForOneStreamManager occupies too much memory.

2017-04-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978415#comment-15978415 ] Saisai Shao commented on SPARK-20426: - Currently I don't have a clear fix a

[jira] [Commented] (SPARK-20426) OneForOneStreamManager occupies too much memory.

2017-04-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978411#comment-15978411 ] Saisai Shao commented on SPARK-20426: - My simple doubt is that your large clu

[jira] [Commented] (SPARK-20426) OneForOneStreamManager occupies too much memory.

2017-04-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978391#comment-15978391 ] Saisai Shao commented on SPARK-20426: - Is your yarn cluster a large cluster,

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

2017-04-20 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976319#comment-15976319 ] Saisai Shao commented on SPARK-20391: - I'm in favor of using new REST API

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

2017-04-20 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976298#comment-15976298 ] Saisai Shao commented on SPARK-20391: - bq. I assume managed memory her

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

2017-04-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976145#comment-15976145 ] Saisai Shao commented on SPARK-20391: - Thanks [~irashid] [~tgraves] for

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

2017-04-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974685#comment-15974685 ] Saisai Shao commented on SPARK-20391: - [~irashid], would be grateful to hear

[jira] [Created] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

2017-04-19 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-20391: --- Summary: Properly rename the memory related fields in ExecutorSummary REST API Key: SPARK-20391 URL: https://issues.apache.org/jira/browse/SPARK-20391 Project: Spark

[jira] [Updated] (SPARK-20365) Not so accurate classpath format for AM and Containers

2017-04-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20365: Summary: Not so accurate classpath format for AM and Containers (was: Inaccurate classpath format

[jira] [Created] (SPARK-20365) Inaccurate classpath format for AM and Containers

2017-04-18 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-20365: --- Summary: Inaccurate classpath format for AM and Containers Key: SPARK-20365 URL: https://issues.apache.org/jira/browse/SPARK-20365 Project: Spark Issue Type

Re: Spark API authentication

2017-04-14 Thread Saisai Shao
> is a bug or expected behavior? > > On 14.04.2017 13:22, Saisai Shao wrote: > > AFAIK, For the first line, custom filter should be worked. But for the > latter it is not supported. > > On Fri, Apr 14, 2017 at 6:17 PM, Sergey Grigorev > wrote: > >> GET requ

Re: Spark API authentication

2017-04-14 Thread Saisai Shao
/master:6066/v1/submissions/status/driver-20170414025324- > <http://master:6066/v1/submissions/status/driver-20170414025324-> *return > successful result. But if I open the spark master web ui then it requests > username and password. > > > On 14.04.2017 12:46, Saisai Shao w

Re: Spark API authentication

2017-04-14 Thread Saisai Shao
Hi, What specifically are you referring to "Spark API endpoint"? Filter can only be worked with Spark Live and History web UI. On Fri, Apr 14, 2017 at 5:18 PM, Sergey wrote: > Hello all, > > I've added own spark.ui.filters to enable basic authentication to access to > Spark web UI. It works fi

[jira] [Commented] (SPARK-16742) Kerberos support for Spark on Mesos

2017-04-13 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968545#comment-15968545 ] Saisai Shao commented on SPARK-16742: - [~mgummelt], do you have a design doc of

[jira] [Commented] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist

2017-04-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965411#comment-15965411 ] Saisai Shao commented on SPARK-20286: - bq. Maybe the best approach is to change

[jira] [Commented] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist

2017-04-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965306#comment-15965306 ] Saisai Shao commented on SPARK-20286: - So the fix is that if there's n

[jira] [Commented] (SPARK-20244) Incorrect input size in UI with pyspark

2017-04-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15964422#comment-15964422 ] Saisai Shao commented on SPARK-20244: - This actually is not a UI problem, i

[jira] [Updated] (SPARK-20275) HistoryServer page shows incorrect complete date of inprogress apps

2017-04-10 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20275: Description: The HistoryServer's incomplete page shows in-progress application's compl

[jira] [Updated] (SPARK-20275) HistoryServer page shows incorrect complete date of inprogress apps

2017-04-10 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20275: Attachment: screenshot-1.png > HistoryServer page shows incorrect complete date of inprogress a

[jira] [Created] (SPARK-20275) HistoryServer page shows incorrect complete date of inprogress apps

2017-04-10 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-20275: --- Summary: HistoryServer page shows incorrect complete date of inprogress apps Key: SPARK-20275 URL: https://issues.apache.org/jira/browse/SPARK-20275 Project: Spark

[jira] [Commented] (SPARK-16742) Kerberos support for Spark on Mesos

2017-04-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15962455#comment-15962455 ] Saisai Shao commented on SPARK-16742: - Hi [~mgummelt], I'm working on the

[jira] [Updated] (SPARK-20239) Improve HistoryServer ACL mechanism

2017-04-06 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20239: Description: Current SHS (Spark History Server) two different ACLs. * ACL of base URL, it is

[jira] [Created] (SPARK-20239) Improve HistoryServer ACL mechanism

2017-04-06 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-20239: --- Summary: Improve HistoryServer ACL mechanism Key: SPARK-20239 URL: https://issues.apache.org/jira/browse/SPARK-20239 Project: Spark Issue Type: Bug

Re: spark kafka consumer with kerberos

2017-03-31 Thread Saisai Shao
urity.auth.login.LoginException: Unable to obtain > password from user > > > On Fri, Mar 31, 2017 at 9:08 AM, Saisai Shao > wrote: > >> Hi Bill, >> >> The exception is from executor side. From the gist you provided, looks >> like the issue is that you only c

Re: spark kafka consumer with kerberos

2017-03-31 Thread Saisai Shao
Hi Bill, The exception is from executor side. From the gist you provided, looks like the issue is that you only configured java options in driver side, I think you should also configure this in executor side. You could refer to here ( https://github.com/hortonworks-spark/skc#running-on-a-kerberos-

[jira] [Created] (SPARK-20172) Event log without read permission should be filtered out before actually reading it

2017-03-31 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-20172: --- Summary: Event log without read permission should be filtered out before actually reading it Key: SPARK-20172 URL: https://issues.apache.org/jira/browse/SPARK-20172

[jira] [Commented] (SPARK-20128) MetricsSystem not always killed in SparkContext.stop()

2017-03-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946493#comment-15946493 ] Saisai Shao commented on SPARK-20128: - Sorry I cannot access the logs. What I c

[jira] [Comment Edited] (SPARK-20128) MetricsSystem not always killed in SparkContext.stop()

2017-03-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946457#comment-15946457 ] Saisai Shao edited comment on SPARK-20128 at 3/29/17 3:2

[jira] [Commented] (SPARK-20128) MetricsSystem not always killed in SparkContext.stop()

2017-03-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946457#comment-15946457 ] Saisai Shao commented on SPARK-20128: - Here the exception is from MasterSo

[jira] [Commented] (SPARK-20079) Re registration of AM hangs spark cluster in yarn-client mode

2017-03-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944629#comment-15944629 ] Saisai Shao commented on SPARK-20079: - What is the specific symptom you me

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943289#comment-15943289 ] Saisai Shao commented on SPARK-19143: - Thanks [~tgraves], let me see how to pro

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943303#comment-15943303 ] Saisai Shao commented on SPARK-19143: - [~tgraves], can I add you as a *SPIP Shep

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943243#comment-15943243 ] Saisai Shao commented on SPARK-19143: - Attach a WIP branch (https://github

[jira] [Comment Edited] (SPARK-20059) HbaseCredentialProvider uses wrong classloader

2017-03-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942976#comment-15942976 ] Saisai Shao edited comment on SPARK-20059 at 3/27/17 10:0

[jira] [Commented] (SPARK-20059) HbaseCredentialProvider uses wrong classloader

2017-03-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942976#comment-15942976 ] Saisai Shao commented on SPARK-20059: - [~sowen], it probably is not the same i

[jira] [Updated] (SPARK-20059) HbaseCredentialProvider uses wrong classloader

2017-03-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20059: Description: {{HBaseCredentialProvider}} uses system classloader instead of child classloader

Re: spark-submit config via file

2017-03-26 Thread Saisai Shao
It's quite obvious your hdfs URL is not complete, please looks at the exception, your hdfs URI doesn't have host, port. Normally it should be OK if HDFS is your default FS. I think the problem is you're running on HDI, in which default FS is wasb. So here short name without host:port will lead to

[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster

2017-03-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939867#comment-15939867 ] Saisai Shao commented on SPARK-19992: - Sorry I cannot give you valid suggest

[jira] [Commented] (SPARK-20050) Kafka 0.10 DirectStream doesn't commit last processed batch's offset when graceful shutdown

2017-03-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15937845#comment-15937845 ] Saisai Shao commented on SPARK-20050: - I think you could register a commit call

[jira] [Commented] (SPARK-20058) the running application status changed from running to waiting when a master is down and it change to another standy by master

2017-03-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15937841#comment-15937841 ] Saisai Shao commented on SPARK-20058: - I have a PR for this bug very very long

[jira] [Comment Edited] (SPARK-20058) the running application status changed from running to waiting when a master is down and it change to another standy by master

2017-03-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15937841#comment-15937841 ] Saisai Shao edited comment on SPARK-20058 at 3/23/17 7:0

[jira] [Commented] (SPARK-20058) the running application status changed from running to waiting when a master is down and it change to another standy by master

2017-03-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936255#comment-15936255 ] Saisai Shao commented on SPARK-20058: - Please subscribe this spark user mail

[jira] [Comment Edited] (SPARK-20058) the running application status changed from running to waiting when a master is down and it change to another standy by master

2017-03-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936255#comment-15936255 ] Saisai Shao edited comment on SPARK-20058 at 3/22/17 1:0

[jira] [Created] (SPARK-20059) HbaseCredentialProvider uses wrong classloader

2017-03-22 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-20059: --- Summary: HbaseCredentialProvider uses wrong classloader Key: SPARK-20059 URL: https://issues.apache.org/jira/browse/SPARK-20059 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-19992) spark-submit on deployment-mode cluster

2017-03-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936241#comment-15936241 ] Saisai Shao edited comment on SPARK-19992 at 3/22/17 12:5

[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster

2017-03-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936241#comment-15936241 ] Saisai Shao commented on SPARK-19992: - Oh, I see. Check the code again, looks

[jira] [Commented] (SPARK-19941) Spark should not schedule tasks on executors on decommissioning YARN nodes

2017-03-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932088#comment-15932088 ] Saisai Shao commented on SPARK-19941: - I think this scenario is quite simila

[jira] [Created] (SPARK-19995) Using real user to connect HiveMetastore in HiveClientImpl

2017-03-17 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-19995: --- Summary: Using real user to connect HiveMetastore in HiveClientImpl Key: SPARK-19995 URL: https://issues.apache.org/jira/browse/SPARK-19995 Project: Spark

[jira] [Comment Edited] (SPARK-19992) spark-submit on deployment-mode cluster

2017-03-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929547#comment-15929547 ] Saisai Shao edited comment on SPARK-19992 at 3/17/17 7:4

[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster

2017-03-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929547#comment-15929547 ] Saisai Shao commented on SPARK-19992: - Looks like I guess wrong from the url

[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster

2017-03-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929544#comment-15929544 ] Saisai Shao commented on SPARK-19992: - Are you using HDP environment, if so I g

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904497#comment-15904497 ] Saisai Shao commented on SPARK-19143: - Hi all, I wrote a rough design doc base

Re: question on Write Ahead Log (Spark Streaming )

2017-03-08 Thread Saisai Shao
IIUC, your scenario is quite like what currently ReliableKafkaReceiver does. You can only send ack to the upstream source after WAL is persistent, otherwise because of asynchronization of data processing and data receiving, there's still a chance data could be lost if you send out ack before WAL.

[jira] [Comment Edited] (SPARK-19812) YARN shuffle service fails to relocate recovery DB directories

2017-03-07 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15900821#comment-15900821 ] Saisai Shao edited comment on SPARK-19812 at 3/8/17 7:4

[jira] [Comment Edited] (SPARK-19812) YARN shuffle service fails to relocate recovery DB directories

2017-03-07 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15900821#comment-15900821 ] Saisai Shao edited comment on SPARK-19812 at 3/8/17 7:4

[jira] [Commented] (SPARK-19812) YARN shuffle service fails to relocate recovery DB directories

2017-03-07 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15900821#comment-15900821 ] Saisai Shao commented on SPARK-19812: - [~tgraves], I'm not quite sure what

[jira] [Commented] (SPARK-19802) Remote History Server

2017-03-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893580#comment-15893580 ] Saisai Shao commented on SPARK-19802: - Spark's {{ApplicationHistoryProv

Re: How to use ManualClock with Spark streaming

2017-02-28 Thread Saisai Shao
I don't think using ManualClock is a right way to fix your problem here in Spark Streaming. ManualClock in Spark is mainly used for unit test, it should manually advance the time to make the unit test work. The usage looks different compared to the scenario you mentioned. Thanks Jerry On Tue, Fe

[jira] [Commented] (SPARK-19750) Spark UI http -> https redirect error

2017-02-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887204#comment-15887204 ] Saisai Shao commented on SPARK-19750: - This issue was found by [~yeshavora], cre

Re: spark.speculation setting support on standalone mode?

2017-02-27 Thread Saisai Shao
I think it should be. These configurations doesn't depend on specific cluster manager use chooses. On Tue, Feb 28, 2017 at 4:42 AM, satishl wrote: > Are spark.speculation and related settings supported on standalone mode? > > > > -- > View this message in context: http://apache-spark-user-list

[jira] [Created] (SPARK-19750) Spark UI http -> https redirect error

2017-02-27 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-19750: --- Summary: Spark UI http -> https redirect error Key: SPARK-19750 URL: https://issues.apache.org/jira/browse/SPARK-19750 Project: Spark Issue Type:

[jira] [Commented] (SPARK-19688) Spark on Yarn Credentials File set to different application directory

2017-02-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882027#comment-15882027 ] Saisai Shao commented on SPARK-19688: - According to my

[jira] [Updated] (SPARK-19707) Improve the invalid path check for sc.addJar

2017-02-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-19707: Summary: Improve the invalid path check for sc.addJar (was: Improve the invalid path handling for

[jira] [Created] (SPARK-19707) Improve the invalid path handling for sc.addJar

2017-02-23 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-19707: --- Summary: Improve the invalid path handling for sc.addJar Key: SPARK-19707 URL: https://issues.apache.org/jira/browse/SPARK-19707 Project: Spark Issue Type

[jira] [Commented] (SPARK-19688) Spark on Yarn Credentials File set to different application directory

2017-02-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15879893#comment-15879893 ] Saisai Shao commented on SPARK-19688: - I see. So what issue did you encounter

[jira] [Commented] (SPARK-19688) Spark on Yarn Credentials File set to different application directory

2017-02-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15879883#comment-15879883 ] Saisai Shao commented on SPARK-19688: - [~j.devaraj], when you say Spark applica

Re: Why spark history server does not show RDD even if it is persisted?

2017-02-22 Thread Saisai Shao
ri wrote: > Thanks a lot the information! > > Is there any reason why EventLoggingListener ignore this event? > > *Thanks,* > > > *​Parag​* > > On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao > wrote: > >> AFAIK, Spark's EventLoggingListerner ignores Bl

Re: Why spark history server does not show RDD even if it is persisted?

2017-02-22 Thread Saisai Shao
AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it will not be written into event-log, I think that's why you cannot get such info in history server. On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari wrote: > Hi, > > I am running spark shell in spark version 2.0.2. Here is my p

[jira] [Commented] (SPARK-19688) Spark on Yarn Credentials File set to different application directory

2017-02-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15879564#comment-15879564 ] Saisai Shao commented on SPARK-19688: - I see, so we should exclude

[jira] [Updated] (SPARK-19688) Spark on Yarn Credentials File set to different application directory

2017-02-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-19688: Component/s: DStreams > Spark on Yarn Credentials File set to different application direct

[jira] [Commented] (SPARK-19688) Spark on Yarn Credentials File set to different application directory

2017-02-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15877587#comment-15877587 ] Saisai Shao commented on SPARK-19688: - Can you please elaborate the problem you

[jira] [Commented] (SPARK-19649) Spark YARN client throws exception if job succeeds and max-completed-applications=0

2017-02-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874101#comment-15874101 ] Saisai Shao commented on SPARK-19649: - MapReduce could delegate to history serve

[jira] [Commented] (SPARK-19649) Spark YARN client throws exception if job succeeds and max-completed-applications=0

2017-02-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874098#comment-15874098 ] Saisai Shao commented on SPARK-19649: - >From my understanding, this looks

[jira] [Commented] (SPARK-19588) Allow putting keytab file to HDFS location specified in spark.yarn.keytab

2017-02-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865434#comment-15865434 ] Saisai Shao commented on SPARK-19588: - Putting on HDFS still requires downloadin

[jira] [Commented] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865360#comment-15865360 ] Saisai Shao commented on SPARK-19579: - Only this specific part is not supported,

[jira] [Closed] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-19579. --- Resolution: Won't Fix > spark-submit fails to run Kafka Stream pytho

[jira] [Comment Edited] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865360#comment-15865360 ] Saisai Shao edited comment on SPARK-19579 at 2/14/17 8:3

[jira] [Commented] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865346#comment-15865346 ] Saisai Shao commented on SPARK-19579: - The support of Python API is rejected by

[jira] [Commented] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865330#comment-15865330 ] Saisai Shao commented on SPARK-19579: - If you're using Python API t

Review Request 56641: Change Livy recovery folder permission to 0700

2017-02-13 Thread Saisai Shao
--- Manual verification. Thanks, Saisai Shao

<    7   8   9   10   11   12   13   14   15   16   >