Thanks Yan. I will take a look shortly. On Sat, Jul 25, 2015 at 1:20 AM, Yan Fang <yanfang...@gmail.com> wrote:
> Hi Chen Song, > > If you can work on this issue, it will be great. > > 1. the related ticket is https://issues.apache.org/jira/browse/SAMZA-727 > > 2. most of the change will happen in Yarn AM and Yarn client parts. The > code sits in the samza-yarn package > < > https://github.com/apache/samza/tree/master/samza-yarn/src/main/scala/org/apache/samza/job/yarn > > > . > > 3. when you implement this, make sure it does not affect the non-secure > Yarn implementation. Because non-secure cluster implementation has been > proved working, while the secure cluster may have the issue as Yi Pan > mentioned, "For a long-running > Samza job, it does not work. We will need a way to refresh the Kerberos > ticket > periodically, which is not supported yet. " But I am happy to see at least > we have some support for secure cluster. We can figure the issue out later. > > If you want to have some help in understanding the existing code, let me > know. > > Thanks, > > Fang, Yan > yanfang...@gmail.com > > On Fri, Jul 24, 2015 at 7:00 PM, Chen Song <chen.song...@gmail.com> wrote: > > > Can someone give some context on this? I can volunteer myself and try > > working on this. > > > > Chen > > > > On Thu, Jul 2, 2015 at 4:29 AM, Qi Fu <q...@talend.com> wrote: > > > > > Hi Yi & Yan, > > > > > > Many thanks for your information. I have created a jira for this: > > > https://issues.apache.org/jira/browse/SAMZA-727 > > > I'm willing to test it if someone can work on this. > > > > > > > > > -Qi > > > > > > ________________________________________ > > > From: Yi Pan <nickpa...@gmail.com> > > > Sent: Thursday, July 2, 2015 1:38 AM > > > To: dev@samza.apache.org > > > Subject: Re: Security on YARN > > > > > > Hi, Yan, > > > > > > Your memory serves as well as mine. :) I remember that Chris and I > > > discussed this Kerberos ticket expiration issue when we were brain > > storming > > > on how to access HDFS data in Samza. At high-level, what happens is > that > > > the Kerberos ticket to access a secured Hadoop cluster is issued to > Samza > > > containers at the job start time, and will expire later. For a > > long-running > > > Samza job, it does not work. We will need a way to refresh the Kerberos > > > ticket periodically, which is not supported yet. Chris probably can > chime > > > in with more details. > > > > > > -Yi > > > > > > On Wed, Jul 1, 2015 at 4:08 PM, Yan Fang <yanfang...@gmail.com> wrote: > > > > > > > Hi Qi, > > > > > > > > I think this is caused by the fact that Samza currently does not > > support > > > > Yarn with Kerberos. Feel free to open a ticket for this feature. > > > > > > > > But if my memory serves, there was an issue mentioned about the > > Kerberos. > > > > Seems when the Kerberos ticket expires, Samza will have some issues? > > Can > > > > not find the resource. Anyone remember this? > > > > > > > > Cheers, > > > > > > > > Fang, Yan > > > > yanfang...@gmail.com > > > > > > > > On Wed, Jul 1, 2015 at 3:41 AM, Qi Fu <q...@talend.com> wrote: > > > > > > > > > Hi all, > > > > > > > > > > > > > > > I'm testing Samza on YARN and I have encountered a problem on the > > > > security > > > > > setting of YARN (Kerberos). Here is the detail: > > > > > > > > > > 1. My cluster is secured by Kerberos, and I deploy my samza job > from > > > one > > > > > of the cluster. > > > > > > > > > > > > > > > 2. My config file is in ~/.samza/conf/(yarn-site.xml, > core-site.xml, > > > > > hdfs-site.xml) > > > > > > > > > > > > > > > 3. The job is deployed successfully, and I can get the info such > as: > > > > > > > > > > ClientHelper [INFO] set package url to scheme: "hdfs" port: -1 > > > file: > > > > > "/user/test/samzatest.tar.gz" for application_1435680272316_0003 > > > > > > > > > > ClientHelper [INFO] set package size to 212924524 for > > > > > application_1435680272316_0003 > > > > > > > > > > > > > > > > > > > > I think the security setting is correct as it can get the file > > size > > > > > from HDFS. > > > > > > > > > > > > > > > 4. But I get the error from YARN job manager as following: > > > > > > > > > > > > > > > Application application_1435680272316_0003 failed 2 times due > to > > AM > > > > > Container for appattempt_1435680272316_0003_000002 exited with > > > exitCode: > > > > > -1000 > > > > > > > > > > For more detailed output, check application tracking page: > > > > > http://cdh-namenode:8088/proxy/application_1435680272316_0003/Then > , > > > > click > > > > > on links to logs of each attempt. > > > > > > > > > > Diagnostics: Failed on local exception: java.io.IOException: > > > > > org.apache.hadoop.security.AccessControlException: Client cannot > > > > > authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: > > > > > "talend-cdh-datanode8/62.210.141.237"; destination host is: > > > > > "talend-cdh-namenode":8020; > > > > > > > > > > java.io.IOException: Failed on local exception: > java.io.IOException: > > > > > org.apache.hadoop.security.AccessControlException: Client cannot > > > > > authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: > > > > > "cdh-datanode8/62.210.141.237"; destination host is: > > > > "cdh-namenode":8020; > > > > > > > > > > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) > > > > > > > > > > ...... > > > > > > > > > > > > > > > > > > > > Anyone knows how to solve this? > > > > > > > > > > > > > > > Qi FU > > > > > > > > > > > > > > > > > > > > -- > > Chen Song > > > -- Chen Song
