I just read an interesting comment on cloudera: What does it mean by “when the job is submitted,and you have a kinit, you will have TOKEN to access HDFS, you would need to pass that on, or the KERBEROS ticket” ?
Reference <https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/org-apache-hadoop-security-AccessControlException-SIMPLE/td-p/28082> and full quote: In a cluster which is kerberised there is no SIMPLE authentication. Make sure that you have run kinit before you run the application. Second thing to check: In your application you need to do the right thing and either pass on the TOKEN or a KERBEROS ticket. When the job is submitted, and you have done a kinit, you will have TOKEN to access HDFS you would need to pass that on, or the KERBEROS ticket. You will need to handle this in your code. I can not see exactly what you are doing at that point in the startup of your code but any HDFS access will require a TOKEN or KERBEROS ticket. Cheers, Wilfred > On 8 Dec 2016, at 08:35, Gerard Casey <gerardhughca...@gmail.com> wrote: > > Thanks Marcelo. > > I’ve completely removed it. Ok - even if I read/write from HDFS? > > Trying to the SparkPi example now > > G > >> On 7 Dec 2016, at 22:10, Marcelo Vanzin <van...@cloudera.com >> <mailto:van...@cloudera.com>> wrote: >> >> Have you removed all the code dealing with Kerberos that you posted? >> You should not be setting those principal / keytab configs. >> >> Literally all you have to do is login with kinit then run spark-submit. >> >> Try with the SparkPi example for instance, instead of your own code. >> If that doesn't work, you have a configuration issue somewhere. >> >> On Wed, Dec 7, 2016 at 1:09 PM, Gerard Casey <gerardhughca...@gmail.com >> <mailto:gerardhughca...@gmail.com>> wrote: >>> Thanks. >>> >>> I’ve checked the TGT, principal and key tab. Where to next?! >>> >>>> On 7 Dec 2016, at 22:03, Marcelo Vanzin <van...@cloudera.com >>>> <mailto:van...@cloudera.com>> wrote: >>>> >>>> On Wed, Dec 7, 2016 at 12:15 PM, Gerard Casey <gerardhughca...@gmail.com >>>> <mailto:gerardhughca...@gmail.com>> wrote: >>>>> Can anyone point me to a tutorial or a run through of how to use Spark >>>>> with >>>>> Kerberos? This is proving to be quite confusing. Most search results on >>>>> the >>>>> topic point to what needs inputted at the point of `sparks submit` and not >>>>> the changes needed in the actual src/main/.scala file >>>> >>>> You don't need to write any special code to run Spark with Kerberos. >>>> Just write your application normally, and make sure you're logged in >>>> to the KDC (i.e. "klist" shows a valid TGT) before running your app. >>>> >>>> >>>> -- >>>> Marcelo >>> >> >> >> >> -- >> Marcelo >
