Thanks, we'll give this a try!
Niels Basjes
On Mon, Jun 8, 2015 at 10:04 PM, Rohini Palaniswamy <rohini.adi...@gmail.com
> wrote:
> Niels,
> I plan to have PIG-3038 in next two weeks which should simplify
> accessing secure hbase, but it will only be in 0.16 and that is at least
> 3-4 months away.
>
> In the meantime, a hacky way to get this done is:
>
> When running the pig script from commandline, do
> ## Makes bin/pig add hbase jars and hbase-site.xml to classpath
> export HBASE_HOME=/home/gs/hbase/current
> export HBASE_CONF_DIR=/home/gs/conf/hbase
>
> pig --conf /home/gs/conf/hbase/hbase-site.xml
>
> -Dmapreduce.job.credentials.binary=/home/<username>/hbase-creds/myhbaseudf-`date
> +%Y%m%d%H%M`.token myscript.pig
>
> Also define your UDF in the pig script as follows. This is required for the
> pig script to connect to hbase and fetch the hbase delegation token on the
> front end. This is not required if you are running the script through
> Oozie.
>
>
> define myudf com.yahoo.myudfs.MyHBaseUDF();
>
>
> MyHBaseUDF.java :
>
> package com.yahoo.myudfs;
>
> import java.io.File;
> import java.io.IOException;
> import java.lang.reflect.Method;
> import java.lang.reflect.UndeclaredThrowableException;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.hbase.HBaseConfiguration;
> import org.apache.hadoop.hbase.client.HTable;
> import org.apache.hadoop.mapred.JobConf;
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.security.Credentials;
> import org.apache.hadoop.security.UserGroupInformation;
> import org.apache.pig.EvalFunc;
> import org.apache.pig.data.Tuple;
> import org.apache.pig.impl.util.UDFContext;
>
> public class MyHBaseUDF extends EvalFunc<Tuple> {
>
> private final static String HBASE_SECURITY_CONF_KEY =
> "hbase.security.authentication";
> private JobConf hbaseConf;
> private HTable hTable;
>
> public MyHBaseUDF() {
> try {
> if (UDFContext.getUDFContext().isFrontend()) {
> hbaseConf = new JobConf(HBaseConfiguration.create());
>
>
> if(UserGroupInformation.getCurrentUser().hasKerberosCredentials()) {
> // Fetch the hbase delegation and write to a location,
> which can be added to the Job
> // when it is launched. This is not required if you are
> using Oozie to run pig scripts.
> // HBase credentials should be fetched by configuring
> the credentials section through Oozie.
> // and will be available via HADOOP_TOKEN_FILE_LOCATION
> in both FrontEnd and BackEnd.
>
> addHBaseDelegationToken(hbaseConf);
> // Pass
>
> -Dmapreduce.job.credentials.binary=/homes/<username>/hbase-creds/myhbaseudf.token
> // to pig script. It can be any local filesystem
> location where read and write access
> // are restricted to you.
> String binaryTokenFilename =
> System.getProperty("mapreduce.job.credentials.binary");
> Credentials hbaseCreds = hbaseConf.getCredentials();
> hbaseCreds.writeTokenStorageFile(new Path("file:///" +
> binaryTokenFilename),
> hbaseConf);
> } else {
> // Case of Oozie
> String tokenFileLocation = System
>
> .getenv(UserGroupInformation.HADOOP_TOKEN_FILE_LOCATION);
> if (tokenFileLocation != null) {
> Credentials cred =
> Credentials.readTokenStorageFile(new Path("file:///" + tokenFileLocation),
> hbaseConf);
> hbaseConf.getCredentials().addAll(cred);
> }
> }
>
> } else {
> // Construct JobConf with the hbase delegation tokens to
> talk to hbase
> Configuration conf =
> UDFContext.getUDFContext().getJobConf();
> // 1) You need to either ship hbase-site.xml through
> distributed cache (-Dmapred.cache.files)
> // so that it is in classpath (or)
> // 2) pass it to pig using --conf
> /home/gs/conf/hbase/hbase-site.xml, so that it
> // becomes part of the job configuration
> hbaseConf = new JobConf(HBaseConfiguration.create(conf));
> String tokenFileLocation = System
>
> .getenv(UserGroupInformation.HADOOP_TOKEN_FILE_LOCATION);
> if (tokenFileLocation != null) {
> Credentials cred = Credentials.readTokenStorageFile(new
> Path("file:///" + tokenFileLocation), hbaseConf);
> hbaseConf.getCredentials().addAll(cred);
> }
> }
> hTable = new HTable(hbaseConf, "myhbaseTable");
> } catch (IOException e) {
> throw new RuntimeException(e);
> }
> }
>
> // Not required if using Oozie
> @SuppressWarnings({ "rawtypes", "unchecked" })
> private void addHBaseDelegationToken(JobConf hbaseConf) {
>
> if (!UDFContext.getUDFContext().isFrontend()) {
> return;
> }
>
> if
> ("kerberos".equalsIgnoreCase(hbaseConf.get(HBASE_SECURITY_CONF_KEY))) {
> try {
> if
> (UserGroupInformation.getCurrentUser().hasKerberosCredentials()) {
> // Class and method are available only from 0.92
> security release
> Class tokenUtilClass = Class
>
> .forName("org.apache.hadoop.hbase.security.token.TokenUtil");
> Method m =
> tokenUtilClass.getMethod("obtainTokenForJob", new Class[] {
> JobConf.class, UserGroupInformation.class });
> m.invoke(null, new Object[] { hbaseConf,
> UserGroupInformation.getCurrentUser() });
> } else {
> System.out.println("Not fetching hbase delegation token
> as no Kerberos TGT is available");
> }
> } catch (ClassNotFoundException cnfe) {
> throw new RuntimeException("Failure loading TokenUtil
> class, "
> + "is secure RPC available?", cnfe);
> } catch (Exception e) {
> throw new UndeclaredThrowableException(e,
> "Unexpected error calling
> TokenUtil.obtainTokenForJob()");
> }
> }
> }
>
> @Override
> public Tuple exec(Tuple input) throws IOException {
> // Your code goes here
> return null;
> }
>
> }
>
>
>
> Regards,
> Rohini
>
> On Thu, May 28, 2015 at 7:50 AM, Niels Basjes <ni...@basjes.nl> wrote:
>
> > Hi,
> >
> > I have a UDF that needs to go into HBase to get a single value when
> called.
> > The HBase we have has been secured (Kerberos) and so far I have not yet
> > been able to figure out how to get it all running.
> >
> > I did find these two issues but no working example I can copy:
> >
> > "Helper class for dealing with security in HBase for UDFs"
> > https://issues.apache.org/jira/browse/PIG-3030
> >
> > and
> >
> > "Support for Credentials for UDF,Loader and Storer"
> > https://issues.apache.org/jira/browse/PIG-3038
> >
> > Does anyone have a suggestion on how to approach this correctly ??
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
