Re: Facing Issue while connecting with HDFS

[Bryan Bende]

Hello,

By default NiFi will interact with HDFS as the user that started the NiFi
process. If you can only access HDFS as the superuser then you will
probably need to start NiFi as that user.

There is a "run.as" property in conf/bootstrap.conf where you can specify a
username to run NiFi as, but I'm not sure that is necessary here since it
sounds like you need to run it as superuser.

-Bryan

On Sun, Dec 13, 2015 at 11:49 PM, digvijayp 
wrote:

> Hi Bryan,
>
> I am saying that in our HDFS cluster specific user have the
> permission(superuser) to access it.So when ever we have to do any operation
> on cluster we need to first login by our id and then superuser and then we
> can access the cluster.
> When I trying to connect to NIFI with HDFS cluster it is trying to login by
> my id and then not with the superuser .Due to which I am getting the
> authentication issue,so I need to provide superuser credential in the nifi
> so that it will first login and then do connection.not sure where do I add
> these detail in the nifi?
>
> Thanks for support.
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Facing-Issue-while-connecting-with-HDFS-tp5684p5769.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Facing Issue while connecting with HDFS

[digvijayp]

Hi Bryan,

I am saying that in our HDFS cluster specific user have the
permission(superuser) to access it.So when ever we have to do any operation
on cluster we need to first login by our id and then superuser and then we
can access the cluster.
When I trying to connect to NIFI with HDFS cluster it is trying to login by
my id and then not with the superuser .Due to which I am getting the
authentication issue,so I need to provide superuser credential in the nifi
so that it will first login and then do connection.not sure where do I add
these detail in the nifi?

Thanks for support.



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Facing-Issue-while-connecting-with-HDFS-tp5684p5769.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Facing Issue while connecting with HDFS

[Bryan Bende]

Digvijay,

Are you talking about Kerberos authentication to the HDFS cluster?

If so, in nifi.properties you specify your krb5.conf file:

nifi.kerberos.krb5.file=/etc/krb5.conf  (or wherever your conf file is)

 This is the file that would have your realms defined. Then on the HDFS
processors your
properties would be something like:

Kerberos Principal = myprinicpal@MYREALM
Kerberos Keytab = /etc/security/keytabs/myprinciapl.keytab

-Bryan


On Fri, Dec 11, 2015 at 6:00 AM, digvijayp 
wrote:

> Hi Bryan,
>
> Thanks for nice info...It would really help me.
> Facing one issue while designing putHDFS process.After putting the putHDFS
> configuration details I am getting the error mentioning security
> authentication .After the analysis I found that these is due to permission
> of userid.
> So where can we put the userid and password so that when it connect to hdfs
> first login by that id and then pulls back data.As I know it should be done
> in the nifi.properties file.But can you please share where I can add these
> details?
>
> Thanks
> Digvijay P.
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Facing-Issue-while-connecting-with-HDFS-tp5684p5720.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Facing Issue while connecting with HDFS

[digvijayp]

Hi Bryan,
So in edge node approach how data sent in site-to-site ?I mean to say is it
using any protocol to transfer it like FTP,SFTP.
As you are saying If both clusters can fully talk to each other than you
don't need this edge node approach, you could just have a NiFi instance, or
cluster, that pulls from one HDFS and pushes to the other.
so my query is we have to use FetchHDFS/getHDFS process which get data from
HDFS to local machine and putHDFS process which load data from local machine
to HDFS.I dont have yo use the local machin in between .So how can we manage
the transfer data without using local machine? Where can we do such
configuration in nifi? 

Thanks in advance.

Digvijay P.



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Facing-Issue-while-connecting-with-HDFS-tp5684p5712.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Facing Issue while connecting with HDFS

[Bryan Bende]

Site-to-Site is a direct connection between NiFi instances/clusters over a
socket, so TCP based.

There will always have to be at least one local machine involved. When NiFi
pulls/receives data from somewhere, it takes that data under control and
stores it in the NiFi content repository on disk (configured in
nifi.properties). As a FlowFile moves through the flow, a pointer to this
content is being passed around until it needs to be accessed. So when
PutHDFS needs to send to the other cluster it would read the content and
send to the other HDFS. The data would then eventually age-off from the
NiFi content repository depending how it is configured. So it would not
have to hold all of the data on the local machine, but it would always have
some portion of the most recent data that has been moved across.

Let us know if this doesn't make sense.

-Bryan




On Thu, Dec 10, 2015 at 1:52 AM, digvijayp 
wrote:

> Hi Bryan,
> So in edge node approach how data sent in site-to-site ?I mean to say is it
> using any protocol to transfer it like FTP,SFTP.
> As you are saying If both clusters can fully talk to each other than you
> don't need this edge node approach, you could just have a NiFi instance, or
> cluster, that pulls from one HDFS and pushes to the other.
> so my query is we have to use FetchHDFS/getHDFS process which get data from
> HDFS to local machine and putHDFS process which load data from local
> machine
> to HDFS.I dont have yo use the local machin in between .So how can we
> manage
> the transfer data without using local machine? Where can we do such
> configuration in nifi?
>
> Thanks in advance.
>
> Digvijay P.
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Facing-Issue-while-connecting-with-HDFS-tp5684p5712.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Facing Issue while connecting with HDFS

[Bryan Bende]

It doesn't necessarily have to be on the same machine, but the machine NiFi
is on would have to be able to communicate with the name-node and
data-nodes in order to push/pull data to/from HDFS. In your example this
would mean your local machine would need to be able to access the name-node
and data-node on your VM.

In the cluster to cluster scenario... If each cluster is mostly closed off
from a networking perspective, you could potentially have edge nodes on
each cluster that were able to reach each other. Each of those edge nodes
could run a NiFi instance, and the two NiFi instances could talk directly
to each other. The first one would use List/FetchHDFS and would have to be
able to communicate with cluster #1, it would send data via site-to-site to
the second instance which would use PutHDFS and have to be able to
communicate with cluster #2. If both clusters can fully talk to each other
than you don't need this edge node approach, you could just have a NiFi
instance, or cluster, that pulls from one HDFS and pushes to the other.

As far as comparing to distcp, keep in mind that distcp launches a
map-reduce job to perform a heavily parallelized copy, this would work a
little different in NiFi. If you had a NiFi cluster you could scale it so
each node in the cluster was pulling data, otherwise with a single instance
it would be limited to how much processing that instance can perform.

Hope this helps.

-Bryan

On Wed, Dec 9, 2015 at 12:01 AM, digvijayp 
wrote:

> Thanks for the responce Bryan 
>
> I am getting the same error when applying the /root as directory.
> So is it necessary to install nifi on the same machine when we install the
> hadoop?
> Basically i am trying to explore to use nifi to data flow from 1 HDFS
> cluster to other HDFS cluster instead of doing manually by distcp. Is nifi
> recommended to be used in such scenario?
>
> Thanks in advance
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Facing-Issue-while-connecting-with-HDFS-tp5684p5703.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Facing Issue while connecting with HDFS

[Bryan Bende]

Hello,

The directory property should only need the path in HDFS. The other stuff
like the file system, host, and port would be determined from the provided
configuration files.

Do you receive any different response if you set the directory to just
"/root" (or some other path)?

If you still receive the same error you may want to verify you can connect
to that port from outside your vm, possibly without even using NiFi.

-Bryan

On Tuesday, December 8, 2015, digvijayp 
wrote:

> Hi Team,
>
> I have been exploring NiFi for couple of days now.
>
> NiFi is running on a machine which is not a part of Hadoop cluster. I want
> to put files into HDFS (On my machine I have configured the hortonworks
> sandbox by using virtual machine ). As to writ into HDFS I have created the
> process PUTHDFS in Ni-fi.As per my understanding I have to do following
> setting to connect with HDFS:
>
> 1.setting of Hadoop configuration Resources:I have also copied
> hdfs-site.xml
> and core-site.xml into nifi installed windows directory.Path of these xml
> is
> given to the Hadoop configuration Resources.
> 2. setting of Directory:For the directory structure i have given the url
> "hdfs://127.0.0.1:8080//root" which is hortonworks default url and port.
> Still I am getting the error "unresolvedaddressexception".Is something I am
> missing?
>
> Thanks in advance,
> Digvijay P.
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Facing-Issue-while-connecting-with-HDFS-tp5684.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>


-- 
Sent from Gmail Mobile