subject:"Re\: about replication"

Re: about replication

2013-08-05 Thread Mohammad Tariq

Hello Irfan,

You can find all the answers from HDFS architecture
guide.
See the section Data
Organizationin
particular for this question.

Warm Regards,
Tariq
cloudfront.blogspot.com

On Mon, Aug 5, 2013 at 12:23 PM, Irfan Sayed  wrote:

> hi,
>
> i have setup the two node apache hadoop cluster on windows environment
> one is namenode and another is datanode
>
> everything is working fine.
> one thing which i need to know is , how the replication starts
> if i create a.txt in namenode , how it will be appeared in datanodes
>
> please suggest
>
> regards
>
>

Re: about replication

2013-08-05 Thread Irfan Sayed

thanks mohammad
i ran the below command on NameNode

$ ./hadoop dfs -mkdir /wksp

and the "wksp" dir got created in c:\ ( as i have windows environment)

now , when i log in to one of the DataNode , then i am not able to see
c:\wksp

any issue ?
please suggest

regards





On Mon, Aug 5, 2013 at 3:03 PM, Mohammad Tariq  wrote:

> Hello Irfan,
>
> You can find all the answers from HDFS architecture 
> guide.
> See the section Data 
> Organizationin
>  particular for this question.
>
> Warm Regards,
> Tariq
> cloudfront.blogspot.com
>
>
> On Mon, Aug 5, 2013 at 12:23 PM, Irfan Sayed  wrote:
>
>> hi,
>>
>> i have setup the two node apache hadoop cluster on windows environment
>> one is namenode and another is datanode
>>
>> everything is working fine.
>> one thing which i need to know is , how the replication starts
>> if i create a.txt in namenode , how it will be appeared in datanodes
>>
>> please suggest
>>
>> regards
>>
>>
>

Re: about replication

2013-08-05 Thread Mohammad Tariq

You cannot physically see the HDFS files and directories through local FS.
Either use HDFS shell or HDFS webUI(namenode_machine:50070).

Warm Regards,
Tariq
cloudfront.blogspot.com


On Mon, Aug 5, 2013 at 4:46 PM, Irfan Sayed  wrote:

> thanks mohammad
> i ran the below command on NameNode
>
> $ ./hadoop dfs -mkdir /wksp
>
> and the "wksp" dir got created in c:\ ( as i have windows environment)
>
> now , when i log in to one of the DataNode , then i am not able to see
> c:\wksp
>
> any issue ?
> please suggest
>
> regards
>
>
>
>
>
> On Mon, Aug 5, 2013 at 3:03 PM, Mohammad Tariq  wrote:
>
>> Hello Irfan,
>>
>> You can find all the answers from HDFS architecture 
>> guide.
>> See the section Data 
>> Organizationin
>>  particular for this question.
>>
>> Warm Regards,
>> Tariq
>> cloudfront.blogspot.com
>>
>>
>> On Mon, Aug 5, 2013 at 12:23 PM, Irfan Sayed wrote:
>>
>>> hi,
>>>
>>> i have setup the two node apache hadoop cluster on windows environment
>>> one is namenode and another is datanode
>>>
>>> everything is working fine.
>>> one thing which i need to know is , how the replication starts
>>> if i create a.txt in namenode , how it will be appeared in datanodes
>>>
>>> please suggest
>>>
>>> regards
>>>
>>>
>>
>

Re: about replication

2013-08-05 Thread Irfan Sayed

thanks.
please refer below:

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -ls /wksp
Found 1 items
drwxr-xr-x   - Administrator Domain  0 2013-08-05 16:58 /wksp/New
folder

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$


same command if i run on the datanode then it says:

Administrator@DFS-1 /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -ls /wksp
ls: Cannot access /wksp: No such file or directory.

does it mean that replication is not yet started ...?

please suggest

regards



On Mon, Aug 5, 2013 at 5:18 PM, Mohammad Tariq  wrote:

> You cannot physically see the HDFS files and directories through local FS.
> Either use HDFS shell or HDFS webUI(namenode_machine:50070).
>
> Warm Regards,
> Tariq
> cloudfront.blogspot.com
>
>
> On Mon, Aug 5, 2013 at 4:46 PM, Irfan Sayed  wrote:
>
>> thanks mohammad
>> i ran the below command on NameNode
>>
>> $ ./hadoop dfs -mkdir /wksp
>>
>> and the "wksp" dir got created in c:\ ( as i have windows environment)
>>
>> now , when i log in to one of the DataNode , then i am not able to see
>> c:\wksp
>>
>> any issue ?
>> please suggest
>>
>> regards
>>
>>
>>
>>
>>
>> On Mon, Aug 5, 2013 at 3:03 PM, Mohammad Tariq wrote:
>>
>>> Hello Irfan,
>>>
>>> You can find all the answers from HDFS architecture 
>>> guide.
>>> See the section Data 
>>> Organizationin
>>>  particular for this question.
>>>
>>> Warm Regards,
>>> Tariq
>>> cloudfront.blogspot.com
>>>
>>>
>>> On Mon, Aug 5, 2013 at 12:23 PM, Irfan Sayed wrote:
>>>
 hi,

 i have setup the two node apache hadoop cluster on windows environment
 one is namenode and another is datanode

 everything is working fine.
 one thing which i need to know is , how the replication starts
 if i create a.txt in namenode , how it will be appeared in datanodes

 please suggest

 regards


>>>
>>
>

Re: about replication

2013-08-05 Thread manish dunani

You can not physically access the datanode.You have to understand it to
logically and it really happens.

Type "jps" command to check ur datanode was started or not.

when user stores the file into hdfs ,the request is goes to datanode and
datanode will divide the file into number of blocks.
Each block size is 64mb or 128mb.but,default is 64 mb.

The default replication factor is 1 that was u already set in ur
hdfs-site.xml .If u want to change ur replication factor from 1 to 3 for
particular file or particular directory in hdfs then use the below commands
appropriately.

First load any file to hdfs using command:

bin/hadoop  dfs -copyFromLocal /ur/local/directory/path/to/file
 /ur/hdfs/directory/path.

*commands:*

To set replication of an individual file to 4:

./bin/hadoop dfs -setrep -w 4 /path/to/file

You can also do this recursively. To change replication of entire HDFS to 1:

./bin/hadoop dfs -setrep -R -w 1 /
*
*
 If the replication factor is 3 then data will divided into 3 blocks and
replicated to various data nodes of ur production cluster(multi node).

Here Replication means same file copied three times on data nodes for
handling hardware failure.

On Mon, Aug 5, 2013 at 5:29 PM, Irfan Sayed  wrote:

> thanks.
> please refer below:
>
> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
> $ ./hadoop dfs -ls /wksp
> Found 1 items
> drwxr-xr-x   - Administrator Domain  0 2013-08-05 16:58 /wksp/New
> folder
>
> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
> $
>
>
> same command if i run on the datanode then it says:
>
> Administrator@DFS-1 /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
> $ ./hadoop dfs -ls /wksp
> ls: Cannot access /wksp: No such file or directory.
>
> does it mean that replication is not yet started ...?
>
> please suggest
>
> regards
>
>
>
> On Mon, Aug 5, 2013 at 5:18 PM, Mohammad Tariq  wrote:
>
>> You cannot physically see the HDFS files and directories through local
>> FS. Either use HDFS shell or HDFS webUI(namenode_machine:50070).
>>
>> Warm Regards,
>> Tariq
>> cloudfront.blogspot.com
>>
>>
>> On Mon, Aug 5, 2013 at 4:46 PM, Irfan Sayed  wrote:
>>
>>> thanks mohammad
>>> i ran the below command on NameNode
>>>
>>> $ ./hadoop dfs -mkdir /wksp
>>>
>>> and the "wksp" dir got created in c:\ ( as i have windows environment)
>>>
>>> now , when i log in to one of the DataNode , then i am not able to see
>>> c:\wksp
>>>
>>> any issue ?
>>> please suggest
>>>
>>> regards
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Aug 5, 2013 at 3:03 PM, Mohammad Tariq wrote:
>>>
 Hello Irfan,

 You can find all the answers from HDFS architecture 
 guide.
 See the section Data 
 Organizationin
  particular for this question.

 Warm Regards,
 Tariq
 cloudfront.blogspot.com

 On Mon, Aug 5, 2013 at 12:23 PM, Irfan Sayed wrote:

> hi,
>
> i have setup the two node apache hadoop cluster on windows environment
> one is namenode and another is datanode
>
> everything is working fine.
> one thing which i need to know is , how the replication starts
> if i create a.txt in namenode , how it will be appeared in datanodes
>
> please suggest
>
> regards
>
>

>>>
>>
>

-- 
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443
manishd...@gmail.com

Re: about replication

2013-08-05 Thread Irfan Sayed

thanks.
i verified, datanode is up and running

i ran the below command:

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal
C:\\Users\\Administrator\\Desktop\\hadoop-1.1.2.tar /wksp
copyFromLocal: File C:/Users/Administrator/Desktop/hadoop-1.1.2.tar does
not exist.

it says , file does not exist. as my cluster is windows based , i dont know
how dir path needs to be used
i am using cygwin for linux style formatting

i tried below as well :

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.

please suggest

regards



On Mon, Aug 5, 2013 at 6:00 PM, manish dunani  wrote:

> You can not physically access the datanode.You have to understand it to
> logically and it really happens.
>
> Type "jps" command to check ur datanode was started or not.
>
> when user stores the file into hdfs ,the request is goes to datanode and
> datanode will divide the file into number of blocks.
> Each block size is 64mb or 128mb.but,default is 64 mb.
>
> The default replication factor is 1 that was u already set in ur
> hdfs-site.xml .If u want to change ur replication factor from 1 to 3 for
> particular file or particular directory in hdfs then use the below commands
> appropriately.
>
> First load any file to hdfs using command:
>
> bin/hadoop  dfs -copyFromLocal /ur/local/directory/path/to/file
>  /ur/hdfs/directory/path.
>
> *commands:*
>
> To set replication of an individual file to 4:
>
> ./bin/hadoop dfs -setrep -w 4 /path/to/file
>
> You can also do this recursively. To change replication of entire HDFS to
> 1:
>
> ./bin/hadoop dfs -setrep -R -w 1 /
> *
> *
>  If the replication factor is 3 then data will divided into 3 blocks and
> replicated to various data nodes of ur production cluster(multi node).
>
> Here Replication means same file copied three times on data nodes for
> handling hardware failure.
>
>
> On Mon, Aug 5, 2013 at 5:29 PM, Irfan Sayed  wrote:
>
>> thanks.
>> please refer below:
>>
>> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>> $ ./hadoop dfs -ls /wksp
>> Found 1 items
>> drwxr-xr-x   - Administrator Domain  0 2013-08-05 16:58 /wksp/New
>> folder
>>
>> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>> $
>>
>>
>> same command if i run on the datanode then it says:
>>
>> Administrator@DFS-1 /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>> $ ./hadoop dfs -ls /wksp
>> ls: Cannot access /wksp: No such file or directory.
>>
>> does it mean that replication is not yet started ...?
>>
>> please suggest
>>
>> regards
>>
>>
>>
>> On Mon, Aug 5, 2013 at 5:18 PM, Mohammad Tariq wrote:
>>
>>> You cannot physically see the HDFS files and directories through local
>>> FS. Either use HDFS shell or HDFS webUI(namenode_machine:50070).
>>>
>>> Warm Regards,
>>> Tariq
>>> cloudfront.blogspot.com
>>>
>>>
>>> On Mon, Aug 5, 2013 at 4:46 PM, Irfan Sayed wrote:
>>>
 thanks mohammad
 i ran the below command on NameNode

 $ ./hadoop dfs -mkdir /wksp

 and the "wksp" dir got created in c:\ ( as i have windows environment)

 now , when i log in to one of the DataNode , then i am not able to see
 c:\wksp

 any issue ?
 please suggest

 regards





 On Mon, Aug 5, 2013 at 3:03 PM, Mohammad Tariq wrote:

> Hello Irfan,
>
> You can find all the answers from HDFS architecture 
> guide.
> See the section Data 
> Organizationin
>  particular for this question.
>
> Warm Regards,
> Tariq
> cloudfront.blogspot.com
>
>
> On Mon, Aug 5, 2013 at 12:23 PM, Irfan Sayed wrote:
>
>> hi,
>>
>> i have setup the two node apache hadoop cluster on windows environment
>> one is namenode and another is datanode
>>
>> everything is working fine.
>> one thing which i need to know is , how the replication starts
>> if i create a.txt in namenode , how it will be appeared in datanodes
>>
>> please suggest
>>
>> regards
>>
>>
>

>>>
>>
>
>
> --
> MANISH DUNANI
> -THANX
> +91 9426881954,+91 8460656443
> manishd...@gmail.com
>

Re: about replication

2013-08-05 Thread manish dunani

hello,

I think u r newbie.You need to work on to learn from scratch.
Please do not mind.whatever u try to think hadoop should be like this, ur
concept is totally wrong.
but,your effort is positive.

My suggestion is try to learn on ubuntu os.Windows is not good enough for
hadoop.

Follow the link to setup single node cluster on ubuntu.

http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/

If u stop anywhere and do not get idea what to do next? just revert me .

Also read this:

https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxtYW5pc2hkdW5hbml8Z3g6NGRiM2JhODhhOGEzMTcyYw


*Thanks & Regards*
*
*
*Manish dunani*



On Tue, Aug 6, 2013 at 10:29 AM, Irfan Sayed  wrote:

> thanks.
> i verified, datanode is up and running
>
> i ran the below command:
>
> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
> $ ./hadoop dfs -copyFromLocal
> C:\\Users\\Administrator\\Desktop\\hadoop-1.1.2.tar /wksp
> copyFromLocal: File C:/Users/Administrator/Desktop/hadoop-1.1.2.tar does
> not exist.
>
> it says , file does not exist. as my cluster is windows based , i dont
> know how dir path needs to be used
> i am using cygwin for linux style formatting
>
> i tried below as well :
>
> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
> $ ./hadoop dfs -copyFromLocal
> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
> copyFromLocal: File
> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
>
> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
> $ ./hadoop dfs -copyFromLocal
> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
> copyFromLocal: File
> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
>
> please suggest
>
> regards
>
>
>
> On Mon, Aug 5, 2013 at 6:00 PM, manish dunani wrote:
>
>> You can not physically access the datanode.You have to understand it to
>> logically and it really happens.
>>
>> Type "jps" command to check ur datanode was started or not.
>>
>> when user stores the file into hdfs ,the request is goes to datanode and
>> datanode will divide the file into number of blocks.
>> Each block size is 64mb or 128mb.but,default is 64 mb.
>>
>> The default replication factor is 1 that was u already set in ur
>> hdfs-site.xml .If u want to change ur replication factor from 1 to 3 for
>> particular file or particular directory in hdfs then use the below commands
>> appropriately.
>>
>> First load any file to hdfs using command:
>>
>> bin/hadoop  dfs -copyFromLocal /ur/local/directory/path/to/file
>>  /ur/hdfs/directory/path.
>>
>> *commands:*
>>
>> To set replication of an individual file to 4:
>>
>> ./bin/hadoop dfs -setrep -w 4 /path/to/file
>>
>> You can also do this recursively. To change replication of entire HDFS to
>> 1:
>>
>> ./bin/hadoop dfs -setrep -R -w 1 /
>> *
>> *
>>  If the replication factor is 3 then data will divided into 3 blocks and
>> replicated to various data nodes of ur production cluster(multi node).
>>
>> Here Replication means same file copied three times on data nodes for
>> handling hardware failure.
>>
>>
>> On Mon, Aug 5, 2013 at 5:29 PM, Irfan Sayed  wrote:
>>
>>> thanks.
>>> please refer below:
>>>
>>> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>> $ ./hadoop dfs -ls /wksp
>>> Found 1 items
>>> drwxr-xr-x   - Administrator Domain  0 2013-08-05 16:58
>>> /wksp/New folder
>>>
>>> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>> $
>>>
>>>
>>> same command if i run on the datanode then it says:
>>>
>>>  Administrator@DFS-1 /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>> $ ./hadoop dfs -ls /wksp
>>> ls: Cannot access /wksp: No such file or directory.
>>>
>>> does it mean that replication is not yet started ...?
>>>
>>> please suggest
>>>
>>> regards
>>>
>>>
>>>
>>> On Mon, Aug 5, 2013 at 5:18 PM, Mohammad Tariq wrote:
>>>
 You cannot physically see the HDFS files and directories through local
 FS. Either use HDFS shell or HDFS webUI(namenode_machine:50070).

 Warm Regards,
 Tariq
 cloudfront.blogspot.com


 On Mon, Aug 5, 2013 at 4:46 PM, Irfan Sayed wrote:

> thanks mohammad
> i ran the below command on NameNode
>
> $ ./hadoop dfs -mkdir /wksp
>
> and the "wksp" dir got created in c:\ ( as i have windows environment)
>
> now , when i log in to one of the DataNode , then i am not able to see
> c:\wksp
>
> any issue ?
> please suggest
>
> regards
>
>
>
>
>
> On Mon, Aug 5, 2013 at 3:03 PM, Mohammad Tariq wrote:
>
>> Hello Irfan,
>>
>> You can find all the answers from HDFS architecture 
>> guide.
>> See the section Data 
>> Organizationin
>>  particular for this question.
>>
>> Warm Regards,
>>

Re: about replication

2013-08-05 Thread Irfan Sayed

thanks. yes , i am newbie.
however, i need windows setup.

let me surely refer the doc and link which u sent but i need this to be
working ...
can you please help

regards

Re: about replication

2013-08-05 Thread Irfan Sayed

hi,
i have gone high level through the doc and seems very promising. really
nice
but can you please help me on this issue .

i need to do PoC first and get the demo out

regards

On Tue, Aug 6, 2013 at 11:49 AM, Irfan Sayed  wrote:

> thanks. yes , i am newbie.
> however, i need windows setup.
>
> let me surely refer the doc and link which u sent but i need this to be
> working ...
> can you please help
>
> regards
>
>
>
>

Re: about replication

2013-08-06 Thread manish dunani

*You are wrong at this:*

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.

Because,You had wrote both the paths local and You need not to copy hadoop
into hdfs...Hadoop is already working..

Just check out in browser by after starting ur single node cluster :

localhost:50070

then go for browse the filesystem link in it..

If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop there).beacause u are
going to do processing on that data in text file.That's why hadoop is used
for ,first u need to make it clear in ur mind.Then and then u will do
it...otherwise not possible..

*Try this: *

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file
/hdfs/directory/path

On Tue, Aug 6, 2013 at 11:49 AM, Irfan Sayed  wrote:

> thanks. yes , i am newbie.
> however, i need windows setup.
>
> let me surely refer the doc and link which u sent but i need this to be
> working ...
> can you please help
>
> regards
>
>
>
>

-- 
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443
manishd...@gmail.com

Re: about replication

Re: about replication

Re: about replication

Re: about replication

Re: about replication

Re: about replication

Re: about replication

Re: about replication

Re: about replication

Re: about replication

10 matches

Site Navigation

Mail list logo

Footer information