Re: Read namenode data from primary memory
Hi Aditya: This command line may help you: hadoop dfsadmin --saveNamespace you should firstly perform it in safe mode. The corresponding design and implementation is described here: https://issues.apache.org/jira/browse/HADOOP-4826 yours, Ling Kun On Tue, May 14, 2013 at 2:37 PM, Aditya exalter wrote: > Hi, > > How to get the namenode data(edits and fsimage) from the main > memory(RAM) > > > > > -- http://www.lingcc.com
Re: Read namenode data from primary memory
Withuot going to safemode can't we get the Namespace ?? On Tue, May 14, 2013 at 12:18 PM, Ling Kun wrote: > Hi Aditya: >This command line may help you: > > hadoop dfsadmin --saveNamespace > > you should firstly perform it in safe mode. > > The corresponding design and implementation is described here: > https://issues.apache.org/jira/browse/HADOOP-4826 > > > yours, > Ling Kun > > > > On Tue, May 14, 2013 at 2:37 PM, Aditya exalter > wrote: > >> Hi, >> >> How to get the namenode data(edits and fsimage) from the main >> memory(RAM) >> >> >> >> >> > > > -- > http://www.lingcc.com > -- --Regards Sandeep Nemuri
Re: Read namenode data from primary memory
if you do not put namenode in safemode that means your namenode can still accept request to write or delete files from hdfs. This may cause your snapshot to be a problematic in case you want to restore from it. Safemode is just an ensuring point that you have a standstill image to snapshot. On Tue, May 14, 2013 at 12:27 PM, Sandeep Nemuri wrote: > Withuot going to safemode can't we get the Namespace ?? > > > On Tue, May 14, 2013 at 12:18 PM, Ling Kun wrote: > >> Hi Aditya: >>This command line may help you: >> >> hadoop dfsadmin --saveNamespace >> >> you should firstly perform it in safe mode. >> >> The corresponding design and implementation is described here: >> https://issues.apache.org/jira/browse/HADOOP-4826 >> >> >> yours, >> Ling Kun >> >> >> >> On Tue, May 14, 2013 at 2:37 PM, Aditya exalter >> wrote: >> >>> Hi, >>> >>> How to get the namenode data(edits and fsimage) from the main >>> memory(RAM) >>> >>> >>> >>> >>> >> >> >> -- >> http://www.lingcc.com >> > > > > -- > --Regards > Sandeep Nemuri > -- Nitin Pawar
Re: Read namenode data from primary memory
I Agree with you Nitin .. My Requirement is i want to copy the edits and fsimage from memory for every 5mins. How does this works ? On Tue, May 14, 2013 at 12:31 PM, Nitin Pawar wrote: > if you do not put namenode in safemode that means your namenode can still > accept request to write or delete files from hdfs. This may cause your > snapshot to be a problematic in case you want to restore from it. Safemode > is just an ensuring point that you have a standstill image to snapshot. > > > > > On Tue, May 14, 2013 at 12:27 PM, Sandeep Nemuri wrote: > >> Withuot going to safemode can't we get the Namespace ?? >> >> >> On Tue, May 14, 2013 at 12:18 PM, Ling Kun wrote: >> >>> Hi Aditya: >>>This command line may help you: >>> >>> hadoop dfsadmin --saveNamespace >>> >>> you should firstly perform it in safe mode. >>> >>> The corresponding design and implementation is described here: >>> https://issues.apache.org/jira/browse/HADOOP-4826 >>> >>> >>> yours, >>> Ling Kun >>> >>> >>> >>> On Tue, May 14, 2013 at 2:37 PM, Aditya exalter >> > wrote: >>> Hi, How to get the namenode data(edits and fsimage) from the main memory(RAM) >>> >>> >>> -- >>> http://www.lingcc.com >>> >> >> >> >> -- >> --Regards >> Sandeep Nemuri >> > > > > -- > Nitin Pawar > -- --Regards Sandeep Nemuri
Re: Read namenode data from primary memory
every 5 minutes seems too much. why not do a daily or say 8 hours basis On Tue, May 14, 2013 at 12:51 PM, Sandeep Nemuri wrote: > I Agree with you Nitin .. > My Requirement is i want to copy the edits and fsimage from memory for > every 5mins. How does this works ? > > > On Tue, May 14, 2013 at 12:31 PM, Nitin Pawar wrote: > >> if you do not put namenode in safemode that means your namenode can still >> accept request to write or delete files from hdfs. This may cause your >> snapshot to be a problematic in case you want to restore from it. Safemode >> is just an ensuring point that you have a standstill image to snapshot. >> >> >> >> >> On Tue, May 14, 2013 at 12:27 PM, Sandeep Nemuri wrote: >> >>> Withuot going to safemode can't we get the Namespace ?? >>> >>> >>> On Tue, May 14, 2013 at 12:18 PM, Ling Kun wrote: >>> Hi Aditya: This command line may help you: hadoop dfsadmin --saveNamespace you should firstly perform it in safe mode. The corresponding design and implementation is described here: https://issues.apache.org/jira/browse/HADOOP-4826 yours, Ling Kun On Tue, May 14, 2013 at 2:37 PM, Aditya exalter < adityaexal...@gmail.com> wrote: > Hi, > > How to get the namenode data(edits and fsimage) from the main > memory(RAM) > > > > > -- http://www.lingcc.com >>> >>> >>> >>> -- >>> --Regards >>> Sandeep Nemuri >>> >> >> >> >> -- >> Nitin Pawar >> > > > > -- > --Regards > Sandeep Nemuri > -- Nitin Pawar
Re: Read namenode data from primary memory
Done Lets say 8hrs !! now every 8hrs do we need to go into safemode ? On Tue, May 14, 2013 at 12:53 PM, Nitin Pawar wrote: > every 5 minutes seems too much. > > why not do a daily or say 8 hours basis > > > On Tue, May 14, 2013 at 12:51 PM, Sandeep Nemuri wrote: > >> I Agree with you Nitin .. >> My Requirement is i want to copy the edits and fsimage from memory for >> every 5mins. How does this works ? >> >> >> On Tue, May 14, 2013 at 12:31 PM, Nitin Pawar wrote: >> >>> if you do not put namenode in safemode that means your namenode can >>> still accept request to write or delete files from hdfs. This may cause >>> your snapshot to be a problematic in case you want to restore from it. >>> Safemode is just an ensuring point that you have a standstill image to >>> snapshot. >>> >>> >>> >>> >>> On Tue, May 14, 2013 at 12:27 PM, Sandeep Nemuri >>> wrote: >>> Withuot going to safemode can't we get the Namespace ?? On Tue, May 14, 2013 at 12:18 PM, Ling Kun wrote: > Hi Aditya: >This command line may help you: > > hadoop dfsadmin --saveNamespace > > you should firstly perform it in safe mode. > > The corresponding design and implementation is described here: > https://issues.apache.org/jira/browse/HADOOP-4826 > > > yours, > Ling Kun > > > > On Tue, May 14, 2013 at 2:37 PM, Aditya exalter < > adityaexal...@gmail.com> wrote: > >> Hi, >> >> How to get the namenode data(edits and fsimage) from the main >> memory(RAM) >> >> >> >> >> > > > -- > http://www.lingcc.com > -- --Regards Sandeep Nemuri >>> >>> >>> >>> -- >>> Nitin Pawar >>> >> >> >> >> -- >> --Regards >> Sandeep Nemuri >> > > > > -- > Nitin Pawar > -- --Regards Sandeep Nemuri
Re: Read namenode data from primary memory
What is the reason? Could you explain in more details, what do you want to accomplish? Leonid. On Tue, May 14, 2013 at 12:27 AM, Sandeep Nemuri wrote: > Done Lets say 8hrs !! > now every 8hrs do we need to go into safemode ? > > > On Tue, May 14, 2013 at 12:53 PM, Nitin Pawar wrote: > >> every 5 minutes seems too much. >> >> why not do a daily or say 8 hours basis >> >> >> On Tue, May 14, 2013 at 12:51 PM, Sandeep Nemuri wrote: >> >>> I Agree with you Nitin .. >>> My Requirement is i want to copy the edits and fsimage from memory for >>> every 5mins. How does this works ? >>> >>> >>> On Tue, May 14, 2013 at 12:31 PM, Nitin Pawar >>> wrote: >>> if you do not put namenode in safemode that means your namenode can still accept request to write or delete files from hdfs. This may cause your snapshot to be a problematic in case you want to restore from it. Safemode is just an ensuring point that you have a standstill image to snapshot. On Tue, May 14, 2013 at 12:27 PM, Sandeep Nemuri wrote: > Withuot going to safemode can't we get the Namespace ?? > > > On Tue, May 14, 2013 at 12:18 PM, Ling Kun wrote: > >> Hi Aditya: >>This command line may help you: >> >> hadoop dfsadmin --saveNamespace >> >> you should firstly perform it in safe mode. >> >> The corresponding design and implementation is described here: >> https://issues.apache.org/jira/browse/HADOOP-4826 >> >> >> yours, >> Ling Kun >> >> >> >> On Tue, May 14, 2013 at 2:37 PM, Aditya exalter < >> adityaexal...@gmail.com> wrote: >> >>> Hi, >>> >>> How to get the namenode data(edits and fsimage) from the main >>> memory(RAM) >>> >>> >>> >>> >>> >> >> >> -- >> http://www.lingcc.com >> > > > > -- > --Regards > Sandeep Nemuri > -- Nitin Pawar >>> >>> >>> >>> -- >>> --Regards >>> Sandeep Nemuri >>> >> >> >> >> -- >> Nitin Pawar >> > > > > -- > --Regards > Sandeep Nemuri >
Re: Read namenode data from primary memory
if you want to use savenamespace then yes. Normally to have a backup copy of the FSImage, previously there was a secondary namenode and now there is checkpoint NN. SNN was deprecated but later it was undeprecated the sole purpose of checkpoint NN is to have an rolling copy of the NN based on interval you configure. Checkpoint Node is the process which is actually responsible for creating periodic check points. It downloads fsimage and log edits from the active NameNode, merges them locally, and uploads the new image back to the active NameNode. It is also a good practice that you configure multiple directories for saving fsimage and editlogs on different disks. (From cloudera blog: By storing two copies of the edit log and FSImage, on two separate hard disks, a good system administrator can avoid bringing down the NameNode if one of those disks fails.) On Tue, May 14, 2013 at 12:57 PM, Sandeep Nemuri wrote: > Done Lets say 8hrs !! > now every 8hrs do we need to go into safemode ? > > > On Tue, May 14, 2013 at 12:53 PM, Nitin Pawar wrote: > >> every 5 minutes seems too much. >> >> why not do a daily or say 8 hours basis >> >> >> On Tue, May 14, 2013 at 12:51 PM, Sandeep Nemuri wrote: >> >>> I Agree with you Nitin .. >>> My Requirement is i want to copy the edits and fsimage from memory for >>> every 5mins. How does this works ? >>> >>> >>> On Tue, May 14, 2013 at 12:31 PM, Nitin Pawar >>> wrote: >>> if you do not put namenode in safemode that means your namenode can still accept request to write or delete files from hdfs. This may cause your snapshot to be a problematic in case you want to restore from it. Safemode is just an ensuring point that you have a standstill image to snapshot. On Tue, May 14, 2013 at 12:27 PM, Sandeep Nemuri wrote: > Withuot going to safemode can't we get the Namespace ?? > > > On Tue, May 14, 2013 at 12:18 PM, Ling Kun wrote: > >> Hi Aditya: >>This command line may help you: >> >> hadoop dfsadmin --saveNamespace >> >> you should firstly perform it in safe mode. >> >> The corresponding design and implementation is described here: >> https://issues.apache.org/jira/browse/HADOOP-4826 >> >> >> yours, >> Ling Kun >> >> >> >> On Tue, May 14, 2013 at 2:37 PM, Aditya exalter < >> adityaexal...@gmail.com> wrote: >> >>> Hi, >>> >>> How to get the namenode data(edits and fsimage) from the main >>> memory(RAM) >>> >>> >>> >>> >>> >> >> >> -- >> http://www.lingcc.com >> > > > > -- > --Regards > Sandeep Nemuri > -- Nitin Pawar >>> >>> >>> >>> -- >>> --Regards >>> Sandeep Nemuri >>> >> >> >> >> -- >> Nitin Pawar >> > > > > -- > --Regards > Sandeep Nemuri > -- Nitin Pawar
Re: Read namenode data from primary memory
If you're looking to tail the metadata changes for analysis purposes, there's a more pluggable approach available in 2.x versions. If you're looking to backup the metadata outside of NameNode, I'd suggest ditching such an approach and using 2.x's HA HDFS mode instead. On Tue, May 14, 2013 at 12:07 PM, Aditya exalter wrote: > Hi, > > How to get the namenode data(edits and fsimage) from the main > memory(RAM) > > > > -- Harsh J
Re: Read namenode data from primary memory
Nitin, You have the whole bunch of problems to answer/resolve Let me have some research first, then we can discuss the problem... Thank you ! Leonid On Tue, May 14, 2013 at 12:43 AM, Nitin Pawar wrote: > if you want to use savenamespace then yes. > > Normally to have a backup copy of the FSImage, previously there was a > secondary namenode and now there is checkpoint NN. SNN was deprecated but > later it was undeprecated > > the sole purpose of checkpoint NN is to have an rolling copy of the NN > based on interval you configure. > Checkpoint Node is the process which is actually responsible for creating > periodic check points. It downloads fsimage and log edits from the active > NameNode, merges them locally, and uploads the new image back to the active > NameNode. > > It is also a good practice that you configure multiple directories for > saving fsimage and editlogs on different disks. (From cloudera blog: By > storing two copies of the edit log and FSImage, on two separate hard disks, > a good system administrator can avoid bringing down the NameNode if one of > those disks fails.) > > > > > On Tue, May 14, 2013 at 12:57 PM, Sandeep Nemuri wrote: > >> Done Lets say 8hrs !! >> now every 8hrs do we need to go into safemode ? >> >> >> On Tue, May 14, 2013 at 12:53 PM, Nitin Pawar wrote: >> >>> every 5 minutes seems too much. >>> >>> why not do a daily or say 8 hours basis >>> >>> >>> On Tue, May 14, 2013 at 12:51 PM, Sandeep Nemuri >>> wrote: >>> I Agree with you Nitin .. My Requirement is i want to copy the edits and fsimage from memory for every 5mins. How does this works ? On Tue, May 14, 2013 at 12:31 PM, Nitin Pawar wrote: > if you do not put namenode in safemode that means your namenode can > still accept request to write or delete files from hdfs. This may cause > your snapshot to be a problematic in case you want to restore from it. > Safemode is just an ensuring point that you have a standstill image to > snapshot. > > > > > On Tue, May 14, 2013 at 12:27 PM, Sandeep Nemuri > wrote: > >> Withuot going to safemode can't we get the Namespace ?? >> >> >> On Tue, May 14, 2013 at 12:18 PM, Ling Kun wrote: >> >>> Hi Aditya: >>>This command line may help you: >>> >>> hadoop dfsadmin --saveNamespace >>> >>> you should firstly perform it in safe mode. >>> >>> The corresponding design and implementation is described here: >>> https://issues.apache.org/jira/browse/HADOOP-4826 >>> >>> >>> yours, >>> Ling Kun >>> >>> >>> >>> On Tue, May 14, 2013 at 2:37 PM, Aditya exalter < >>> adityaexal...@gmail.com> wrote: >>> Hi, How to get the namenode data(edits and fsimage) from the main memory(RAM) >>> >>> >>> -- >>> http://www.lingcc.com >>> >> >> >> >> -- >> --Regards >> Sandeep Nemuri >> > > > > -- > Nitin Pawar > -- --Regards Sandeep Nemuri >>> >>> >>> >>> -- >>> Nitin Pawar >>> >> >> >> >> -- >> --Regards >> Sandeep Nemuri >> > > > > -- > Nitin Pawar >
