Re: Sync and Data Replication
Hi Mohit, In this scenario is data also replicated as defined by the replication factor to other nodes as well? I am wondering if at this point if crash occurs do I have data in other nodes? What kind of crash are you talking about here? A client crash or a cluster crash? If a cluster, is the loss you're thinking of one DN or all the replicating DNs? If client fails to close a file due to a crash, it is auto-closed later (default is one hour) by the NameNode and whatever the client successfully wrote (i.e. into its last block) is then made available to readers at that point. If the client synced, then its last sync point is always available to readers and whatever it didn't sync is made available when the file is closed later by the NN. For DN failures, read on. Replication in 1.x/0.20.x is done via pipelines. Its done regardless of sync() calls. All write packets are indeed sent to and acknowledged by each DN in the constructed pipeline as the write progresses. For a good diagram on the sequence here, see Figure 3.3 | Page 66 | Chapter 3: The Hadoop Distributed Filesystem, in Tom's Hadoop: The Definitive Guide (2nd ed. page nos. Gotta get 3rd ed. soon :)) The sync behavior is further explained under the 'Coherency Model' title at Page 68 | Chapter 3: The Hadoop Distributed Filesystem of the same book. Think of sync() more as a checkpoint done over the write pipeline, such that new readers can read the length of synced bytes immediately and that they are guaranteed to be outside of the DN application (JVM) buffers (i.e. flushed). Some further notes, for general info: In 0.20.x/1.x releases, there's no hard-guarantee that the write buffer flushing done via sync ensures the data went to the *disk*. It may remain in the OS buffers (a feature in OSes, for performance). This is cause we do not do an fsync() (i.e. calling force on the FileChannel for the block and metadata outputs), but rather just an output stream flush. In the future, via 2.0.1-alpha release (soon to come at this point) and onwards, the specific call hsync() will ensure that this is not the case. However, if you are OK with the OS buffers feature/caveat and primarily need syncing not for reliability but for readers, you may use the call hflush() and save on performance. One place where hsync() is to be preferred instead of hflush() is where you use WALs (for data reliability), and HBase is one such application. With hsync(), HBase can survive potential failures caused by major power failure cases (among others). Let us know if this clears it up for you! On Sat, Jun 9, 2012 at 4:58 AM, Mohit Anchlia mohitanch...@gmail.com wrote: I am wondering the role of sync in replication of data to other nodes. Say client writes a line to a file in Hadoop, at this point file handle is open and sync has not been called. In this scenario is data also replicated as defined by the replication factor to other nodes as well? I am wondering if at this point if crash occurs do I have data in other nodes? -- Harsh J
Re: memory usage tasks
Hey Koert, Do you have JVM reuse enabled? It may be a bug related to that. If its still around, can you jstack the idle JVMs (i.e. the JVMs that lie around when no jobs are running) and attach them somewhere to look at their state? You'll need to run jstack on their PID as the mapred user (sudo -u mapred jstack PID jstack.PID.txt usually does it nicely). On Sat, Jun 9, 2012 at 6:19 AM, Koert Kuipers ko...@tresata.com wrote: silly question, but i have our hadoop slave boxes configured with 7 mappers each, yet i see java 14 process for user mapred on each box. and each process takes up about 2GB, which is equals to my memory allocation (mapred.child.java.opts=-Xmx2048m). so it is using twice as much memory as i expected! why is that? Thanks, -- Harsh J
Re: hbase client security (cluster is secure)
Hi again Tony,
Moving this to u...@hbase.apache.org (bcc'd
common-user@hadoop.apache.org). Please use the right user group lists
for best responses. I've added you to CC in case you aren't subscribed
to the HBase user lists.
Can you share the whole error/stacktrace-if-any/logs you get at the
HMaster that says AccessControlException? Would be helpful to see what
particular class/operation logged it to help you specifically.
I have an instance of 0.92-based cluster running after having followed
http://hbase.apache.org/book.html#zookeeper and
https://ccp.cloudera.com/display/CDH4DOC/HBase+Security+Configuration
and it seems to work well enough with auth enabled.
On Sat, Jun 9, 2012 at 3:41 AM, Tony Dean tony.d...@sas.com wrote:
Hi all,
I have created a hadoop/hbase/zookeeper cluster that is secured and verified.
Now a simple test is to connect an hbase client (e.g, shell) to see its
behavior.
Well, I get the following message on the hbase master:
AccessControlException: authentication is required.
Looking at the code it appears that the client passed simple authentication
byte in the rpc header. Why, I don't know?
My client configuration is as follows:
hbase-site.xml:
property
namehbase.security.authentication/name
valuekerberos/value
/property
property
namehbase.rpc.engine/name
valueorg.apache.hadoop.hbase.ipc.SecureRpcEngine/value
/property
hbase-env.sh:
export HBASE_OPTS=$HBASE_OPTS
-Djava.security.auth.login.config=/usr/local/hadoop/hbase/conf/hbase.jaas
hbase.jaas:
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=false
useTicketCache=true
};
I issue kinit for the client I want to use. Then invoke hbase shell. I
simply issue list and see the error on the server.
Any ideas what I am doing wrong?
Thanks so much!
_
From: Tony Dean
Sent: Tuesday, June 05, 2012 5:41 PM
To: common-user@hadoop.apache.org
Subject: hadoop file permission 1.0.3 (security)
Can someone detail the options that are available to set file permissions at
the hadoop and os level? Here's what I have discovered thus far:
dfs.permissions = true|false (works as advertised)
dfs.supergroup = supergroup (works as advertised)
dfs.umaskmode = umask (I believe this should be used in lieu of dfs.umask) -
it appears to set the permissions for files created in hadoop fs (minus
execute permission).
why was dffs.umask deprecated? what's difference between the 2.
dfs.datanode.data.dir.perm = perm (not sure this is working at all?) I
thought it was supposed to set permission on blks at the os level.
Are there any other file permission configuration properties?
What I would really like to do is set data blk file permissions at the os
level so that the blocks can be locked down from all users except super and
supergroup, but allow it to be used accessed by hadoop API as specified by
hdfs permissions. Is this possible?
Thanks.
Tony Dean
SAS Institute Inc.
Senior Software Developer
919-531-6704
OLE Object: Picture (Device Independent Bitmap)
--
Harsh J
Re: Sync and Data Replication
Thanks Harsh for detailed info. It clears things up. Only thing from those page is concerning is what happens when client crashes. It says you could lose upto a block worth of information. Is this still true given that NN would auto close the file? Is it a good practice to reduce NN default value so that it auto-closes before 1 hr. Regarding OS cache, I think it should be ok since chances of loosing replica nodes all at the same time is low. On Sat, Jun 9, 2012 at 5:13 AM, Harsh J ha...@cloudera.com wrote: Hi Mohit, In this scenario is data also replicated as defined by the replication factor to other nodes as well? I am wondering if at this point if crash occurs do I have data in other nodes? What kind of crash are you talking about here? A client crash or a cluster crash? If a cluster, is the loss you're thinking of one DN or all the replicating DNs? If client fails to close a file due to a crash, it is auto-closed later (default is one hour) by the NameNode and whatever the client successfully wrote (i.e. into its last block) is then made available to readers at that point. If the client synced, then its last sync point is always available to readers and whatever it didn't sync is made available when the file is closed later by the NN. For DN failures, read on. Replication in 1.x/0.20.x is done via pipelines. Its done regardless of sync() calls. All write packets are indeed sent to and acknowledged by each DN in the constructed pipeline as the write progresses. For a good diagram on the sequence here, see Figure 3.3 | Page 66 | Chapter 3: The Hadoop Distributed Filesystem, in Tom's Hadoop: The Definitive Guide (2nd ed. page nos. Gotta get 3rd ed. soon :)) The sync behavior is further explained under the 'Coherency Model' title at Page 68 | Chapter 3: The Hadoop Distributed Filesystem of the same book. Think of sync() more as a checkpoint done over the write pipeline, such that new readers can read the length of synced bytes immediately and that they are guaranteed to be outside of the DN application (JVM) buffers (i.e. flushed). Some further notes, for general info: In 0.20.x/1.x releases, there's no hard-guarantee that the write buffer flushing done via sync ensures the data went to the *disk*. It may remain in the OS buffers (a feature in OSes, for performance). This is cause we do not do an fsync() (i.e. calling force on the FileChannel for the block and metadata outputs), but rather just an output stream flush. In the future, via 2.0.1-alpha release (soon to come at this point) and onwards, the specific call hsync() will ensure that this is not the case. However, if you are OK with the OS buffers feature/caveat and primarily need syncing not for reliability but for readers, you may use the call hflush() and save on performance. One place where hsync() is to be preferred instead of hflush() is where you use WALs (for data reliability), and HBase is one such application. With hsync(), HBase can survive potential failures caused by major power failure cases (among others). Let us know if this clears it up for you! On Sat, Jun 9, 2012 at 4:58 AM, Mohit Anchlia mohitanch...@gmail.com wrote: I am wondering the role of sync in replication of data to other nodes. Say client writes a line to a file in Hadoop, at this point file handle is open and sync has not been called. In this scenario is data also replicated as defined by the replication factor to other nodes as well? I am wondering if at this point if crash occurs do I have data in other nodes? -- Harsh J
Re: master and slaves are running but they seem disconnected
Not sure, but I did notice that safe mode is still. I would investigate that and see if the other nodes show up. /* Joey */ On Jun 9, 2012 2:52 PM, Pierre Antoine DuBoDeNa pad...@gmail.com wrote: Hello everyone.. I have a cluster of 5 VMs, 1 as master/slave the rest are slaves. I run bin/start-all.sh everything seems ok i get no errors.. I check with jps in all server they run: master: 22418 Jps 21497 NameNode 21886 SecondaryNameNode 21981 JobTracker 22175 TaskTracker 21688 DataNode slave: 3161 Jps 2953 DataNode 3105 TaskTracker But in the web interface i get only 1 server connected.. is like the others are ignored.. Any clue why this can happen? where to look for errors? The hdfs web interface: Live Nodes http://fusemaster.cs.columbia.edu:50070/dfsnodelist.jsp?whatNodes=LIVE : 1 Dead Nodes http://fusemaster.cs.columbia.edu:50070/dfsnodelist.jsp?whatNodes=DEAD : 0 it doesn't even show the rest slaves as dead.. can it be a networking issue? (but i start all processes from master and it starts all processes to all others). best, PA
Re: master and slaves are running but they seem disconnected
Please send the content of all the hosts file from all the machines. and master and slaves contents from all the machines from master and the slave machines. On Sun, Jun 10, 2012 at 1:39 AM, Joey Krabacher jkrabac...@gmail.comwrote: Not sure, but I did notice that safe mode is still. I would investigate that and see if the other nodes show up. /* Joey */ On Jun 9, 2012 2:52 PM, Pierre Antoine DuBoDeNa pad...@gmail.com wrote: Hello everyone.. I have a cluster of 5 VMs, 1 as master/slave the rest are slaves. I run bin/start-all.sh everything seems ok i get no errors.. I check with jps in all server they run: master: 22418 Jps 21497 NameNode 21886 SecondaryNameNode 21981 JobTracker 22175 TaskTracker 21688 DataNode slave: 3161 Jps 2953 DataNode 3105 TaskTracker But in the web interface i get only 1 server connected.. is like the others are ignored.. Any clue why this can happen? where to look for errors? The hdfs web interface: Live Nodes http://fusemaster.cs.columbia.edu:50070/dfsnodelist.jsp?whatNodes=LIVE : 1 Dead Nodes http://fusemaster.cs.columbia.edu:50070/dfsnodelist.jsp?whatNodes=DEAD : 0 it doesn't even show the rest slaves as dead.. can it be a networking issue? (but i start all processes from master and it starts all processes to all others). best, PA -- ∞ Shashwat Shriparv
Re: master and slaves are running but they seem disconnected
Oh i fixed the problem.. I changed for another application the hosts files to include localhost at 127.0.0.1 .. that seems to have destroyed everything.. Thanks for the quick responses guys. 2012/6/9 shashwat shriparv dwivedishash...@gmail.com Please send the content of all the hosts file from all the machines. and master and slaves contents from all the machines from master and the slave machines. On Sun, Jun 10, 2012 at 1:39 AM, Joey Krabacher jkrabac...@gmail.com wrote: Not sure, but I did notice that safe mode is still. I would investigate that and see if the other nodes show up. /* Joey */ On Jun 9, 2012 2:52 PM, Pierre Antoine DuBoDeNa pad...@gmail.com wrote: Hello everyone.. I have a cluster of 5 VMs, 1 as master/slave the rest are slaves. I run bin/start-all.sh everything seems ok i get no errors.. I check with jps in all server they run: master: 22418 Jps 21497 NameNode 21886 SecondaryNameNode 21981 JobTracker 22175 TaskTracker 21688 DataNode slave: 3161 Jps 2953 DataNode 3105 TaskTracker But in the web interface i get only 1 server connected.. is like the others are ignored.. Any clue why this can happen? where to look for errors? The hdfs web interface: Live Nodes http://fusemaster.cs.columbia.edu:50070/dfsnodelist.jsp?whatNodes=LIVE : 1 Dead Nodes http://fusemaster.cs.columbia.edu:50070/dfsnodelist.jsp?whatNodes=DEAD : 0 it doesn't even show the rest slaves as dead.. can it be a networking issue? (but i start all processes from master and it starts all processes to all others). best, PA -- ∞ Shashwat Shriparv
