Hi Amanda,
I was a little surprised to see: Name: namenode_java_heapsize Value: 1GB (or 1073741824 bytes) Why: Still needed: I thought we discovered during some testing that doing this was actually causing the heapsize to be reduced from the 2GB on one system to 1GB? If that’s still true, then we definitely don’t want to continue to do that. Although I was not involved in the initial decision to add this setting, I would assume based on our experience that as the system grows having this set to too small of a value will cause the Name Node server to not continue to run if there are too many files/blocks. If that happens, the Name Node stops and basically Hadoop, HBase, and Trafodion are useless. I have the most experience using Cloudera Manager and I do think CM might try to initially set this value based on the cluster size, memory available, etc. but I’m not positive that is the case. Even if it is, the fact that Trafodion is installed after Cloudera Manager or Hortonworks probably means either will assume a smaller heapsize is adequate. Given Trafodion is intended to manage larger workloads, though it can handle smaller workloads too, the act of installing Trafodion probably does imply a certain minimum value for this setting. So I see several options: 1) The Installer queries the current setting, as long as it is 1GiB or larger, do nothing. If it is less than 1 GiB, then increase it to 1GiB and perhaps put out a message saying it was done—if possible. 2) Remove the current setting. We may already have documentation that suggests the need for increasing this value much like we do with the Region Server’s heapsize (at least we used to on the wiki). I would vote for #1 if possible. The Name Node is a vital component of the system and even if the original (smaller) heapsize works, as files and blocks are added the potential for an unexpected outage after restarting Hadoop grows. That’s exactly what happened to several of us. One minute the system is running fine, you decide to restart HDFS perhaps to change another setting, and the Name Node fails to come up because it encounters too many files/blocks while starting up. I would assume even without the restart, eventually as files/blocks are added it might outgrow the heap and cause an outage. Correcting the issue is simple enough, but it still takes time to determine that is the problem. Regards, Dennis From: Sean Broeder [mailto:sean.broe...@esgyn.com] Sent: Tuesday, May 03, 2016 12:12 PM To: user@trafodion.incubator.apache.org Subject: RE: 答复: HDFS/HBase/Zookeeper Settings set by Installer Hi Amanda, hbase.regionserver.region.transactional.tlog is a setting that was used prior to Trafodion becoming an open source project. It is no longer needed. Thanks, Sean From: Amanda Moran [mailto:amanda.mo...@esgyn.com <mailto:amanda.mo...@esgyn.com> ] Sent: Tuesday, May 3, 2016 10:02 AM To: user@trafodion.incubator.apache.org <mailto:user@trafodion.incubator.apache.org> Subject: Re: 答复: HDFS/HBase/Zookeeper Settings set by Installer Thanks a lot for the help Ming (and thanks for the kind words Pierre!). I will take a look at testing without adding the coprocessor settings (but I won't remove from the installer for now). Thanks! Amanda On Tue, May 3, 2016 at 12:42 AM, Liu, Ming (Ming) <ming....@esgyn.cn <mailto:ming....@esgyn.cn> > wrote: Yes, this is very important topic. I can confirm to remove two of them : Name: hbase.bulkload.staging.dir Value: /hbase-staging Why: Still Needed: NO Name: hbase.regionserver.region.transactional.tlog Value: true Why: Still Needed: NO We need more test for : Name: hbase_coprocessor_region_classes Value: "org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,org.apache.hadoop.hbase.coprocessor.AggregateImplementation" Why: Not fully tested to remove it. Still needed: Yes There is already a fix that can get rid of this settings. But we need more test. That is, remove this from installer and see if any issues. We tested it briefly and it works well, but never get it full QAed. Trafodion now should add coprocessor at runtime if installer not doing this. Another one I have comment: Name: hbase.hregion.impl Value: org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion Why: Trafodion coprocessor need to invoke a private API of HBase Region, so have to overwrite this class now. When Trafodion move to HBase 1.2, it is possible to get rid of it, since HBase R1.2 public one required method for us. See https://issues.apache.org/jira/browse/HBASE-15076?filter=-2 Still Needed: Yes. IMHO, all other settings should not be mandatory, but kind of performance/stability tunings. However, I am not the author of those settings, so not very sure. It is better others to comment more. Yes, thanks Amanda to bring this up, and I hope you can keep pushing this topic. I feel some settings there is too old that nobody knows the reason or just forget, I think testing is a solution: remove it, and test to see if any issue. But if someone knows that one is MUST-TO-KEEP or DELETE-IT, it is better to reply. Thanks, Ming 发件人: Pierre Smits [mailto:pierre.sm...@gmail.com <mailto:pierre.sm...@gmail.com> ] 发送时间: 2016年5月3日 15:20 收件人: user@trafodion.incubator.apache.org <mailto:user@trafodion.incubator.apache.org> 主题: Re: HDFS/HBase/Zookeeper Settings set by Installer Hi Amanda, Thanks for bringing this up. Especially the 'why' aspect. This will be good input for the documentation. Best regards, Pierre Smits ORRTIZ.COM <http://www.orrtiz.com> OFBiz based solutions & services OFBiz Extensions Marketplace http://oem.ofbizci.net/oci-2/ On Mon, May 2, 2016 at 10:54 PM, Amanda Moran <amanda.mo...@esgyn.com <mailto:amanda.mo...@esgyn.com> > wrote: Hi there All- I have been looking over the HDFS/HBase/Zookeeper settings that get set in the installer and I am wondering if they are all still needed. If you have requested a setting in the past, could you please add a description of why it is needed (and if it is still needed)? Thanks a bunch! **Note: I know this would have looked much better in a spreadsheet.... but I want to make sure everyone can see... and email is the best for that! HDFS Settings Name: namenode_java_heapsize Value: 1GB (or 1073741824 bytes) Why: Still needed: Name: secondary_namenode_java_heapsize Value: 1 GB (or 1073741824 bytes) Why: Still needed: Name: dfs_namenode_acls_enabled Value: true Why: Still needed: HBase Master Settings =HBase Master Config Safety Valve= Name: hbase_master_distributed_log_splitting Value: false Why: Still needed: Name: hbase_snapshot_master_timeoutMillis Value: 600000 Why: Still needed: HBase Region Server Settings Name: hbase_coprocessor_region_classes Value: "org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,org.apache.hadoop.hbase.coprocessor.AggregateImplementation" Why: Still needed: Name: hbase_regionserver_lease_period Value: 600000 Why: Still Needed: =HBase RegionServer Config Safety Valve= Name: hbase.hregion.impl Value: org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion Why: Still Needed: Name: hbase.regionserver.region.split.policy Value: org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy Why: Still Needed: Name: hbase.snapshot.enabled Value: true Why: Still Needed: Name: hbase.bulkload.staging.dir Value: /hbase-staging Why: Still Needed: Name: hbase.regionserver.region.transactional.tlog Value: true Why: Still Needed: Name: hbase.snapshot.region.timeout Value: 600000 Why: Still Needed: Zookeeper Settings Name: maxClientCnxns Value: 0 Why: Still Needed: -- Thanks, Amanda Moran -- Thanks, Amanda Moran