THANK YOU MING! And during your holidays too!!! Gung hay fat choy! Really appreciate the explanation ( I am sure Gunnar does too!).
On Wed, Feb 10, 2016 at 5:07 AM, Liu, Ming (Ming) <ming....@esgyn.cn> wrote: > Hi, all, > > > > Very glad we are discussing this. Although I think this is more suitable > for the dev mail list J > > > > First reply to Amanda’s question , > > “hbase-site hbase.regionserver.region.transactional.tlog true “ is no > longer need, you can remove it from traf_*modes for sure. > > I also strongly think “hbase-site hbase.bulkload.staging.dir > /hbase-staging” is no longer required, but I cannot make 100% sure of it. > Bulkloader expert should confirm it. The only reference seems like a dead > code for me, HBulkLoadClient.java line 96. > > > > And, I have started a similar discussion for some other traf_*mods changes > at Jira 1729/1730.. > > With TRAFODION-1729, there is no need for the Trafodion installer to > change this specific hbase-site.xml setting : “hbase-site > hbase.coprocessor.region.classes”. > > But we leave it there in installer, since there are still needs to have > hbase-site.xml changed for some developers to easily change them for debug > purpose. But I really think no end user ever need to understand it or even > config it. And, there are still some serious concerns about this JIRA, so > we keep current behavior unchanged until all people are confident about > this change. J ( In fact, there is already a problem reported and solved > by Dave due to this checkin…, but it is a minor issue comparing the problem > it aims to solve), But technically, this item can now be removed from > traf_*mod change list, Trafodion is smart enough now to add required > coprocessor on the fly , if hbase-site.xml doesn’t have these settings. > > With TRAFODION-1730, which in fact solved by HBase community > (HBASE-15076), it is also possible the installer no longer need to change > the “hbase-site hbase.hregion.impl” settings soon. However, That HBase > change is merged into HBase R1.2, so until Trafodion can support HBase > R1.2, installer still need to change this setting item. And Trafodion DTM > code need some change for this as well, which not done yet. For Gunnar’s > documentation purpose, i.e. why Trafodion needs to change this item? I have > a lengthy explanation at HBASE-15076, don’t want to paste that long text > here. https://issues.apache.org/jira/browse/HBASE-15076?filter=-2 > > > > So sorry that although we are working on these two jiras, installer must > still take care of them for now… And as Amanda listed, there are still more > items unknown than known, so I really hope more people can join this > discussion and we can finally remove all of them if possible, at least > minimize the list. > > > > Although a long reply, the only item I came out for sure now is one can > remove the “hbase.regionserver.region.transactional.tlog” from traf_*mod > scripts... > > > > Thanks, > > Ming > > > > *发件人:* Amanda Moran [mailto:amanda.mo...@esgyn.com] > *发送时间:* 2016年2月10日 1:43 > *收件人:* user@trafodion.incubator.apache.org > *主题:* Re: traf_*mods in trafodion installation > > > > Amanda: it should be possible and desireable to make all these > configuration changes in advance. They're totally valid HBase configuration > settings that could very-well exist in advance. > > > > [Amanda]: I would like to know the smallest number of these that must be > set in order for Trafodion to be installer. Are all these settings > mandatory for installation/start? This is what I want/need to know. Does a > user have to go and set all those settings by hand in advance of > installation Trafodion (assuming we 'remove' the traf_*mods scripts)? > > > > > > On Tue, Feb 9, 2016 at 9:32 AM, Gunnar Tapper <tapper.gun...@gmail.com> > wrote: > > Amanda: it should be possible and desireable to make all these > configuration changes in advance. They're totally valid HBase configuration > settings that could very-well exist in advance. > > > > The worse situation is if the Trafodion settings clash with some setting > that the customer already has for some other reason. For example, can we > really change the staging setting without affecting something else? This is > why I am documenting these settings as requirements in the upcoming > Provisioning Guide with the reasoning behind the setting. > > > > Further, I really recommend that we separate Trafodion installation from > configuration changes in the infrastructure to ensure that it's possible to > change the configuration changes, test their impact, and THEN install > Trafodion. The testing could consist of something as simple as verify the > restart worked or as advanced as an application regression test. Very user > specific and dependent what else HDFS and HBase is being used for. For > example, what's the impact on the rest of the Hadoop environment if turning > on HDFS ACLs? > > > > Thanks, > > > > Gunnar > > > > Gunnar > > > > On Tue, Feb 9, 2016 at 9:45 AM, Amanda Moran <amanda.mo...@esgyn.com> > wrote: > > I agree with that :) Thanks Qifan! > > > > On Tue, Feb 9, 2016 at 8:43 AM, Qifan Chen <qifan.c...@esgyn.com> wrote: > > Hi Amanda, > > > > I know the topic that I brought up is a little bit different :-). > > > > But the objectives of both thoughts are pretty much in line with the > improvement of install experiences. > > > > Thank you. > > > > --Qifan > > > > On Tue, Feb 9, 2016 at 10:36 AM, Amanda Moran <amanda.mo...@esgyn.com> > wrote: > > Thanks for the response Qifan... but I am not sure we are talking about > the same thing. > > > > I am talking about the traf_<hadoop distro>_mods98 script that is run as > apart of the Trafodion Installation scripts. It is used to setup HBase > settings, copy hbase-trx*.jar file, and restart HBase, HDFS, and Zookeeper. > > > > I would like to know what are the smallest number of settings that need to > be set in advance for Trafodion to install/start. > > > > Examples of things set: > > hbase-site hbase.master.distributed.log.splitting false > > > > hbase-site hbase.coprocessor.region.classes > "org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,org.apache.hadoop.hbase.coprocessor.AggregateImplementation" > > > > hbase-site hbase.hregion.impl > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion > > > > hbase-site hbase.regionserver.region.split.policy > org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy > > > > hbase-site hbase.snapshot.enabled true > > > > hbase-site hbase.bulkload.staging.dir /hbase-staging > > > > hbase-site hbase.regionserver.region.transactional.tlog true > > > > hbase-site hbase.snapshot.master.timeoutMillis 600000 > > > > hbase-site hbase.snapshot.region.timeout 600000 > > > > hdfs-site dfs.namenode.acls.enabled true > > > > hbase-site hbase.client.scanner.timeout.period 600000 > > > > Thanks all! > > > > On Thu, Feb 4, 2016 at 5:40 AM, Qifan Chen <qifan.c...@esgyn.com> wrote: > > Hi Amanda, > > > > I like your idea. > > > > On Trafodion's modification of config files for HBase, HDFS, and > Zookeeper, I wonder if it is possible to re-difine the port range for an > install, without going through all the install steps. This has the > practical value that due to port conflict, an install could not start > successfully, and one can easily switch to a different port range to retry > with this capability. > > > > In addition, some of the install steps, such as TPCDS data set generation, > and conversion to UTF8, could be made parallel to shorten the install > duration. They are done one table at a time in serial mode now. > > > > TPCDS table data is generated through the command "dsdgen -scale ..." > which could take the option "-parallel". > > Data conversion to UTF8 is done through the iconv utility. > > > > Regards, --Qifan > > > > > > > > > > > > > > > > On Wed, Feb 3, 2016 at 10:34 PM, Amanda Moran <amanda.mo...@esgyn.com> > wrote: > > HI there All- > > > > The traf_*_mods scripts are used during installation to set HBase, HDFS, > and Zookeeper settings needed by Trafodion. > > > > The script takes a long time to run (slowing down installation by a lot) > because it needs to restart all these services for the changes to be picked > up. > > > > My question is: What if I wanted to set the settings in advance of running > the installer. What would be the absolute minimum set of settings that I > need to get installed? I could set other settings later with a script or by > hand. > > > > Thanks! > > -- > > Thanks, > > > > Amanda Moran > > > > > > -- > > Regards, --Qifan > > > > > > > > -- > > Thanks, > > > > Amanda Moran > > > > > > -- > > Regards, --Qifan > > > > > > > > -- > > Thanks, > > > > Amanda Moran > > > > > > -- > > Thanks, > > > > Gunnar > > *If you think you can you can, if you think you can't you're right.* > > > > > > -- > > Thanks, > > > > Amanda Moran > -- Thanks, Amanda Moran
