+1. It sounds to me that the installer should default to an end-user mode where it installs as little as possible and then have a command-line argument for developer mode installing whatever is needed for debugging?
Thanks, Gunnar On Wed, Feb 10, 2016 at 11:05 AM, Amanda Moran <amanda.mo...@esgyn.com> wrote: > THANK YOU MING! And during your holidays too!!! Gung hay fat choy! > > Really appreciate the explanation ( I am sure Gunnar does too!). > > > On Wed, Feb 10, 2016 at 5:07 AM, Liu, Ming (Ming) <ming....@esgyn.cn> > wrote: > >> Hi, all, >> >> >> >> Very glad we are discussing this. Although I think this is more suitable >> for the dev mail list J >> >> >> >> First reply to Amanda’s question , >> >> “hbase-site hbase.regionserver.region.transactional.tlog true “ is no >> longer need, you can remove it from traf_*modes for sure. >> >> I also strongly think “hbase-site hbase.bulkload.staging.dir >> /hbase-staging” is no longer required, but I cannot make 100% sure of it. >> Bulkloader expert should confirm it. The only reference seems like a dead >> code for me, HBulkLoadClient.java line 96. >> >> >> >> And, I have started a similar discussion for some other traf_*mods >> changes at Jira 1729/1730.. >> >> With TRAFODION-1729, there is no need for the Trafodion installer to >> change this specific hbase-site.xml setting : “hbase-site >> hbase.coprocessor.region.classes”. >> >> But we leave it there in installer, since there are still needs to have >> hbase-site.xml changed for some developers to easily change them for debug >> purpose. But I really think no end user ever need to understand it or even >> config it. And, there are still some serious concerns about this JIRA, so >> we keep current behavior unchanged until all people are confident about >> this change. J ( In fact, there is already a problem reported and solved >> by Dave due to this checkin…, but it is a minor issue comparing the problem >> it aims to solve), But technically, this item can now be removed from >> traf_*mod change list, Trafodion is smart enough now to add required >> coprocessor on the fly , if hbase-site.xml doesn’t have these settings. >> >> With TRAFODION-1730, which in fact solved by HBase community >> (HBASE-15076), it is also possible the installer no longer need to change >> the “hbase-site hbase.hregion.impl” settings soon. However, That HBase >> change is merged into HBase R1.2, so until Trafodion can support HBase >> R1.2, installer still need to change this setting item. And Trafodion DTM >> code need some change for this as well, which not done yet. For Gunnar’s >> documentation purpose, i.e. why Trafodion needs to change this item? I have >> a lengthy explanation at HBASE-15076, don’t want to paste that long text >> here. https://issues.apache.org/jira/browse/HBASE-15076?filter=-2 >> >> >> >> So sorry that although we are working on these two jiras, installer must >> still take care of them for now… And as Amanda listed, there are still more >> items unknown than known, so I really hope more people can join this >> discussion and we can finally remove all of them if possible, at least >> minimize the list. >> >> >> >> Although a long reply, the only item I came out for sure now is one can >> remove the “hbase.regionserver.region.transactional.tlog” from traf_*mod >> scripts... >> >> >> >> Thanks, >> >> Ming >> >> >> >> *发件人:* Amanda Moran [mailto:amanda.mo...@esgyn.com] >> *发送时间:* 2016年2月10日 1:43 >> *收件人:* user@trafodion.incubator.apache.org >> *主题:* Re: traf_*mods in trafodion installation >> >> >> >> Amanda: it should be possible and desireable to make all these >> configuration changes in advance. They're totally valid HBase configuration >> settings that could very-well exist in advance. >> >> >> >> [Amanda]: I would like to know the smallest number of these that must be >> set in order for Trafodion to be installer. Are all these settings >> mandatory for installation/start? This is what I want/need to know. Does a >> user have to go and set all those settings by hand in advance of >> installation Trafodion (assuming we 'remove' the traf_*mods scripts)? >> >> >> >> >> >> On Tue, Feb 9, 2016 at 9:32 AM, Gunnar Tapper <tapper.gun...@gmail.com> >> wrote: >> >> Amanda: it should be possible and desireable to make all these >> configuration changes in advance. They're totally valid HBase configuration >> settings that could very-well exist in advance. >> >> >> >> The worse situation is if the Trafodion settings clash with some setting >> that the customer already has for some other reason. For example, can we >> really change the staging setting without affecting something else? This is >> why I am documenting these settings as requirements in the upcoming >> Provisioning Guide with the reasoning behind the setting. >> >> >> >> Further, I really recommend that we separate Trafodion installation from >> configuration changes in the infrastructure to ensure that it's possible to >> change the configuration changes, test their impact, and THEN install >> Trafodion. The testing could consist of something as simple as verify the >> restart worked or as advanced as an application regression test. Very user >> specific and dependent what else HDFS and HBase is being used for. For >> example, what's the impact on the rest of the Hadoop environment if turning >> on HDFS ACLs? >> >> >> >> Thanks, >> >> >> >> Gunnar >> >> >> >> Gunnar >> >> >> >> On Tue, Feb 9, 2016 at 9:45 AM, Amanda Moran <amanda.mo...@esgyn.com> >> wrote: >> >> I agree with that :) Thanks Qifan! >> >> >> >> On Tue, Feb 9, 2016 at 8:43 AM, Qifan Chen <qifan.c...@esgyn.com> wrote: >> >> Hi Amanda, >> >> >> >> I know the topic that I brought up is a little bit different :-). >> >> >> >> But the objectives of both thoughts are pretty much in line with the >> improvement of install experiences. >> >> >> >> Thank you. >> >> >> >> --Qifan >> >> >> >> On Tue, Feb 9, 2016 at 10:36 AM, Amanda Moran <amanda.mo...@esgyn.com> >> wrote: >> >> Thanks for the response Qifan... but I am not sure we are talking about >> the same thing. >> >> >> >> I am talking about the traf_<hadoop distro>_mods98 script that is run as >> apart of the Trafodion Installation scripts. It is used to setup HBase >> settings, copy hbase-trx*.jar file, and restart HBase, HDFS, and Zookeeper. >> >> >> >> I would like to know what are the smallest number of settings that need >> to be set in advance for Trafodion to install/start. >> >> >> >> Examples of things set: >> >> hbase-site hbase.master.distributed.log.splitting false >> >> >> >> hbase-site hbase.coprocessor.region.classes >> "org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,org.apache.hadoop.hbase.coprocessor.AggregateImplementation" >> >> >> >> hbase-site hbase.hregion.impl >> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion >> >> >> >> hbase-site hbase.regionserver.region.split.policy >> org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy >> >> >> >> hbase-site hbase.snapshot.enabled true >> >> >> >> hbase-site hbase.bulkload.staging.dir /hbase-staging >> >> >> >> hbase-site hbase.regionserver.region.transactional.tlog true >> >> >> >> hbase-site hbase.snapshot.master.timeoutMillis 600000 >> >> >> >> hbase-site hbase.snapshot.region.timeout 600000 >> >> >> >> hdfs-site dfs.namenode.acls.enabled true >> >> >> >> hbase-site hbase.client.scanner.timeout.period 600000 >> >> >> >> Thanks all! >> >> >> >> On Thu, Feb 4, 2016 at 5:40 AM, Qifan Chen <qifan.c...@esgyn.com> wrote: >> >> Hi Amanda, >> >> >> >> I like your idea. >> >> >> >> On Trafodion's modification of config files for HBase, HDFS, and >> Zookeeper, I wonder if it is possible to re-difine the port range for an >> install, without going through all the install steps. This has the >> practical value that due to port conflict, an install could not start >> successfully, and one can easily switch to a different port range to retry >> with this capability. >> >> >> >> In addition, some of the install steps, such as TPCDS data set >> generation, and conversion to UTF8, could be made parallel to shorten the >> install duration. They are done one table at a time in serial mode now. >> >> >> >> TPCDS table data is generated through the command "dsdgen -scale ..." >> which could take the option "-parallel". >> >> Data conversion to UTF8 is done through the iconv utility. >> >> >> >> Regards, --Qifan >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Feb 3, 2016 at 10:34 PM, Amanda Moran <amanda.mo...@esgyn.com> >> wrote: >> >> HI there All- >> >> >> >> The traf_*_mods scripts are used during installation to set HBase, HDFS, >> and Zookeeper settings needed by Trafodion. >> >> >> >> The script takes a long time to run (slowing down installation by a lot) >> because it needs to restart all these services for the changes to be picked >> up. >> >> >> >> My question is: What if I wanted to set the settings in advance of >> running the installer. What would be the absolute minimum set of settings >> that I need to get installed? I could set other settings later with a >> script or by hand. >> >> >> >> Thanks! >> >> -- >> >> Thanks, >> >> >> >> Amanda Moran >> >> >> >> >> >> -- >> >> Regards, --Qifan >> >> >> >> >> >> >> >> -- >> >> Thanks, >> >> >> >> Amanda Moran >> >> >> >> >> >> -- >> >> Regards, --Qifan >> >> >> >> >> >> >> >> -- >> >> Thanks, >> >> >> >> Amanda Moran >> >> >> >> >> >> -- >> >> Thanks, >> >> >> >> Gunnar >> >> *If you think you can you can, if you think you can't you're right.* >> >> >> >> >> >> -- >> >> Thanks, >> >> >> >> Amanda Moran >> > > > > -- > Thanks, > > Amanda Moran > -- Thanks, Gunnar *If you think you can you can, if you think you can't you're right.*
