I prefer to stay with Selva's opinion. If we have large matrix to support different Hadoop and Hbase version in ONE Trafodion release, the testing effort is also huge.
Br, Seth -----Original Message----- From: Qifan Chen [mailto:[email protected]] Sent: 2016年6月22日 1:25 To: dev <[email protected]> Subject: Re: discuss on support hbase 1.1 I personally would prefer that a common Trafodion code base to deal with different storage engines (HBase, Hive etc), in different versions. This will be a ideal situation for feature development and code maintenance. The initial requirement is that only one version of a storage engine per sqstart instance. Within SQL, it probably is a straight forward task to call javaObjectInterface::buildClassPath() method to load JVM with a specific class path to a specific JAR file for that version of the storage engine. 1. We can set CLASSPATH during install to point to /home/trafodion/0621a/hbase1.0/jars if asked to run against Hbase 1.0. 2. we can set CLASSPATH during install to point to /home/trafodion/0621a/hbase1.1/jars if asked to run against Hbase 1.1. Of course, within SQL, we need to distinguish calls to different versions of storage engines. Maybe hide all these under a layer of abstraction. --Qifan int JavaObjectInterface::createJVM() { JavaVMInitArgs jvm_args; JavaVMOption jvm_options[MAX_NO_JVM_OPTIONS]; char* classPathArg = buildClassPath(); int numJVMOptions = 0; jvm_options[numJVMOptions].optionString = classPathArg; QRLogger::log(CAT_SQL_HDFS_JNI_TOP, LL_DEBUG, "Using classpath: %s", jvm_options[numJVMOptions].optionString); numJVMOptions++; char maxHeapOptions[64]; bool passMaxHeapToJVM = true; int maxHeapEnvvarMB = DEFAULT_JVM_MAX_HEAP_SIZE; const char *maxHeapSizeStr = getenv("JVM_MAX_HEAP_SIZE_MB"); if (maxHeapSizeStr) { maxHeapEnvvarMB = atoi(maxHeapSizeStr); if (maxHeapEnvvarMB <= 0) passMaxHeapToJVM = false; } char* JavaObjectInterface::buildClassPath() { char* classPath = getenv("CLASSPATH"); Int32 size = strlen(classPath) + 128; char* classPathBuffer = (char*)malloc(size); strcpy(classPathBuffer, "-Djava.class.path="); if (isHBaseCompatibilityMode()) strcat(classPathBuffer, "/opt/home/tools/hbase-0.94.5/lib/hadoop-core-1.0.4.jar:"); strcat(classPathBuffer, classPath); return classPathBuffer; } On Tue, Jun 21, 2016 at 11:36 AM, Amanda Moran <[email protected]> wrote: > So, it narrows down the question of should Trafodion support both > Apache Hbase 1.0 and Apache Hbase 1.1 from the same code base. Or is > it ok to say Trafodion 2.0 version needs Apache hbase 1.0 and > Trafodion 2.1 version needs Apache hbase 1.1. > > +1 to this. > > It's always good to support has many versions as possible, but > sometimes it is just too difficult. > > On Tue, Jun 21, 2016 at 8:41 AM, Selva Govindarajan < > [email protected]> wrote: > > > My understanding is that install_features file lists the Trafodion > required > > products and its version. At the time of install, if the dependent > product > > listed in this file doesn't exist or the version below exists in the > > cluster, install will fail. > > > > With the change in sqenvcom.sh, you are determining which version of > > Trx-jar to be built for apache hbase version. This would mean, > > > > Change install_features to build Apache_1.0, build trx jar files > correspond > > to Apache hbase 1.0. And then change install_features to build > Apache_1.1, > > build trx jar files correspond to Apache hbase 1.1. > > > > Trafodion build process doesn't support this concept. > > > > So, it narrows down the question of should Trafodion support both > > Apache Hbase 1.0 and Apache Hbase 1.1 from the same code base. Or > > is it ok to say Trafodion 2.0 version needs Apache hbase 1.0 and > > Trafodion 2.1 version needs Apache hbase 1.1. > > > > My vote is for the latter option. > > > > If so, basically you just need to remove the change in sqenvcom.sh > > and rework it to build for Apache hbase 1.1. > > > > Selva > > > > -----Original Message----- > > From: Ma, Sheng-Chen (Aven) [mailto:[email protected]] > > Sent: Tuesday, June 21, 2016 2:43 AM > > To: [email protected] > > Subject: 答复: discuss on support hbase 1.1 > > > > Hi > > Seemingly it's not a good solution to use install_features to > > determine which trx file will use. I 'm try to add more trx files. > > So that install_features file will just indicate what trx files are > > available > > > > > > -----邮件原件----- > > 发件人: Ma, Sheng-Chen (Aven) [mailto:[email protected]] > > 发送时间: 2016年6月21日 13:15 > > 收件人: [email protected] > > 主题: 答复: discuss on support hbase 1.1 > > > > Quote Steve's view first > > "In other cases, we support multiple versions simultaneously by > generating > > multiple TRX files. I'm not sure we want to add more TRX files, but > > I'm also not sure we want to switch how the build works based on > > content of install_features file. > > > > Previously it works the other way round -- the install_features file > > indicates what TRX files are available, hence which distros are > supported." > > > > No more trx files will be added. > > Have to use install_features to decide whice hbase version will be > > used.(discussed in the previous msg) > > > > > > -----邮件原件----- > > 发件人: Selva Govindarajan [mailto:[email protected]] > > 发送时间: 2016年6月21日 12:51 > > 收件人: [email protected] > > 主题: RE: discuss on support hbase 1.1 > > > > I have the same observation as that of Steve for PR 503. Can you > > please address them? > > > > Selva > > > > -----Original Message----- > > From: Ma, Sheng-Chen (Aven) [mailto:[email protected]] > > Sent: Monday, June 20, 2016 9:00 PM > > To: [email protected] > > Subject: 答复: discuss on support hbase 1.1 > > > > Hi All: > > Can someone review #503 > > (https://github.com/apache/incubator-trafodion/pull/503 )? > > Since there is a long time ,and several time discussion shows it's > > the > only > > solution. > > > > Thanks > > > > -----邮件原件----- > > 发件人: Ma, Sheng-Chen (Aven) [mailto:[email protected]] > > 发送时间: 2016年6月16日 10:51 > > 收件人: [email protected] > > 主题: 答复: discuss on support hbase 1.1 > > > > Yes reflection is a good option, but I have do some research , it's > > hard > to > > use reflection, > > eg: TrxTransactionState.java need add @Override to next(..) for > > hbase > 1.1, > > and this I think can't use reflect #ifndef HDP2.3 HBASE1.1 > > @Override > > #endif > > public synchronized boolean next(final List<Cell> results) > > throws IOException { > > return next(results, -1); > > } > > > > Another is extends from new class, ScheduledChore is new class in > hbase1.1 > > while Chore is disappear in hbase 1.0 #ifdef HDP2.3 HBASE1.1 public > > class CleanOldTransactionsChore extends ScheduledChore { #else > > public class CleanOldTransactionsChore extends Chore { #endif > > > > And there still some kind of changes I can hardly use reflection... > > > > > > > > -----邮件原件----- > > 发件人: Selva Govindarajan [mailto:[email protected]] > > 发送时间: 2016年6月15日 22:36 > > 收件人: [email protected] > > 主题: RE: discuss on support hbase 1.1 > > > > Another option is to check the code where Trafodion issues different > > APIs and rework so that common set of API can be used. Reflection > > concept in java might also work in some cases. If the rework is not > > possible then it > might > > be necessary to stick with higher version. > > > > Selva > > > > -----Original Message----- > > From: Ma, Sheng-Chen (Aven) [mailto:[email protected]] > > Sent: Wednesday, June 15, 2016 2:14 AM > > To: [email protected] > > Subject: discuss on support hbase 1.1 > > > > Hi all: > > Apologize for this taking me so long... and thanks Amanda's reply > > (your reply gives me confidence :) > > > > Last time I had an inspiration to solve support hbase 1.1 without > > use install_features, but the truth is it's a misconception... > > > > I'd like to summary it first, according to the manual (see > > > > > https://cwiki.apache.org/confluence/display/TRAFODION/Build+Source#Bui > ldSource-makeall > > ) > > : > > 1. download source code. > > 2. source sqenvr.sh & make package-all > > 3. install Hadoop & hbase > > 4. install trafodion > > > > Base on the above steps, there is no idea knows which hbase version > > the user will install. I can't mandatory provisions install Hadoop > > & hbase before make trafodion package... > > So the only way is to predefine hbase version in a config file, so > > there appear the install_features. > > > > Another question is why not support both hbase 1.0 & 1.1 at the same > time? > > The answer is hbase1.0 &1.1 have different api , so I can > > only support one version at one package. > > Further question is why not package two package eg: > hbase-trx-apache1.0.jar > > & hbase-trx-apache1.1.jar > > It seems a solution, but I don't think it's a good way, > > reason is similar why there is only hbase-trx-cdh5_4.jar and > > hbase-trx-hdp2_3.jar. > > > > So I think this release the apache hbase ,we support 1.0...the next > release > > we support 1.1 > > ==================================================================== > > All above is my view, maybe not correct... if some one good at this > > pls correct. > > > > Thanks. > > > > > > -----邮件原件----- > > 发件人: Amanda Moran [mailto:[email protected]] > > 发送时间: 2016年6月4日 2:02 > > 收件人: dev <[email protected]> > > 抄送: Steve Varnau <[email protected]> > > 主题: Re: 答复: about pull 503 > > > > Hi there Aven- > > > > You can you install_features, actually you must so that the > > installer > knows > > what version of HBase is supported in that particular package. The > > installer can install all versions of trafodion. So there will need > > to be HBase 1.0 support in the installer and HBase 1.1 . > > install_features is used during installation not during sqenvcom.sh. > > Take a look at traf_config_check to see the details. > > > > Thanks. > > > > On Fri, Jun 3, 2016 at 1:01 AM, Ma, Sheng-Chen (Aven) < > > [email protected] > > > wrote: > > > > > hi Steve: > > > I think u are right . I will do some change which will not rely on > > > install_feature. Then I will have this discussion on dev list. > > > > > > Thanks > > > Aven > > > > > > > > > 发件人: Steve Varnau [mailto:[email protected]] > > > 发送时间: 2016年6月3日 6:40 > > > 收件人: Ma, Sheng-Chen (Aven) <[email protected]> > > > 抄送: Amanda Moran <[email protected]> > > > 主题: RE: about pull 503 > > > > > > Hi Aven, > > > > > > I think it would be really good to have this discussion on the dev > > > list or on github. Others may also be confused by my remarks or > > > may have different opinions than I do. Maybe I’m wrong. Either > > > way, it would be good record for next person trying to add support > > > of another HBase version. But I did not want to post your message > > > without permission, so I’ll reply directly. > > > > > > Your point #1 about installer script is a good one. But I‘d say > > > that the installer script could be enhanced to recognize which > > > version of HBase is there and then move only correct TRX file. > > > Either as part of your change or separately. > > > > > > If you are changing build to support apache hbase1.1, then why are > > > you trying to retain hbase1.0? If we want both supported, we > > > should be able to choose at install time not at build time. > > > > > > It is hbase-trx makefile that determines which TRX files are > > > built. If developr wants to switch hbase version, they should > > > change makefile, and maybe the install_features file too. But the > > > sqenvcom.sh file should not read the install_features file. Your > > > changes work differently than all the other versions of TRX. I > > > would prefer consistency. > > > > > > --Steve > > > > > > From: Ma, Sheng-Chen (Aven) [mailto:[email protected]<mailto: > > > [email protected]>] > > > Sent: Thursday, June 2, 2016 3:02 AM > > > To: Steve Varnau > > > <[email protected]<mailto:[email protected]>> > > > Subject: about pull 503 > > > > > > Hi Steve: > > > > > > I send this mail to discuss #503 ( > > > https://github.com/apache/incubator-trafodion/pull/503) > > > As your comment is a little confused me, I’d like to make sure > something. > > > > > > As u commented: > > > > > > In other cases, we support multiple versions simultaneously by > > > generating multiple TRX files. I'm not sure we want to add more > > > TRX files, but I'm also not sure we want to switch how the build > > > works based on content of install_features file. > > > > > > Previously it works the other way round -- the install_features > > > file indicates what TRX files are available, hence which distros > > > are > > supported. > > > > > > This change will not add new trx file ,it just change > > > hbase-trx-apache.jar from 1.0 to 1.1, It’s not good idea gen two > > > apache-trx.jar, because there are two reason, one is our install > > > script will add hbase-trx-apache*.jar to classpath, see > > > /incubator-trafodion/install/installer/traf_apache_mods > > > line51. Two is if 2 different version hbase-trx-apache*.jar added > > > to classpath, hbase will only use the first meet class in the jar > > > package, it may lead to conflict. > > > > > > About install_features file. I am not sure we want to switch how > > > the build works based on content of install_features file either. > > > But the only way I can distinguish which hbase version to use is > > > this file. If there is another place to distinguish this ,pls tell me > > > ,thanks. > > > > > > About your last sentence , I am really confuse, is that means we > > > indeed use install_features to distinguish hbase version? > > > > > > > > > Thanks, > > > Aven > > > > > > > > > > > > -- > > Thanks, > > > > Amanda Moran > > > > > > -- > Thanks, > > Amanda Moran > -- Regards, --Qifan
