I personally would prefer that a common Trafodion code base to deal with
different storage engines (HBase, Hive etc), in different versions. This
will be a ideal situation for feature development and code maintenance.
The initial requirement is that only one version of a storage engine per
sqstart instance.
Within SQL, it probably is a straight forward task to call
javaObjectInterface::buildClassPath() method to load JVM with a specific
class path to a specific JAR file for that version of the storage engine.
1. We can set CLASSPATH during install to point to
/home/trafodion/0621a/hbase1.0/jars if asked to run against Hbase 1.0.
2. we can set CLASSPATH during install to point to
/home/trafodion/0621a/hbase1.1/jars if asked to run against Hbase 1.1.
Of course, within SQL, we need to distinguish calls to different versions
of storage engines. Maybe hide all these under a layer of abstraction.
--Qifan
int JavaObjectInterface::createJVM()
{
JavaVMInitArgs jvm_args;
JavaVMOption jvm_options[MAX_NO_JVM_OPTIONS];
char* classPathArg = buildClassPath();
int numJVMOptions = 0;
jvm_options[numJVMOptions].optionString = classPathArg;
QRLogger::log(CAT_SQL_HDFS_JNI_TOP, LL_DEBUG, "Using classpath: %s",
jvm_options[numJVMOptions].optionString);
numJVMOptions++;
char maxHeapOptions[64];
bool passMaxHeapToJVM = true;
int maxHeapEnvvarMB = DEFAULT_JVM_MAX_HEAP_SIZE;
const char *maxHeapSizeStr = getenv("JVM_MAX_HEAP_SIZE_MB");
if (maxHeapSizeStr)
{
maxHeapEnvvarMB = atoi(maxHeapSizeStr);
if (maxHeapEnvvarMB <= 0)
passMaxHeapToJVM = false;
}
char* JavaObjectInterface::buildClassPath()
{
char* classPath = getenv("CLASSPATH");
Int32 size = strlen(classPath) + 128;
char* classPathBuffer = (char*)malloc(size);
strcpy(classPathBuffer, "-Djava.class.path=");
if (isHBaseCompatibilityMode())
strcat(classPathBuffer,
"/opt/home/tools/hbase-0.94.5/lib/hadoop-core-1.0.4.jar:");
strcat(classPathBuffer, classPath);
return classPathBuffer;
}
On Tue, Jun 21, 2016 at 11:36 AM, Amanda Moran <[email protected]>
wrote:
> So, it narrows down the question of should Trafodion support both Apache
> Hbase 1.0 and Apache Hbase 1.1 from the same code base. Or is it ok to
> say
> Trafodion 2.0 version needs Apache hbase 1.0 and Trafodion 2.1 version
> needs
> Apache hbase 1.1.
>
> +1 to this.
>
> It's always good to support has many versions as possible, but sometimes it
> is just too difficult.
>
> On Tue, Jun 21, 2016 at 8:41 AM, Selva Govindarajan <
> [email protected]> wrote:
>
> > My understanding is that install_features file lists the Trafodion
> required
> > products and its version. At the time of install, if the dependent
> product
> > listed in this file doesn't exist or the version below exists in the
> > cluster, install will fail.
> >
> > With the change in sqenvcom.sh, you are determining which version of
> > Trx-jar
> > to be built for apache hbase version. This would mean,
> >
> > Change install_features to build Apache_1.0, build trx jar files
> correspond
> > to Apache hbase 1.0. And then change install_features to build
> Apache_1.1,
> > build trx jar files correspond to Apache hbase 1.1.
> >
> > Trafodion build process doesn't support this concept.
> >
> > So, it narrows down the question of should Trafodion support both Apache
> > Hbase 1.0 and Apache Hbase 1.1 from the same code base. Or is it ok to
> > say
> > Trafodion 2.0 version needs Apache hbase 1.0 and Trafodion 2.1 version
> > needs
> > Apache hbase 1.1.
> >
> > My vote is for the latter option.
> >
> > If so, basically you just need to remove the change in sqenvcom.sh and
> > rework it to build for Apache hbase 1.1.
> >
> > Selva
> >
> > -----Original Message-----
> > From: Ma, Sheng-Chen (Aven) [mailto:[email protected]]
> > Sent: Tuesday, June 21, 2016 2:43 AM
> > To: [email protected]
> > Subject: 答复: discuss on support hbase 1.1
> >
> > Hi
> > Seemingly it's not a good solution to use install_features to determine
> > which trx file will use. I 'm try to add more trx files. So that
> > install_features file will just indicate what trx files are available
> >
> >
> > -----邮件原件-----
> > 发件人: Ma, Sheng-Chen (Aven) [mailto:[email protected]]
> > 发送时间: 2016年6月21日 13:15
> > 收件人: [email protected]
> > 主题: 答复: discuss on support hbase 1.1
> >
> > Quote Steve's view first
> > "In other cases, we support multiple versions simultaneously by
> generating
> > multiple TRX files. I'm not sure we want to add more TRX files, but I'm
> > also
> > not sure we want to switch how the build works based on content of
> > install_features file.
> >
> > Previously it works the other way round -- the install_features file
> > indicates what TRX files are available, hence which distros are
> supported."
> >
> > No more trx files will be added.
> > Have to use install_features to decide whice hbase version will be
> > used.(discussed in the previous msg)
> >
> >
> > -----邮件原件-----
> > 发件人: Selva Govindarajan [mailto:[email protected]]
> > 发送时间: 2016年6月21日 12:51
> > 收件人: [email protected]
> > 主题: RE: discuss on support hbase 1.1
> >
> > I have the same observation as that of Steve for PR 503. Can you please
> > address them?
> >
> > Selva
> >
> > -----Original Message-----
> > From: Ma, Sheng-Chen (Aven) [mailto:[email protected]]
> > Sent: Monday, June 20, 2016 9:00 PM
> > To: [email protected]
> > Subject: 答复: discuss on support hbase 1.1
> >
> > Hi All:
> > Can someone review #503
> > (https://github.com/apache/incubator-trafodion/pull/503 )?
> > Since there is a long time ,and several time discussion shows it's the
> only
> > solution.
> >
> > Thanks
> >
> > -----邮件原件-----
> > 发件人: Ma, Sheng-Chen (Aven) [mailto:[email protected]]
> > 发送时间: 2016年6月16日 10:51
> > 收件人: [email protected]
> > 主题: 答复: discuss on support hbase 1.1
> >
> > Yes reflection is a good option, but I have do some research , it's hard
> to
> > use reflection,
> > eg: TrxTransactionState.java need add @Override to next(..) for hbase
> 1.1,
> > and this I think can't use reflect #ifndef HDP2.3 HBASE1.1
> > @Override
> > #endif
> > public synchronized boolean next(final List<Cell> results) throws
> > IOException {
> > return next(results, -1);
> > }
> >
> > Another is extends from new class, ScheduledChore is new class in
> hbase1.1
> > while Chore is disappear in hbase 1.0 #ifdef HDP2.3 HBASE1.1 public class
> > CleanOldTransactionsChore extends ScheduledChore { #else public class
> > CleanOldTransactionsChore extends Chore { #endif
> >
> > And there still some kind of changes I can hardly use reflection...
> >
> >
> >
> > -----邮件原件-----
> > 发件人: Selva Govindarajan [mailto:[email protected]]
> > 发送时间: 2016年6月15日 22:36
> > 收件人: [email protected]
> > 主题: RE: discuss on support hbase 1.1
> >
> > Another option is to check the code where Trafodion issues different APIs
> > and rework so that common set of API can be used. Reflection concept in
> > java
> > might also work in some cases. If the rework is not possible then it
> might
> > be necessary to stick with higher version.
> >
> > Selva
> >
> > -----Original Message-----
> > From: Ma, Sheng-Chen (Aven) [mailto:[email protected]]
> > Sent: Wednesday, June 15, 2016 2:14 AM
> > To: [email protected]
> > Subject: discuss on support hbase 1.1
> >
> > Hi all:
> > Apologize for this taking me so long... and thanks Amanda's reply (your
> > reply gives me confidence :)
> >
> > Last time I had an inspiration to solve support hbase 1.1 without use
> > install_features, but the truth is it's a misconception...
> >
> > I'd like to summary it first, according to the manual (see
> >
> >
> https://cwiki.apache.org/confluence/display/TRAFODION/Build+Source#BuildSource-makeall
> > )
> > :
> > 1. download source code.
> > 2. source sqenvr.sh & make package-all
> > 3. install Hadoop & hbase
> > 4. install trafodion
> >
> > Base on the above steps, there is no idea knows which hbase version the
> > user
> > will install. I can't mandatory provisions install Hadoop & hbase before
> > make trafodion package...
> > So the only way is to predefine hbase version in a config file, so there
> > appear the install_features.
> >
> > Another question is why not support both hbase 1.0 & 1.1 at the same
> time?
> > The answer is hbase1.0 &1.1 have different api , so I can only
> > support one
> > version at one package.
> > Further question is why not package two package eg:
> hbase-trx-apache1.0.jar
> > & hbase-trx-apache1.1.jar
> > It seems a solution, but I don't think it's a good way, reason is
> > similar
> > why there is only hbase-trx-cdh5_4.jar and hbase-trx-hdp2_3.jar.
> >
> > So I think this release the apache hbase ,we support 1.0...the next
> release
> > we support 1.1
> > ====================================================================
> > All above is my view, maybe not correct... if some one good at this pls
> > correct.
> >
> > Thanks.
> >
> >
> > -----邮件原件-----
> > 发件人: Amanda Moran [mailto:[email protected]]
> > 发送时间: 2016年6月4日 2:02
> > 收件人: dev <[email protected]>
> > 抄送: Steve Varnau <[email protected]>
> > 主题: Re: 答复: about pull 503
> >
> > Hi there Aven-
> >
> > You can you install_features, actually you must so that the installer
> knows
> > what version of HBase is supported in that particular package. The
> > installer
> > can install all versions of trafodion. So there will need to be HBase 1.0
> > support in the installer and HBase 1.1 . install_features is used during
> > installation not during sqenvcom.sh. Take a look at traf_config_check to
> > see
> > the details.
> >
> > Thanks.
> >
> > On Fri, Jun 3, 2016 at 1:01 AM, Ma, Sheng-Chen (Aven) <
> > [email protected]
> > > wrote:
> >
> > > hi Steve:
> > > I think u are right . I will do some change which will not rely on
> > > install_feature. Then I will have this discussion on dev list.
> > >
> > > Thanks
> > > Aven
> > >
> > >
> > > 发件人: Steve Varnau [mailto:[email protected]]
> > > 发送时间: 2016年6月3日 6:40
> > > 收件人: Ma, Sheng-Chen (Aven) <[email protected]>
> > > 抄送: Amanda Moran <[email protected]>
> > > 主题: RE: about pull 503
> > >
> > > Hi Aven,
> > >
> > > I think it would be really good to have this discussion on the dev
> > > list or on github. Others may also be confused by my remarks or may
> > > have different opinions than I do. Maybe I’m wrong. Either way, it
> > > would be good record for next person trying to add support of another
> > > HBase version. But I did not want to post your message without
> > > permission, so I’ll reply directly.
> > >
> > > Your point #1 about installer script is a good one. But I‘d say that
> > > the installer script could be enhanced to recognize which version of
> > > HBase is there and then move only correct TRX file. Either as part of
> > > your change or separately.
> > >
> > > If you are changing build to support apache hbase1.1, then why are you
> > > trying to retain hbase1.0? If we want both supported, we should be
> > > able to choose at install time not at build time.
> > >
> > > It is hbase-trx makefile that determines which TRX files are built. If
> > > developr wants to switch hbase version, they should change makefile,
> > > and maybe the install_features file too. But the sqenvcom.sh file
> > > should not read the install_features file. Your changes work
> > > differently than all the other versions of TRX. I would prefer
> > > consistency.
> > >
> > > --Steve
> > >
> > > From: Ma, Sheng-Chen (Aven) [mailto:[email protected]<mailto:
> > > [email protected]>]
> > > Sent: Thursday, June 2, 2016 3:02 AM
> > > To: Steve Varnau
> > > <[email protected]<mailto:[email protected]>>
> > > Subject: about pull 503
> > >
> > > Hi Steve:
> > >
> > > I send this mail to discuss #503 (
> > > https://github.com/apache/incubator-trafodion/pull/503)
> > > As your comment is a little confused me, I’d like to make sure
> something.
> > >
> > > As u commented:
> > >
> > > In other cases, we support multiple versions simultaneously by
> > > generating multiple TRX files. I'm not sure we want to add more TRX
> > > files, but I'm also not sure we want to switch how the build works
> > > based on content of install_features file.
> > >
> > > Previously it works the other way round -- the install_features file
> > > indicates what TRX files are available, hence which distros are
> > supported.
> > >
> > > This change will not add new trx file ,it just change
> > > hbase-trx-apache.jar from 1.0 to 1.1, It’s not good idea gen two
> > > apache-trx.jar, because there are two reason, one is our install
> > > script will add hbase-trx-apache*.jar to classpath, see
> > > /incubator-trafodion/install/installer/traf_apache_mods
> > > line51. Two is if 2 different version hbase-trx-apache*.jar added to
> > > classpath, hbase will only use the first meet class in the jar
> > > package, it may lead to conflict.
> > >
> > > About install_features file. I am not sure we want to switch how the
> > > build works based on content of install_features file either. But the
> > > only way I can distinguish which hbase version to use is this file. If
> > > there is another place to distinguish this ,pls tell me ,thanks.
> > >
> > > About your last sentence , I am really confuse, is that means we
> > > indeed use install_features to distinguish hbase version?
> > >
> > >
> > > Thanks,
> > > Aven
> > >
> > >
> >
> >
> > --
> > Thanks,
> >
> > Amanda Moran
> >
>
>
>
> --
> Thanks,
>
> Amanda Moran
>
--
Regards, --Qifan
