Whatever is in super.getConf() should get overriden by hbase-site.xml
because addHbaseResources because will layer on hbase-site.xml last. The
question is which one got picked up... (maybe there is another one on the
classpath, is that possible?)
On Wed, Feb 20, 2019 at 4:10 PM Xiaoxiao Wang <xxw...@23andme.com.invalid>
wrote:
> I'm trying out on the mapreduce application, I made it work on my toy
> application
>
> On Wed, Feb 20, 2019 at 4:09 PM William Shen <wills...@marinsoftware.com>
> wrote:
>
> > A bit of a long shot, but do you happen to have another hbase-site.xml
> > bundled in your jar accidentally that might be overriding what is on the
> > classpath?
> >
> > On Wed, Feb 20, 2019 at 3:58 PM Xiaoxiao Wang <xxw...@23andme.com.invalid
> >
> > wrote:
> >
> > > A bit more information, I feel the classpath didn't get passed in
> > correctly
> > > by doing
> > >
> > > conf = HBaseConfiguration.addHbaseResources(super.getConf());
> > >
> > > and this conf also didn't pick up the expected properties
> > >
> > >
> > > On Wed, Feb 20, 2019 at 3:56 PM Xiaoxiao Wang <xxw...@23andme.com>
> > wrote:
> > >
> > > > Pedro
> > > >
> > > > thanks for your info, yes, I have tried both
> > > > HADOOP_CLASSPATH=/etc/hbase/conf/hbase-site.xml and
> > > > HADOOP_CLASSPATH=/etc/hbase/conf/ (without file), and yes checked
> > > > hadoop-env.sh as well to make sure it did
> > > > HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/others
> > > >
> > > > And also for your second question, it is indeed a map reduce job, and
> > it
> > > > is trying to query phoenix from map function! (and we make sure all
> the
> > > > nodes have hbase-site.xml installed properly )
> > > >
> > > > thanks
> > > >
> > > > On Wed, Feb 20, 2019 at 3:53 PM Pedro Boado <pedro.bo...@gmail.com>
> > > wrote:
> > > >
> > > >> Your classpath variable should be pointing to the folder containing
> > your
> > > >> hbase-site.xml and not directly to the file.
> > > >>
> > > >> But certain distributions tend to override that envvar inside
> > > >> hadoop-env.sh
> > > >> or hadoop.sh .
> > > >>
> > > >> Out of curiosity, have you written a map-reduce application and are
> > you
> > > >> querying phoenix from map functions?
> > > >>
> > > >> On Wed, 20 Feb 2019, 23:34 Xiaoxiao Wang,
> <xxw...@23andme.com.invalid
> > >
> > > >> wrote:
> > > >>
> > > >> > HI Pedro
> > > >> >
> > > >> > thanks for your help, I think we know that we need to set the
> > > classpath
> > > >> to
> > > >> > the hadoop program, and what we tried was
> > > >> > HADOOP_CLASSPATH=/etc/hbase/conf/hbase-site.xml hadoop jar
> $test_jar
> > > >> but it
> > > >> > didn't work
> > > >> > So we are wondering if anything we did wrong?
> > > >> >
> > > >> > On Wed, Feb 20, 2019 at 3:24 PM Pedro Boado <pbo...@apache.org>
> > > wrote:
> > > >> >
> > > >> > > Hi,
> > > >> > >
> > > >> > > How many concurrent client connections are we talking about? You
> > > >> might be
> > > >> > > opening more connections than the RS can handle ( under these
> > > >> > circumstances
> > > >> > > most of the client threads would end exhausting their retry
> count
> > )
> > > .
> > > >> I
> > > >> > > would bet that you've get a bottleneck in the RS keeping
> > > >> SYSTEM.CATALOG
> > > >> > > table (this was an issue in 4.7 ) as every new connection would
> be
> > > >> > querying
> > > >> > > this table first.
> > > >> > >
> > > >> > > Try to update to our cloudera-compatible parcels instead of
> using
> > > >> clabs -
> > > >> > > which are discontinued by Cloudera and not supported by the
> Apache
> > > >> > Phoenix
> > > >> > > project - .
> > > >> > >
> > > >> > > Once updated to phoenix 4.14 you should be able to use
> > > >> > > UPDATE_CACHE_FREQUENCY
> > > >> > > property in order to reduce pressure on system tables.
> > > >> > >
> > > >> > > Adding an hbase-site.xml with the required properties to the
> > client
> > > >> > > application classpath should just work.
> > > >> > >
> > > >> > > I hope it helps.
> > > >> > >
> > > >> > > On Wed, 20 Feb 2019, 22:50 Xiaoxiao Wang,
> > > <xxw...@23andme.com.invalid
> > > >> >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi, who may help
> > > >> > > >
> > > >> > > > We are running a Hadoop application that needs to use phoenix
> > JDBC
> > > >> > > > connection from the workers.
> > > >> > > > The connection works, but when too many connection established
> > at
> > > >> the
> > > >> > > same
> > > >> > > > time, it throws RPC timeouts
> > > >> > > >
> > > >> > > > Error: java.io.IOException:
> > > >> > > > org.apache.phoenix.exception.PhoenixIOException: Failed after
> > > >> > > attempts=36,
> > > >> > > > exceptions: Wed Feb 20 20:02:43 UTC 2019, null, java.net
> > > >> > > .SocketTimeoutException:
> > > >> > > > callTimeout=60000, callDuration=60506. ...
> > > >> > > >
> > > >> > > > So we have figured we should probably set a higher
> > > >> hbase.rpc.timeout
> > > >> > > > value, but then it comes to the issue:
> > > >> > > >
> > > >> > > > A little bit background on how we run the application
> > > >> > > >
> > > >> > > > Here is how we get PhoenixConnection from java program
> > > >> > > > DriverManager.getConnection("jdbc:phoenix:host", props)
> > > >> > > > And we trigger the program by using
> > > >> > > > hadoop jar $test_jar
> > > >> > > >
> > > >> > > >
> > > >> > > > We have tried multiple approaches to load hbase/phoenix
> > > >> configuration,
> > > >> > > but
> > > >> > > > none of them get respected by PhoenixConnection, here are the
> > > >> methods
> > > >> > we
> > > >> > > > tried
> > > >> > > > * Pass hbase_conf_dir through HADOOP_CLASSPATH, so run the
> > hadoop
> > > >> > > > application like HADOOP_CLASSPATH=/etc/hbase/conf/ hadoop jar
> > > >> > $test_jar .
> > > >> > > > However, PhoenixConnection doesn’t respect the parameters
> > > >> > > > * Tried passing -Dhbase.rpc.timeout=1800, which is picked up
> by
> > > >> hbase
> > > >> > > conf
> > > >> > > > object, but not PhoniexConnection
> > > >> > > > * Explicitly set those parameters and pass them to the
> > > >> > PhoenixConnection
> > > >> > > > props.setProperty("hbase.rpc.timeout", "1800");
> > > >> > > > props.setProperty(“phoenix.query.timeoutMs", "1800");
> > > >> > > > Also didn’t get respected by PhoenixConnection
> > > >> > > > * also tried what is suggested by phoenix here
> > > >> > > > https://phoenix.apache.org/#connStr , use :longRunning
> together
> > > >> with
> > > >> > > > those properties, still didn’t seem to work
> > > >> > > >
> > > >> > > >
> > > >> > > > Besides all those approaches we tried, I have explicitly
> output
> > > >> those
> > > >> > > > parameters we care from the connection,
> > > >> > > > connection.getQueryServices().getProps()
> > > >> > > > The default values I got are 60000 for hbase.rpc.timeout, and
> > 600k
> > > >> for
> > > >> > > > phoenix.query.timeoutMs , so I have tried to run a query lthat
> > > would
> > > >> > run
> > > >> > > > longer than 10 mins, Ideally it should timeout, however, it
> runs
> > > >> over
> > > >> > 20
> > > >> > > > mins and didn’t timeout. So I’m wondering how
> PhoenixConnection
> > > >> respect
> > > >> > > > those properties?
> > > >> > > >
> > > >> > > >
> > > >> > > > So with some of your help, we’d like to know if there’s any
> > thing
> > > >> wrong
> > > >> > > > with our approaches. And we’d like to get rid of those
> > > >> > > SocketTimeExceptions.
> > > >> > > > We are using phoenix-core version is 4.7.0-clabs-phoenix1.3.0
> ,
> > > and
> > > >> our
> > > >> > > > phoenix-client version is phoenix-4.7.0-clabs-phoenix1.3.0.23
> > (we
> > > >> have
> > > >> > > > tried phoenix-4.14.0-HBase-1.3 as well, which didn’t work
> > either).
> > > >> > > >
> > > >> > > >
> > > >> > > > Thanks for your time
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
