A bit more information, I feel the classpath didn't get passed in correctly
by doing
conf = HBaseConfiguration.addHbaseResources(super.getConf());
and this conf also didn't pick up the expected properties
On Wed, Feb 20, 2019 at 3:56 PM Xiaoxiao Wang <xxw...@23andme.com> wrote:
> Pedro
>
> thanks for your info, yes, I have tried both
> HADOOP_CLASSPATH=/etc/hbase/conf/hbase-site.xml and
> HADOOP_CLASSPATH=/etc/hbase/conf/ (without file), and yes checked
> hadoop-env.sh as well to make sure it did
> HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/others
>
> And also for your second question, it is indeed a map reduce job, and it
> is trying to query phoenix from map function! (and we make sure all the
> nodes have hbase-site.xml installed properly )
>
> thanks
>
> On Wed, Feb 20, 2019 at 3:53 PM Pedro Boado <pedro.bo...@gmail.com> wrote:
>
>> Your classpath variable should be pointing to the folder containing your
>> hbase-site.xml and not directly to the file.
>>
>> But certain distributions tend to override that envvar inside
>> hadoop-env.sh
>> or hadoop.sh .
>>
>> Out of curiosity, have you written a map-reduce application and are you
>> querying phoenix from map functions?
>>
>> On Wed, 20 Feb 2019, 23:34 Xiaoxiao Wang, <xxw...@23andme.com.invalid>
>> wrote:
>>
>> > HI Pedro
>> >
>> > thanks for your help, I think we know that we need to set the classpath
>> to
>> > the hadoop program, and what we tried was
>> > HADOOP_CLASSPATH=/etc/hbase/conf/hbase-site.xml hadoop jar $test_jar
>> but it
>> > didn't work
>> > So we are wondering if anything we did wrong?
>> >
>> > On Wed, Feb 20, 2019 at 3:24 PM Pedro Boado <pbo...@apache.org> wrote:
>> >
>> > > Hi,
>> > >
>> > > How many concurrent client connections are we talking about? You
>> might be
>> > > opening more connections than the RS can handle ( under these
>> > circumstances
>> > > most of the client threads would end exhausting their retry count ) .
>> I
>> > > would bet that you've get a bottleneck in the RS keeping
>> SYSTEM.CATALOG
>> > > table (this was an issue in 4.7 ) as every new connection would be
>> > querying
>> > > this table first.
>> > >
>> > > Try to update to our cloudera-compatible parcels instead of using
>> clabs -
>> > > which are discontinued by Cloudera and not supported by the Apache
>> > Phoenix
>> > > project - .
>> > >
>> > > Once updated to phoenix 4.14 you should be able to use
>> > > UPDATE_CACHE_FREQUENCY
>> > > property in order to reduce pressure on system tables.
>> > >
>> > > Adding an hbase-site.xml with the required properties to the client
>> > > application classpath should just work.
>> > >
>> > > I hope it helps.
>> > >
>> > > On Wed, 20 Feb 2019, 22:50 Xiaoxiao Wang, <xxw...@23andme.com.invalid
>> >
>> > > wrote:
>> > >
>> > > > Hi, who may help
>> > > >
>> > > > We are running a Hadoop application that needs to use phoenix JDBC
>> > > > connection from the workers.
>> > > > The connection works, but when too many connection established at
>> the
>> > > same
>> > > > time, it throws RPC timeouts
>> > > >
>> > > > Error: java.io.IOException:
>> > > > org.apache.phoenix.exception.PhoenixIOException: Failed after
>> > > attempts=36,
>> > > > exceptions: Wed Feb 20 20:02:43 UTC 2019, null, java.net
>> > > .SocketTimeoutException:
>> > > > callTimeout=60000, callDuration=60506. ...
>> > > >
>> > > > So we have figured we should probably set a higher
>> hbase.rpc.timeout
>> > > > value, but then it comes to the issue:
>> > > >
>> > > > A little bit background on how we run the application
>> > > >
>> > > > Here is how we get PhoenixConnection from java program
>> > > > DriverManager.getConnection("jdbc:phoenix:host", props)
>> > > > And we trigger the program by using
>> > > > hadoop jar $test_jar
>> > > >
>> > > >
>> > > > We have tried multiple approaches to load hbase/phoenix
>> configuration,
>> > > but
>> > > > none of them get respected by PhoenixConnection, here are the
>> methods
>> > we
>> > > > tried
>> > > > * Pass hbase_conf_dir through HADOOP_CLASSPATH, so run the hadoop
>> > > > application like HADOOP_CLASSPATH=/etc/hbase/conf/ hadoop jar
>> > $test_jar .
>> > > > However, PhoenixConnection doesn’t respect the parameters
>> > > > * Tried passing -Dhbase.rpc.timeout=1800, which is picked up by
>> hbase
>> > > conf
>> > > > object, but not PhoniexConnection
>> > > > * Explicitly set those parameters and pass them to the
>> > PhoenixConnection
>> > > > props.setProperty("hbase.rpc.timeout", "1800");
>> > > > props.setProperty(“phoenix.query.timeoutMs", "1800");
>> > > > Also didn’t get respected by PhoenixConnection
>> > > > * also tried what is suggested by phoenix here
>> > > > https://phoenix.apache.org/#connStr , use :longRunning together
>> with
>> > > > those properties, still didn’t seem to work
>> > > >
>> > > >
>> > > > Besides all those approaches we tried, I have explicitly output
>> those
>> > > > parameters we care from the connection,
>> > > > connection.getQueryServices().getProps()
>> > > > The default values I got are 60000 for hbase.rpc.timeout, and 600k
>> for
>> > > > phoenix.query.timeoutMs , so I have tried to run a query lthat would
>> > run
>> > > > longer than 10 mins, Ideally it should timeout, however, it runs
>> over
>> > 20
>> > > > mins and didn’t timeout. So I’m wondering how PhoenixConnection
>> respect
>> > > > those properties?
>> > > >
>> > > >
>> > > > So with some of your help, we’d like to know if there’s any thing
>> wrong
>> > > > with our approaches. And we’d like to get rid of those
>> > > SocketTimeExceptions.
>> > > > We are using phoenix-core version is 4.7.0-clabs-phoenix1.3.0 , and
>> our
>> > > > phoenix-client version is phoenix-4.7.0-clabs-phoenix1.3.0.23 (we
>> have
>> > > > tried phoenix-4.14.0-HBase-1.3 as well, which didn’t work either).
>> > > >
>> > > >
>> > > > Thanks for your time
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> >
>>
>
