HI Pedro
thanks for your help, I think we know that we need to set the classpath to
the hadoop program, and what we tried was
HADOOP_CLASSPATH=/etc/hbase/conf/hbase-site.xml hadoop jar $test_jar but it
didn't work
So we are wondering if anything we did wrong?
On Wed, Feb 20, 2019 at 3:24 PM Pedro Boado <pbo...@apache.org> wrote:
> Hi,
>
> How many concurrent client connections are we talking about? You might be
> opening more connections than the RS can handle ( under these circumstances
> most of the client threads would end exhausting their retry count ) . I
> would bet that you've get a bottleneck in the RS keeping SYSTEM.CATALOG
> table (this was an issue in 4.7 ) as every new connection would be querying
> this table first.
>
> Try to update to our cloudera-compatible parcels instead of using clabs -
> which are discontinued by Cloudera and not supported by the Apache Phoenix
> project - .
>
> Once updated to phoenix 4.14 you should be able to use
> UPDATE_CACHE_FREQUENCY
> property in order to reduce pressure on system tables.
>
> Adding an hbase-site.xml with the required properties to the client
> application classpath should just work.
>
> I hope it helps.
>
> On Wed, 20 Feb 2019, 22:50 Xiaoxiao Wang, <xxw...@23andme.com.invalid>
> wrote:
>
> > Hi, who may help
> >
> > We are running a Hadoop application that needs to use phoenix JDBC
> > connection from the workers.
> > The connection works, but when too many connection established at the
> same
> > time, it throws RPC timeouts
> >
> > Error: java.io.IOException:
> > org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36,
> > exceptions: Wed Feb 20 20:02:43 UTC 2019, null, java.net
> .SocketTimeoutException:
> > callTimeout=60000, callDuration=60506. ...
> >
> > So we have figured we should probably set a higher hbase.rpc.timeout
> > value, but then it comes to the issue:
> >
> > A little bit background on how we run the application
> >
> > Here is how we get PhoenixConnection from java program
> > DriverManager.getConnection("jdbc:phoenix:host", props)
> > And we trigger the program by using
> > hadoop jar $test_jar
> >
> >
> > We have tried multiple approaches to load hbase/phoenix configuration,
> but
> > none of them get respected by PhoenixConnection, here are the methods we
> > tried
> > * Pass hbase_conf_dir through HADOOP_CLASSPATH, so run the hadoop
> > application like HADOOP_CLASSPATH=/etc/hbase/conf/ hadoop jar $test_jar .
> > However, PhoenixConnection doesn’t respect the parameters
> > * Tried passing -Dhbase.rpc.timeout=1800, which is picked up by hbase
> conf
> > object, but not PhoniexConnection
> > * Explicitly set those parameters and pass them to the PhoenixConnection
> > props.setProperty("hbase.rpc.timeout", "1800");
> > props.setProperty(“phoenix.query.timeoutMs", "1800");
> > Also didn’t get respected by PhoenixConnection
> > * also tried what is suggested by phoenix here
> > https://phoenix.apache.org/#connStr , use :longRunning together with
> > those properties, still didn’t seem to work
> >
> >
> > Besides all those approaches we tried, I have explicitly output those
> > parameters we care from the connection,
> > connection.getQueryServices().getProps()
> > The default values I got are 60000 for hbase.rpc.timeout, and 600k for
> > phoenix.query.timeoutMs , so I have tried to run a query lthat would run
> > longer than 10 mins, Ideally it should timeout, however, it runs over 20
> > mins and didn’t timeout. So I’m wondering how PhoenixConnection respect
> > those properties?
> >
> >
> > So with some of your help, we’d like to know if there’s any thing wrong
> > with our approaches. And we’d like to get rid of those
> SocketTimeExceptions.
> > We are using phoenix-core version is 4.7.0-clabs-phoenix1.3.0 , and our
> > phoenix-client version is phoenix-4.7.0-clabs-phoenix1.3.0.23 (we have
> > tried phoenix-4.14.0-HBase-1.3 as well, which didn’t work either).
> >
> >
> > Thanks for your time
> >
> >
> >
> >
> >
>
