With the fix of PHOENIX-2447<https://issues.apache.org/jira/browse/PHOENIX-2447> in 4.7.0, now Phoenix will try multiple times for StaleRegionBoundaryCacheException until query timeout.
Alicia From: Josh Mahonin <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, January 14, 2016 at 12:33 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase 1.0 and Spark 1.4.1 Hi Li, I've not seen this error myself, though some searching returns a possible root cause: http://mail-archives.apache.org/mod_mbox/incubator-phoenix-user/201507.mbox/%3CCAAF1JdjNW98dAnxf3kx=ndkswyorpt1redb9dqwjbhvvlfn...@mail.gmail.com%3E Could you file a JIRA ticket for this please? It's possible the MapReduce (and by extension, Spark) integration isn't handling this gracefully. Also, if possible, any information you have about your environment, table sizes, read/write workload, etc. would be helpful as well. Thanks! Josh On Thu, Jan 14, 2016 at 3:14 PM, Li Gao <[email protected]<mailto:[email protected]>> wrote: Hi Phoenix users, We are seeing occasionally (maybe 30 ~ 50% of time) this StaleRegionBoundaryCacheException when running Spark 1.4.1 and Phoenix 4.5.1-HBase 1.0. Not sure how to troubleshoot such issue. Any hints and insights are greatly appreciated! Thanks, Li PS: The following are the exception stack trace: 16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 110.0 failed 4 times, most recent failure: Lost task 5.3 in stage 110.0 (TID 35526, datanode-123.somewhere): java.lang.RuntimeException: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are out of date. at com.google.common.base.Throwables.propagate(Throwables.java:156) at org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126) at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133) at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104) at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66) at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) at org.apache.spark.scheduler.Task.run(Task.scala:70) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are out of date. at org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131) at org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115) at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104) at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70) at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88) at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79) at org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111) ... 18 more
