Hi Li, This should be resolved in Phoenix 4.7.0. You can also apply the patch yourself from here: https://issues.apache.org/jira/browse/PHOENIX-2599
Please let us know if it fixes the problem for you. Thanks! Josh On Thu, Jan 14, 2016 at 6:49 PM, James Taylor <jamestay...@apache.org> wrote: > That was my first thought too, Alicia. However, our MR and Spark > integration uses a different code path. See my comment on PHOENIX-2599 for > a potential fix. > > Thanks, > James > > On Thu, Jan 14, 2016 at 2:55 PM, Alicia Shu <a...@hortonworks.com> wrote: > >> With the fix of PHOENIX-2447 >> <https://issues.apache.org/jira/browse/PHOENIX-2447> in 4.7.0, now >> Phoenix will try multiple times for StaleRegionBoundaryCacheException until >> query timeout. >> >> Alicia >> >> From: Josh Mahonin <jmaho...@gmail.com> >> Reply-To: "user@phoenix.apache.org" <user@phoenix.apache.org> >> Date: Thursday, January 14, 2016 at 12:33 PM >> To: "user@phoenix.apache.org" <user@phoenix.apache.org> >> Subject: Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase >> 1.0 and Spark 1.4.1 >> >> Hi Li, >> >> I've not seen this error myself, though some searching returns a possible >> root cause: >> >> http://mail-archives.apache.org/mod_mbox/incubator-phoenix-user/201507.mbox/%3CCAAF1JdjNW98dAnxf3kx=ndkswyorpt1redb9dqwjbhvvlfn...@mail.gmail.com%3E >> >> Could you file a JIRA ticket for this please? It's possible the MapReduce >> (and by extension, Spark) integration isn't handling this gracefully. >> >> Also, if possible, any information you have about your environment, table >> sizes, read/write workload, etc. would be helpful as well. >> >> Thanks! >> >> Josh >> >> On Thu, Jan 14, 2016 at 3:14 PM, Li Gao <g...@marinsoftware.com> wrote: >> >>> Hi Phoenix users, >>> >>> >>> We are seeing occasionally (maybe 30 ~ 50% of time) this >>> StaleRegionBoundaryCacheException when running Spark 1.4.1 and Phoenix >>> 4.5.1-HBase 1.0. >>> >>> Not sure how to troubleshoot such issue. Any hints and insights are >>> greatly appreciated! >>> >>> Thanks, >>> >>> Li >>> >>> PS: The following are the exception stack trace: >>> >>> 16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw >>> exception: org.apache.spark.SparkException: Job aborted due to stage >>> failure: Task 5 in stage 110.0 failed 4 times, most recent failure: Lost >>> task 5.3 in stage 110.0 (TID 35526, datanode-123.somewhere): >>> java.lang.RuntimeException: >>> org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 >>> (XCL08): Cache of region boundaries are out of date. >>> >>> at com.google.common.base.Throwables.propagate(Throwables.java:156) >>> >>> at >>> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126) >>> >>> at >>> org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133) >>> >>> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104) >>> >>> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66) >>> >>> at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52) >>> >>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >>> >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >>> >>> at >>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >>> >>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >>> >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >>> >>> at >>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >>> >>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >>> >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >>> >>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) >>> >>> at org.apache.spark.scheduler.Task.run(Task.scala:70) >>> >>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> >>> at java.lang.Thread.run(Thread.java:745) >>> >>> Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: >>> ERROR 1108 (XCL08): Cache of region boundaries are out of date. >>> >>> at >>> org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304) >>> >>> at >>> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) >>> >>> at >>> org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131) >>> >>> at >>> org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115) >>> >>> at >>> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104) >>> >>> at >>> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70) >>> >>> at >>> org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88) >>> >>> at >>> org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79) >>> >>> at >>> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111) >>> >>> ... 18 more >>> >> >> >
