[ https://issues.apache.org/jira/browse/PHOENIX-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abhishek Kothalikar updated PHOENIX-2599: ----------------------------------------- Description: When running Spark 1.4.1 and Phoenix 4.5.1 via Phoenix-Spark connector. We notice sometimes (30~50%) time the following error would appear and kill the running spark job: 16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 110.0 failed 4 times, most recent failure: Lost task 5.3 in stage 110.0 (TID 35526, datanode-123.somewhere): java.lang.RuntimeException: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are out of date. at com.google.common.base.Throwables.propagate(Throwables.java:156) at org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126) at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133) at org.apache.spark.rdd.NewHadoopRDD.compute(`.scala:104) at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66) at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) at org.apache.spark.scheduler.Task.run(Task.scala:70) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are out of date. at org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131) at org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115) at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104) at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70) at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88) at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79) at org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111) ... 18 more was: When running Spark 1.4.1 and Phoenix 4.5.1 via Phoenix-Spark connector. We notice sometimes (30~50%) time the following error would appear and kill the running spark job: 16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 110.0 failed 4 times, most recent failure: Lost task 5.3 in stage 110.0 (TID 35526, datanode-123.somewhere): java.lang.RuntimeException: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are out of date. at com.google.common.base.Throwables.propagate(Throwables.java:156) at org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126) at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133) at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104) at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66) at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) at org.apache.spark.scheduler.Task.run(Task.scala:70) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are out of date. at org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131) at org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115) at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104) at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70) at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88) at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79) at org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111) ... 18 more > PhoenixRecordReader does not handle StaleRegionBoundaryCacheException > --------------------------------------------------------------------- > > Key: PHOENIX-2599 > URL: https://issues.apache.org/jira/browse/PHOENIX-2599 > Project: Phoenix > Issue Type: Bug > Affects Versions: 4.5.1 > Environment: HBase 1.0 + Linux > Reporter: Li Gao > Assignee: Josh Mahonin > Priority: Major > Fix For: 4.7.0 > > Attachments: PHOENIX-2599.patch > > > When running Spark 1.4.1 and Phoenix 4.5.1 via Phoenix-Spark connector. We > notice sometimes (30~50%) time the following error would appear and kill the > running spark job: > 16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw exception: > org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in > stage 110.0 failed 4 times, most recent failure: Lost task 5.3 in stage 110.0 > (TID 35526, datanode-123.somewhere): java.lang.RuntimeException: > org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 > (XCL08): Cache of region boundaries are out of date. > at com.google.common.base.Throwables.propagate(Throwables.java:156) > at > org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126) > at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133) > at org.apache.spark.rdd.NewHadoopRDD.compute(`.scala:104) > at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66) > at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) > at org.apache.spark.scheduler.Task.run(Task.scala:70) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR > 1108 (XCL08): Cache of region boundaries are out of date. > at > org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304) > at > org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) > at > org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131) > at > org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115) > at > org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104) > at > org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70) > at > org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88) > at > org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79) > at > org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111) > ... 18 more -- This message was sent by Atlassian Jira (v8.20.10#820010)