Hi Marek, could you share the logs statements which happened before the SIGSEGV with us? They might be helpful to understand what happened before. Moreover, it would be helpful to get access to your custom serializer implementations. I'm also pulling in Gordon who worked on the TypeSerializerSnapshot improvements.
Cheers, Till On Thu, Sep 12, 2019 at 9:28 AM Marek Maj <marekm...@gmail.com> wrote: > Hi everyone, > > Recently we decided to upgrade from flink 1.7.2 to 1.8.1. After an upgrade > our task managers started to fail with SIGSEGV error from time to time. > > In process of adjusting the code to 1.8.1, we noticed that there were some > changes around TypeSerializerSnapshot interface and its implementations. > At that time we had a few custom serializers which we decided to throw out > during migration and then leverage flink default serializers. We don't mind > clearing the state in the process of migration, an effort to migrate with > state seems to be not worth it. > > Unfortunately after running new version we see SIGSEGV errors from time to > time. It may be that serialization is not the real cause, but at the moment > it seems to be the most probable reason. We have not performed any > significant code changes besides serialization area. > > We run job on yarn, hdp version 2.7.3.2.6.2.0-205. > Checkpoint configuration: RocksDB backend, not incremental, 50s min > processing time > > You can find parts of JobManager log and ErrorFile log of failed container > included below. > > Any suggestions are welcome > > Best regards > Marek Maj > > jobmanager.log > > 019-09-10 16:30:28.177 INFO o.a.f.r.c.CheckpointCoordinator - Completed > checkpoint 47 for job c8a9ae03785ade86348c3189cf7dd965 (18532488122 bytes > in 60871 ms). > > 2019-09-10 16:31:19.223 INFO o.a.f.r.c.CheckpointCoordinator - > Triggering checkpoint 48 @ 1568111478177 for job > c8a9ae03785ade86348c3189cf7dd965. > > 2019-09-10 16:32:19.280 INFO o.a.f.r.c.CheckpointCoordinator - > Completed checkpoint 48 for job c8a9ae03785ade86348c3189cf7dd965 > (19049515705 bytes in 61083 ms). > > 2019-09-10 16:33:10.480 INFO o.a.f.r.c.CheckpointCoordinator - > Triggering checkpoint 49 @ 1568111589279 for job > c8a9ae03785ade86348c3189cf7dd965. > > 2019-09-10 16:33:36.773 WARN o.a.f.r.r.h.l.m.MetricFetcherImpl - > Requesting TaskManager's path for query services failed. > > java.util.concurrent.CompletionException: > akka.pattern.AskTimeoutException: Ask timed out on > [Actor[akka://flink/user/dispatcher#374570759]] after [10000 ms]. > Sender[null] sent message of type > "org.apache.flink.runtime.rpc.messages.LocalFencedMessage". > > at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) > > at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) > > at > java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) > > at > java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) > > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > > at > org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:816) > > at akka.dispatch.OnComplete.internal(Future.scala:258) > > at akka.dispatch.OnComplete.internal(Future.scala:256) > > at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186) > > at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183) > > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) > > at > org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74) > > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44) > > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252) > > at > akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603) > > at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126) > > at > scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601) > > at > scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) > > at > scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599) > > at > akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329) > > at > akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280) > > at > akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284) > > at > akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: akka.pattern.AskTimeoutException: Ask timed out on > [Actor[akka://flink/user/dispatcher#374570759]] after [10000 ms]. > Sender[null] sent message of type > "org.apache.flink.runtime.rpc.messages.LocalFencedMessage". > > at > akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604) > > ... 9 common frames omitted > > 2019-09-10 16:33:48.782 WARN o.a.f.r.r.h.l.m.MetricFetcherImpl - > Requesting TaskManager's path for query services failed. > > java.util.concurrent.CompletionException: > akka.pattern.AskTimeoutException: Ask timed out on > [Actor[akka://flink/user/dispatcher#374570759]] after [10000 ms]. > Sender[null] sent message of type > "org.apache.flink.runtime.rpc.messages.LocalFencedMessage". > > at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) > > at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) > > at > java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) > > at > java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) > > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > > at > org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:816) > > at akka.dispatch.OnComplete.internal(Future.scala:258) > > at akka.dispatch.OnComplete.internal(Future.scala:256) > > at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186) > > at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183) > > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) > > at > org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74) > > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44) > > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252) > > at > akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603) > > at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126) > > at > scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601) > > at > scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) > > at > scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599) > > at > akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329) > > at > akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280) > > at > akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284) > > at > akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: akka.pattern.AskTimeoutException: Ask timed out on > [Actor[akka://flink/user/dispatcher#374570759]] after [10000 ms]. > Sender[null] sent message of type > "org.apache.flink.runtime.rpc.messages.LocalFencedMessage". > > at > akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604) > > ... 9 common frames omitted > > 2019-09-10 16:34:00.802 WARN o.a.f.r.r.h.l.m.MetricFetcherImpl - > Requesting TaskManager's path for query services failed. > > java.util.concurrent.CompletionException: > akka.pattern.AskTimeoutException: Ask timed out on > [Actor[akka://flink/user/dispatcher#374570759]] after [10000 ms]. > Sender[null] sent message of type > "org.apache.flink.runtime.rpc.messages.LocalFencedMessage". > > at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) > > at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) > > at > java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) > > at > java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) > > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > > at > org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:816) > > at akka.dispatch.OnComplete.internal(Future.scala:258) > > at akka.dispatch.OnComplete.internal(Future.scala:256) > > at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186) > > at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183) > > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) > > at > org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74) > > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44) > > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252) > > at > akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603) > > at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126) > > at > scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601) > > at > scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) > > at > scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599) > > at > akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329) > > at > akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280) > > at > akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284) > > at > akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: akka.pattern.AskTimeoutException: Ask timed out on > [Actor[akka://flink/user/dispatcher#374570759]] after [10000 ms]. > Sender[null] sent message of type > "org.apache.flink.runtime.rpc.messages.LocalFencedMessage". > > at > akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604) > > ... 9 common frames omitted > > 2019-09-10 16:34:03.800 INFO o.a.flink.yarn.YarnResourceManager - The > heartbeat of TaskManager with id container_e67_1568017536744_0044_01_000023 > timed out. > > 2019-09-10 16:34:03.801 INFO o.a.flink.yarn.YarnResourceManager - > Closing TaskExecutor connection container_e67_1568017536744_0044_01_000023 > because: The heartbeat of TaskManager with id > container_e67_1568017536744_0044_01_000023 timed out. > > 2019-09-10 16:34:03.803 INFO o.a.f.r.e.ExecutionGraph - my-function > (1/32) (ae416d03ddc94a3633673c4050b8f2ae) switched from RUNNING to FAILED. > > org.apache.flink.util.FlinkException: The assigned slot > container_e67_1568017536744_0044_01_000023_0 was removed. > > at > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.removeSlot(SlotManager.java:899) > > at > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.removeSlots(SlotManager.java:869) > > at > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.internalUnregisterTaskManager(SlotManager.java:1080) > > at > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.unregisterTaskManager(SlotManager.java:391) > > at > org.apache.flink.runtime.resourcemanager.ResourceManager.closeTaskManagerConnection(ResourceManager.java:845) > > at > org.apache.flink.runtime.resourcemanager.ResourceManager$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(ResourceManager.java:1187) > > at > org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl$HeartbeatMonitor.run(HeartbeatManagerImpl.java:318) > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:392) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:185) > > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:147) > > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40) > > at > akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165) > > at akka.actor.Actor$class.aroundReceive(Actor.scala:502) > > at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95) > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) > > at akka.actor.ActorCell.invoke(ActorCell.scala:495) > > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) > > at akka.dispatch.Mailbox.run(Mailbox.scala:224) > > at akka.dispatch.Mailbox.exec(Mailbox.scala:234) > > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > > at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > 2019-09-10 16:34:03.803 INFO o.a.f.r.c.CheckpointCoordinator - > Discarding checkpoint 49 of job c8a9ae03785ade86348c3189cf7dd965. > > org.apache.flink.util.FlinkException: The assigned slot > container_e67_1568017536744_0044_01_000023_0 was removed. > > at > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.removeSlot(SlotManager.java:899) > > at > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.removeSlots(SlotManager.java:869) > > at > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.internalUnregisterTaskManager(SlotManager.java:1080) > > at > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.unregisterTaskManager(SlotManager.java:391) > > at > org.apache.flink.runtime.resourcemanager.ResourceManager.closeTaskManagerConnection(ResourceManager.java:845) > > at > org.apache.flink.runtime.resourcemanager.ResourceManager$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(ResourceManager.java:1187) > > at > org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl$HeartbeatMonitor.run(HeartbeatManagerImpl.java:318) > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:392) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:185) > > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:147) > > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40) > > at > akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165) > > at akka.actor.Actor$class.aroundReceive(Actor.scala:502) > > at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95) > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) > > at akka.actor.ActorCell.invoke(ActorCell.scala:495) > > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) > > at akka.dispatch.Mailbox.run(Mailbox.scala:224) > > at akka.dispatch.Mailbox.exec(Mailbox.scala:234) > > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > > at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > 2019-09-10 16:34:03.803 INFO o.a.f.r.e.ExecutionGraph - Job > ProcessingJob (c8a9ae03785ade86348c3189cf7dd965) switched from state > RUNNING to FAILING. > > org.apache.flink.util.FlinkException: The assigned slot > container_e67_1568017536744_0044_01_000023_0 was removed. > > at > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.removeSlot(SlotManager.java:899) > > at > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.removeSlots(SlotManager.java:869) > > at > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.internalUnregisterTaskManager(SlotManager.java:1080) > > at > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.unregisterTaskManager(SlotManager.java:391) > > at > org.apache.flink.runtime.resourcemanager.ResourceManager.closeTaskManagerConnection(ResourceManager.java:845) > > at > org.apache.flink.runtime.resourcemanager.ResourceManager$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(ResourceManager.java:1187) > > at > org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl$HeartbeatMonitor.run(HeartbeatManagerImpl.java:318) > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:392) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:185) > > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:147) > > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40) > > at > akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165) > > at akka.actor.Actor$class.aroundReceive(Actor.scala:502) > > at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95) > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) > > at akka.actor.ActorCell.invoke(ActorCell.scala:495) > > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) > > at akka.dispatch.Mailbox.run(Mailbox.scala:224) > > at akka.dispatch.Mailbox.exec(Mailbox.scala:234) > > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > > at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > > > hs_err_pid_262348.log for failed container > > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x00007f294944b2c2, pid=262348, > tid=0x00007f2916833700 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_112-b15) (build > 1.8.0_112-b15) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode > linux-amd64 compressed oops) > # Problematic frame: > # C [libzip.so+0xb2c2] inflateEnd+0x32 > # > # Core dump written. Default location: > /data/hadoop/yarn/local/usercache/flink/appcache/application_1568017536744_0044/container_e67_1568017536744_0044_01_000023/core > or core.262348 > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > > --------------- T H R E A D --------------- > > Current thread (0x00007f29440e8000): JavaThread "Finalizer" daemon > [_thread_in_native, id=262401, stack(0x00007f2916733000,0x00007f2916834000)] > > siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: > 0x0000000000001080 > > Registers: > RAX=0x00007f0100000001, RBX=0x00007f2945e52770, RCX=0x0000000000000180, > RDX=0x00007f2945e52770 > RSP=0x00007f29168323d0, RBP=0x00007f29168323e0, RSI=0x0000000000001040, > RDI=0x00007f2945e52770 > R8 =0x00000007bff0f170, R9 =0x0000000000000006, R10=0x00007f2935017a08, > R11=0x00007f294b583d50 > R12=0x00007f29440e81f8, R13=0x00007f293135cc58, R14=0x00007f2916832490, > R15=0x00007f29440e8000 > RIP=0x00007f294944b2c2, EFLAGS=0x0000000000010202, > CSGSFS=0x0000000000000033, ERR=0x0000000000000004 > TRAPNO=0x000000000000000e > > Top of Stack: (sp=0x00007f29168323d0) > 0x00007f29168323d0: ffffffff440e8000 00007f2945e52770 > 0x00007f29168323e0: 00007f2916832400 00007f294944338e > 0x00007f29168323f0: 00007f293135cc58 0000000000000000 > 0x00007f2916832400: 00007f2916832468 00007f2935017a34 > 0x00007f2916832410: 00007f2916832540 00007f293501306d > 0x00007f2916832420: 00007f29350055d0 00007f2916832428 > 0x00007f2916832430: 0000000000000000 00007f2916832490 > 0x00007f2916832440: 00007f293135cd70 0000000000000000 > 0x00007f2916832450: 00007f293135cc58 0000000000000000 > 0x00007f2916832460: 00007f2916832488 00007f29168324e8 > 0x00007f2916832470: 00007f29350082bd 00000006ab616900 > 0x00007f2916832480: 00007f2935011538 00007f2945e52770 > 0x00007f2916832490: 00000007bff0f1e8 00000007bff0f1e8 > 0x00007f29168324a0: 00000007bff0f1e8 00007f2916832498 > 0x00007f29168324b0: 00007f293135c5e5 00007f2916832518 > 0x00007f29168324c0: 00007f293135cd70 00007f29313f9840 > 0x00007f29168324d0: 00007f293135c618 00007f2916832488 > 0x00007f29168324e0: 00007f2916832518 00007f2916832580 > 0x00007f29168324f0: 00007f29350082bd 0000000000000000 > 0x00007f2916832500: 00007f2945e52770 0000000000000000 > 0x00007f2916832510: 00000007bff0f1e8 00000007bff0cd38 > 0x00007f2916832520: 0000000000000009 00000007bff0f158 > 0x00007f2916832530: 0000006ce4720709 00000007bff0cd98 > 0x00007f2916832540: 00007f2916832520 00007f293132f631 > 0x00007f2916832550: 00007f29168325d8 00007f2931330ce0 > 0x00007f2916832560: 0000000000000000 00007f293132f6c0 > 0x00007f2916832570: 00007f2916832518 00007f29168325d8 > 0x00007f2916832580: 00007f2916832620 00007f29350082bd > 0x00007f2916832590: 0000000000000000 0000000000000000 > 0x00007f29168325a0: 0000000000000000 0000000000000000 > 0x00007f29168325b0: 0000000000000000 0000000000000000 > 0x00007f29168325c0: 00000007bff0f158 00000007bff0cd38 > > Instructions: (pc=0x00007f294944b2c2) > 0x00007f294944b2a2: fe ff ff ff 48 83 c4 08 5b c9 c3 0f 1f 00 48 8b > 0x00007f294944b2b2: 77 28 48 85 f6 74 e8 48 8b 47 38 48 85 c0 74 df > 0x00007f294944b2c2: 48 8b 56 40 48 85 d2 74 11 48 89 d6 48 8b 7f 40 > 0x00007f294944b2d2: ff d0 48 8b 43 38 48 8b 73 28 48 8b 7b 40 ff d0 > > Register to memory mapping: > > RAX=0x00007f0100000001 is an unknown value > RBX=0x00007f2945e52770 is an unknown value > RCX=0x0000000000000180 is an unknown value > RDX=0x00007f2945e52770 is an unknown value > RSP=0x00007f29168323d0 is pointing into the stack for thread: > 0x00007f29440e8000 > RBP=0x00007f29168323e0 is pointing into the stack for thread: > 0x00007f29440e8000 > RSI=0x0000000000001040 is an unknown value > RDI=0x00007f2945e52770 is an unknown value > R8 =0x00000007bff0f170 is an oop > [Ljava.lang.Object; > - klass: 'java/lang/Object'[] > - length: 16 > R9 =0x0000000000000006 is an unknown value > R10=0x00007f2935017a08 is at code_begin+808 in an Interpreter codelet > method entry point (kind = native) [0x00007f29350176e0, > 0x00007f2935017fe0] 2304 bytes > R11=0x00007f294b583d50: <offset 0x9c3d50> in > /usr/jdk64/jdk1.8.0_112/jre/lib/amd64/server/libjvm.so at 0x00007f294abc0000 > R12=0x00007f29440e81f8 is an unknown value > R13={method} {0x00007f293135cc58} 'end' '(J)V' in 'java/util/zip/Inflater' > R14=0x00007f2916832490 is pointing into the stack for thread: > 0x00007f29440e8000 > R15=0x00007f29440e8000 is a thread > > > Stack: [0x00007f2916733000,0x00007f2916834000], sp=0x00007f29168323d0, > free space=1020k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [libzip.so+0xb2c2] inflateEnd+0x32 > C [libzip.so+0x338e] Java_java_util_zip_Inflater_end+0x1e > j java.util.zip.Inflater.end(J)V+0 > j java.util.zip.Inflater.end()V+29 > j java.util.zip.ZipFile.close()V+169 > j sun.net.www.protocol.jar.URLJarFile.close()V+18 > j sun.net.www.protocol.jar.URLJarFile.finalize()V+1 > J 9535% C2 java.lang.ref.Finalizer$FinalizerThread.run()V (55 bytes) @ > 0x00007f293674cec0 [0x00007f293674cc00+0x2c0] > v ~StubRoutines::call_stub > V [libjvm.so+0x690c66] JavaCalls::call_helper(JavaValue*, methodHandle*, > JavaCallArguments*, Thread*)+0x1056 > V [libjvm.so+0x691171] JavaCalls::call_virtual(JavaValue*, KlassHandle, > Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321 > V [libjvm.so+0x691617] JavaCalls::call_virtual(JavaValue*, Handle, > KlassHandle, Symbol*, Symbol*, Thread*)+0x47 > V [libjvm.so+0x72c990] thread_entry(JavaThread*, Thread*)+0xa0 > V [libjvm.so+0xa755f3] JavaThread::thread_main_inner()+0x103 > V [libjvm.so+0xa7573c] JavaThread::run()+0x11c > V [libjvm.so+0x926138] java_start(Thread*)+0x108 > C [libpthread.so.0+0x7e25] start_thread+0xc5 > > Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) > j java.util.zip.Inflater.end(J)V+0 > j java.util.zip.Inflater.end()V+29 > j java.util.zip.ZipFile.close()V+169 > j sun.net.www.protocol.jar.URLJarFile.close()V+18 > j sun.net.www.protocol.jar.URLJarFile.finalize()V+1 > J 9535% C2 java.lang.ref.Finalizer$FinalizerThread.run()V (55 bytes) @ > 0x00007f293674cec0 [0x00007f293674cc00+0x2c0] > v ~StubRoutines::call_stub >
