Re: [External] Regarding java.lang.IllegalStateException

2024-04-26 Thread Maxim Senin via user 
My guess it’s a major known issue. Need a workaround.

https://issues.apache.org/jira/browse/FLINK-32212

/Maxim

From: prashant parbhane 
Date: Tuesday, April 23, 2024 at 11:09 PM
To: user@flink.apache.org 
Subject: [External] Regarding java.lang.IllegalStateException
Hello,

We have been facing this weird issue, where we are getting below exception and 
the job is getting restarted with new task managers. We are using flink 1.17.
Same job works fine with lower number of task managers. (<10)

java.lang.IllegalStateException: The library registration references a 
different set of library BLOBs than previous registrations for this job:
old:[p-ecf88f3c5c35842d7bca235cfd7592c53f9fdbe0-e06122dd51cccba866932318dc031d68]
new:[p-ecf88f3c5c35842d7bca235cfd7592c53f9fdbe0-7651f69109c915ac830aa42ce2ab67f0]
at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:419)
at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$500(BlobLibraryCacheManager.java:359)
at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:235)
at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1100(BlobLibraryCacheManager.java:202)
at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:336)
at 
org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:1041)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:624)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
at java.base/java.lang.Thread.run(Unknown Source)

Thanks,
Prashant



COGILITY SOFTWARE CORPORATION LEGAL DISCLAIMER: The information in this email 
is confidential and is intended solely for the addressee. Access to this email 
by anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful.

Re: Regarding java.lang.IllegalStateException

2024-04-26 Thread Maxim Senin via user 
We are also seeing something similar:

2024-04-26 16:30:44,401 INFO  
org.apache.flink.runtime.executiongraph.ExecutionGraph   [] - Source: Power 
Consumption:power_consumption -> Ingest Power Consumption -> PopSysFields -> 
WindowingWatermarkPreCheck (1/1) 
(cb8c425b6463b1ade9b8359c0514386b_28bc590bb7896e1df191c306d7cb6d23_0_11) 
switched from DEPLOYING to FAILED on 
f-a9ad4438-cddf-512f-94c6-c5f921f66078-taskmanager-1-1 @ 10.111.164.30 
(dataPort=42621).
java.lang.IllegalStateException: The library registration references a 
different set of library BLOBs than previous registrations for this job:
old:[p-1b01ac6374a137939ffa18432714b7c9af30dc3f-522999dc4412a76d33728a97225de573]
new:[p-1b01ac6374a137939ffa18432714b7c9af30dc3f-819e2ecffa2f22c5e0f4d88ef5789421]
  at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:437)
 ~[flink-dist-1.19.0.jar:1.19.0]
  at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$600(BlobLibraryCacheManager.java:373)
 ~[flink-dist-1.19.0.jar:1.19.0]
  at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:249)
 ~[flink-dist-1.19.0.jar:1.19.0]
  at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:210)
 ~[flink-dist-1.19.0.jar:1.19.0]
  at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:350)
 ~[flink-dist-1.19.0.jar:1.19.0]
  at 
org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:1047)
 ~[flink-dist-1.19.0.jar:1.19.0]
  at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:628) 
~[flink-dist-1.19.0.jar:1.19.0]
  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) 
~[flink-dist-1.19.0.jar:1.19.0]
  at java.lang.Thread.run(Unknown Source) ~[?:?]
2024-04-26 16:30:44,402 INFO  
org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler [] - Restarting 
job.
java.lang.IllegalStateException: The library registration references a 
different set of library BLOBs than previous registrations for this job:
old:[p-1b01ac6374a137939ffa18432714b7c9af30dc3f-522999dc4412a76d33728a97225de573]
new:[p-1b01ac6374a137939ffa18432714b7c9af30dc3f-819e2ecffa2f22c5e0f4d88ef5789421]
  at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:437)
 ~[flink-dist-1.19.0.jar:1.19.0]
  at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$600(BlobLibraryCacheManager.java:373)
 ~[flink-dist-1.19.0.jar:1.19.0]
  at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:249)
 ~[flink-dist-1.19.0.jar:1.19.0]
  at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:210)
 ~[flink-dist-1.19.0.jar:1.19.0]
  at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:350)
 ~[flink-dist-1.19.0.jar:1.19.0]
  at 
org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:1047)
 ~[flink-dist-1.19.0.jar:1.19.0]
  at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:628) 
~[flink-dist-1.19.0.jar:1.19.0]
  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) 
~[flink-dist-1.19.0.jar:1.19.0]
  at java.lang.Thread.run(Unknown Source) ~[?:?]

Flink 1.19, operator 1.9.0


. . . . . . . . . . . . . . . . . . . . .
  Maxim Senin
   Senior Backend Engineer
 COGILITY<http://cogility.com/>

From: prashant parbhane 
Date: Tuesday, April 23, 2024 at 11:09 PM
To: user@flink.apache.org 
Subject: Regarding java.lang.IllegalStateException
Hello,

We have been facing this weird issue, where we are getting below exception and 
the job is getting restarted with new task managers. We are using flink 1.17.
Same job works fine with lower number of task managers. (<10)

java.lang.IllegalStateException: The library registration references a 
different set of library BLOBs than previous registrations for this job:
old:[p-ecf88f3c5c35842d7bca235cfd7592c53f9fdbe0-e06122dd51cccba866932318dc031d68]
new:[p-ecf88f3c5c35842d7bca235cfd7592c53f9fdbe0-7651f69109c915ac830aa42ce2ab67f0]
at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(BlobLibraryCacheManager.java:419)
at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$ResolvedClassLoader.access$500(BlobLibraryCacheManage

Regarding java.lang.IllegalStateException

2024-04-23 Thread prashant parbhane 
Hello,

We have been facing this weird issue, where we are getting below
exception and the job is getting restarted with new task managers. We are
using flink 1.17.
Same job works fine with lower number of task managers. (<10)

java.lang.IllegalStateException: The library registration references a
different set of library BLOBs than previous registrations for this job:
old:[p-ecf88f3c5c35842d7bca235cfd7592c53f9fdbe0-e06122dd51cccba866932318dc031d68]
new:[p-ecf88f3c5c35842d7bca235cfd7592c53f9fdbe0-7651f69109
c915ac830aa42ce2ab67f0]
at org.apache.flink.runtime.execution.librarycache.
BlobLibraryCacheManager$ResolvedClassLoader.verifyClassLoader(
BlobLibraryCacheManager.java:419)
at org.apache.flink.runtime.execution.librarycache.
BlobLibraryCacheManager$ResolvedClassLoader.access$500(
BlobLibraryCacheManager.java:359)
at org.apache.flink.runtime.execution.librarycache.
BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(
BlobLibraryCacheManager.java:235)
at org.apache.flink.runtime.execution.librarycache.
BlobLibraryCacheManager$LibraryCacheEntry.access$1100(
BlobLibraryCacheManager.java:202)
at org.apache.flink.runtime.execution.librarycache.
BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(
BlobLibraryCacheManager.java:336)
at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task
.java:1041)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:624)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
at java.base/java.lang.Thread.run(Unknown Source)

Thanks,
Prashant