Riddle4045 commented on issue #9435:
URL: https://github.com/apache/hudi/issues/9435#issuecomment-1687220558
@danny0405 I looked at the JM logs and it looks like the same issue about
`calcite/plan/RelOptRule` not being found, what's interesting is that even when
I manually ADD JAR to my SQL Client I am seeing this exception - does classpath
for compaction task differ from the job at all?
StackTracke
```
2023-08-21 23:56:26.647 [] flink-akka.actor.default-dispatcher-19 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 561 Deploying Source:
Values[1] -> Calc[2] -> row_data_to_hoodie_record (1/1) (attempt #0) with
attempt id
1ce0c00a771c0214a3141bd885bcfdb0_cbc357ccb763df2852fee8c4fc7d55f2_0_0 and
vertex id cbc357ccb763df2852fee8c4fc7d55f2_0 to 10.244.7.3:6122-d80422 @
10.244.7.3 (dataPort=40143) with allocation id 2786e9f15bcf024e1208268a4eb1a9f4
2023-08-21 23:56:26.656 [] flink-akka.actor.default-dispatcher-19 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 bucket_assigner
(1/1) (1ce0c00a771c0214a3141bd885bcfdb0_e5df0a348bd9bf84b2755743802fc12d_0_0)
switched from SCHEDULED to DEPLOYING.
2023-08-21 23:56:26.656 [] flink-akka.actor.default-dispatcher-19 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 561 Deploying
bucket_assigner (1/1) (attempt #0) with attempt id
1ce0c00a771c0214a3141bd885bcfdb0_e5df0a348bd9bf84b2755743802fc12d_0_0 and
vertex id e5df0a348bd9bf84b2755743802fc12d_0 to 10.244.7.3:6122-d80422 @
10.244.7.3 (dataPort=40143) with allocation id 2786e9f15bcf024e1208268a4eb1a9f4
2023-08-21 23:56:26.656 [] flink-akka.actor.default-dispatcher-19 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 stream_write: mor
(1/1) (1ce0c00a771c0214a3141bd885bcfdb0_de00fca8057a992b9689a76776a87e88_0_0)
switched from SCHEDULED to DEPLOYING.
2023-08-21 23:56:26.657 [] flink-akka.actor.default-dispatcher-19 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 561 Deploying
stream_write: mor (1/1) (attempt #0) with attempt id
1ce0c00a771c0214a3141bd885bcfdb0_de00fca8057a992b9689a76776a87e88_0_0 and
vertex id de00fca8057a992b9689a76776a87e88_0 to 10.244.7.3:6122-d80422 @
10.244.7.3 (dataPort=40143) with allocation id 2786e9f15bcf024e1208268a4eb1a9f4
2023-08-21 23:56:26.657 [] flink-akka.actor.default-dispatcher-19 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435
compact_plan_generate (1/1)
(1ce0c00a771c0214a3141bd885bcfdb0_454d170138cbd567f9c11b29e690b935_0_0)
switched from SCHEDULED to DEPLOYING.
2023-08-21 23:56:26.657 [] flink-akka.actor.default-dispatcher-19 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 561 Deploying
compact_plan_generate (1/1) (attempt #0) with attempt id
1ce0c00a771c0214a3141bd885bcfdb0_454d170138cbd567f9c11b29e690b935_0_0 and
vertex id 454d170138cbd567f9c11b29e690b935_0 to 10.244.7.3:6122-d80422 @
10.244.7.3 (dataPort=40143) with allocation id 2786e9f15bcf024e1208268a4eb1a9f4
2023-08-21 23:56:26.657 [] flink-akka.actor.default-dispatcher-19 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 compact_task ->
Sink: compact_commit (1/1)
(1ce0c00a771c0214a3141bd885bcfdb0_848dfd76319f29eb5fbacd6c182b1d35_0_0)
switched from SCHEDULED to DEPLOYING.
2023-08-21 23:56:26.657 [] flink-akka.actor.default-dispatcher-19 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 561 Deploying
compact_task -> Sink: compact_commit (1/1) (attempt #0) with attempt id
1ce0c00a771c0214a3141bd885bcfdb0_848dfd76319f29eb5fbacd6c182b1d35_0_0 and
vertex id 848dfd76319f29eb5fbacd6c182b1d35_0 to 10.244.7.3:6122-d80422 @
10.244.7.3 (dataPort=40143) with allocation id 2786e9f15bcf024e1208268a4eb1a9f4
2023-08-21 23:56:28.173 [] flink-akka.actor.default-dispatcher-15 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 stream_write: mor
(1/1) (1ce0c00a771c0214a3141bd885bcfdb0_de00fca8057a992b9689a76776a87e88_0_0)
switched from DEPLOYING to INITIALIZING.
2023-08-21 23:56:28.173 [] flink-akka.actor.default-dispatcher-15 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 compact_task ->
Sink: compact_commit (1/1)
(1ce0c00a771c0214a3141bd885bcfdb0_848dfd76319f29eb5fbacd6c182b1d35_0_0)
switched from DEPLOYING to INITIALIZING.
2023-08-21 23:56:28.187 [] flink-akka.actor.default-dispatcher-14 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435
compact_plan_generate (1/1)
(1ce0c00a771c0214a3141bd885bcfdb0_454d170138cbd567f9c11b29e690b935_0_0)
switched from DEPLOYING to INITIALIZING.
2023-08-21 23:56:28.187 [] flink-akka.actor.default-dispatcher-14 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 Source: Values[1]
-> Calc[2] -> row_data_to_hoodie_record (1/1)
(1ce0c00a771c0214a3141bd885bcfdb0_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
switched from DEPLOYING to INITIALIZING.
2023-08-21 23:56:28.188 [] flink-akka.actor.default-dispatcher-14 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 bucket_assigner
(1/1) (1ce0c00a771c0214a3141bd885bcfdb0_e5df0a348bd9bf84b2755743802fc12d_0_0)
switched from DEPLOYING to INITIALIZING.
2023-08-21 23:56:28.305 [] flink-akka.actor.default-dispatcher-15 WARN
flink apache.flink.runtime.taskmanager.TaskManagerLocation 267 No hostname
could be resolved for the IP address 10.244.7.3, using IP address as host name.
Local input split assignment (such as for HDFS files) may be impacted.
2023-08-21 23:56:28.307 [] flink-akka.actor.default-dispatcher-14 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 Source: Values[1]
-> Calc[2] -> row_data_to_hoodie_record (1/1)
(1ce0c00a771c0214a3141bd885bcfdb0_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
switched from INITIALIZING to RUNNING.
2023-08-21 23:56:28.329 [] flink-akka.actor.default-dispatcher-14 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 Source: Values[1]
-> Calc[2] -> row_data_to_hoodie_record (1/1)
(1ce0c00a771c0214a3141bd885bcfdb0_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
switched from RUNNING to FINISHED.
2023-08-21 23:56:28.605 [] flink-akka.actor.default-dispatcher-15 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435
compact_plan_generate (1/1)
(1ce0c00a771c0214a3141bd885bcfdb0_454d170138cbd567f9c11b29e690b935_0_0)
switched from INITIALIZING to RUNNING.
2023-08-21 23:56:28.806 [] JettyServerThreadPool-8637 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 130 Loading
HoodieTableMetaClient from abfs://fl...@test.dfs.core.windows.net/hudi/mor
2023-08-21 23:56:28.849 [] JettyServerThreadPool-8637 INFO flink
apache.hudi.common.table.HoodieTableConfig 268 Loading table properties from
abfs://fl...@test.dfs.core.windows.net/hudi/mor/.hoodie/hoodie.properties
2023-08-21 23:56:28.876 [] JettyServerThreadPool-8637 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 149 Finished Loading Table of
type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from
abfs://fl...@test.dfs.core.windows.net/hudi/mor
2023-08-21 23:56:28.876 [] JettyServerThreadPool-8637 INFO flink
hudi.common.table.view.FileSystemViewManager 165 Creating InMemory based view
for basePath abfs://fl...@test.dfs.core.windows.net/hudi/mor
2023-08-21 23:56:28.896 [] JettyServerThreadPool-8637 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 171 Loaded instants upto :
Option{val=[20230821235145161__deltacommit__COMPLETED]}
2023-08-21 23:56:28.899 [] JettyServerThreadPool-8637 INFO flink
hudi.common.table.view.AbstractTableFileSystemView 252 Took 0 ms to read 0
instants, 0 replaced file groups
2023-08-21 23:56:28.926 [] JettyServerThreadPool-8637 INFO flink
apache.hudi.common.util.ClusteringUtils 139 Found 0 files in pending clustering
operations
2023-08-21 23:56:30.251 [] flink-akka.actor.default-dispatcher-15 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 compact_task ->
Sink: compact_commit (1/1)
(1ce0c00a771c0214a3141bd885bcfdb0_848dfd76319f29eb5fbacd6c182b1d35_0_0)
switched from INITIALIZING to RUNNING.
2023-08-21 23:56:30.326 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 130 Loading
HoodieTableMetaClient from abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:30.326 [] flink-akka.actor.default-dispatcher-15 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 stream_write: mor
(1/1) (1ce0c00a771c0214a3141bd885bcfdb0_de00fca8057a992b9689a76776a87e88_0_0)
switched from INITIALIZING to RUNNING.
2023-08-21 23:56:30.345 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableConfig 268 Loading table properties from
abfs://fl...@test.dfs.core.windows.net/hudi/mor/.hoodie/hoodie.properties
2023-08-21 23:56:30.371 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 149 Finished Loading Table of
type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from
abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:30.392 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 171 Loaded instants upto :
Option{val=[20230821235145161__deltacommit__COMPLETED]}
2023-08-21 23:56:30.392 [] pool-5111-thread-1 INFO flink
org.apache.hudi.sink.StreamWriteOperatorCoordinator 131 Executor executes
action [handle write metadata event for instant ] success!
2023-08-21 23:56:30.393 [] pool-5111-thread-1 INFO flink
apache.hudi.common.util.CleanerUtils 155 Cleaned failed attempts if any
2023-08-21 23:56:30.394 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 130 Loading
HoodieTableMetaClient from abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:30.405 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableConfig 268 Loading table properties from
abfs://fl...@test.dfs.core.windows.net/hudi/mor/.hoodie/hoodie.properties
2023-08-21 23:56:30.428 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 149 Finished Loading Table of
type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from
abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:30.428 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 152 Loading Active commit
timeline for abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:30.447 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 171 Loaded instants upto :
Option{val=[20230821235145161__deltacommit__COMPLETED]}
2023-08-21 23:56:30.448 [] pool-5111-thread-1 INFO flink
hudi.common.table.view.FileSystemViewManager 245 Creating View Manager with
storage type :REMOTE_FIRST
2023-08-21 23:56:30.448 [] pool-5111-thread-1 INFO flink
hudi.common.table.view.FileSystemViewManager 265 Creating remote first table
view
2023-08-21 23:56:30.450 [] pool-5111-thread-1 INFO flink
org.apache.hudi.client.BaseHoodieWriteClient 841 Generate a new instant time:
20230821235630450 action: deltacommit
2023-08-21 23:56:30.471 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 171 Loaded instants upto :
Option{val=[20230821235145161__deltacommit__COMPLETED]}
2023-08-21 23:56:30.471 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 202 Creating a new instant
[==>20230821235630450__deltacommit__REQUESTED]
2023-08-21 23:56:30.527 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 632 Checking for file exists
?abfs://fl...@test.dfs.core.windows.net/hudi/mor/.hoodie/20230821235630450.deltacommit.requested
2023-08-21 23:56:30.595 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 640 Create new file for
toInstant
?abfs://fl...@test.dfs.core.windows.net/hudi/mor/.hoodie/20230821235630450.deltacommit.inflight
2023-08-21 23:56:30.651 [] flink-akka.actor.default-dispatcher-14 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 bucket_assigner
(1/1) (1ce0c00a771c0214a3141bd885bcfdb0_e5df0a348bd9bf84b2755743802fc12d_0_0)
switched from INITIALIZING to RUNNING.
2023-08-21 23:56:30.658 [] JettyServerThreadPool-8636 INFO flink
hudi.common.table.view.AbstractTableFileSystemView 418 Building file system
view for partition ()
2023-08-21 23:56:30.688 [] pool-5111-thread-1 INFO flink
org.apache.hudi.sink.StreamWriteOperatorCoordinator 380 Create instant
[20230821235630450] for table [mor] with type [MERGE_ON_READ]
2023-08-21 23:56:30.689 [] pool-5111-thread-1 INFO flink
org.apache.hudi.sink.StreamWriteOperatorCoordinator 131 Executor executes
action [initialize instant ] success!
2023-08-21 23:56:30.730 [] JettyServerThreadPool-8636 INFO flink
hudi.common.table.view.AbstractTableFileSystemView 160 addFilesToView:
NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=3, StoreTimeTaken=0
2023-08-21 23:56:30.775 [] flink-akka.actor.default-dispatcher-15 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 bucket_assigner
(1/1) (1ce0c00a771c0214a3141bd885bcfdb0_e5df0a348bd9bf84b2755743802fc12d_0_0)
switched from RUNNING to FINISHED.
2023-08-21 23:56:31.705 [] pool-5111-thread-1 INFO flink
org.apache.hudi.client.BaseHoodieWriteClient 219 Committing 20230821235630450
action deltacommit
2023-08-21 23:56:31.705 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 130 Loading
HoodieTableMetaClient from abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:31.718 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableConfig 268 Loading table properties from
abfs://fl...@test.dfs.core.windows.net/hudi/mor/.hoodie/hoodie.properties
2023-08-21 23:56:31.744 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 149 Finished Loading Table of
type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from
abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:31.744 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 152 Loading Active commit
timeline for abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:31.769 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 171 Loaded instants upto :
Option{val=[==>20230821235630450__deltacommit__INFLIGHT]}
2023-08-21 23:56:31.769 [] pool-5111-thread-1 INFO flink
hudi.common.table.view.FileSystemViewManager 245 Creating View Manager with
storage type :REMOTE_FIRST
2023-08-21 23:56:31.769 [] pool-5111-thread-1 INFO flink
hudi.common.table.view.FileSystemViewManager 265 Creating remote first table
view
2023-08-21 23:56:31.770 [] pool-5111-thread-1 INFO flink
apache.hudi.common.util.CommitUtils 115 Creating metadata for UPSERT
numWriteStats:1 numReplaceFileIds:0
2023-08-21 23:56:31.770 [] pool-5111-thread-1 INFO flink
org.apache.hudi.client.BaseHoodieWriteClient 272 Committing 20230821235630450
action deltacommit
2023-08-21 23:56:31.812 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 130 Loading
HoodieTableMetaClient from abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:31.831 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableConfig 268 Loading table properties from
abfs://fl...@test.dfs.core.windows.net/hudi/mor/.hoodie/hoodie.properties
2023-08-21 23:56:31.859 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 149 Finished Loading Table of
type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from
abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:31.859 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 152 Loading Active commit
timeline for abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:31.878 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 171 Loaded instants upto :
Option{val=[==>20230821235630450__deltacommit__INFLIGHT]}
2023-08-21 23:56:31.878 [] pool-5111-thread-1 INFO flink
hudi.common.table.view.FileSystemViewManager 245 Creating View Manager with
storage type :REMOTE_FIRST
2023-08-21 23:56:31.878 [] pool-5111-thread-1 INFO flink
hudi.common.table.view.FileSystemViewManager 265 Creating remote first table
view
2023-08-21 23:56:31.888 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 220 Marking instant complete
[==>20230821235630450__deltacommit__INFLIGHT]
2023-08-21 23:56:31.888 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 632 Checking for file exists
?abfs://fl...@test.dfs.core.windows.net/hudi/mor/.hoodie/20230821235630450.deltacommit.inflight
2023-08-21 23:56:31.982 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 640 Create new file for
toInstant
?abfs://fl...@test.dfs.core.windows.net/hudi/mor/.hoodie/20230821235630450.deltacommit
2023-08-21 23:56:31.982 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 224 Completed
[==>20230821235630450__deltacommit__INFLIGHT]
2023-08-21 23:56:31.982 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 130 Loading
HoodieTableMetaClient from abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:31.993 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableConfig 268 Loading table properties from
abfs://fl...@test.dfs.core.windows.net/hudi/mor/.hoodie/hoodie.properties
2023-08-21 23:56:32.018 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 149 Finished Loading Table of
type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from
abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:32.018 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 152 Loading Active commit
timeline for abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:32.036 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 171 Loaded instants upto :
Option{val=[20230821235630450__deltacommit__COMPLETED]}
2023-08-21 23:56:32.036 [] pool-5111-thread-1 INFO flink
hudi.common.table.view.FileSystemViewManager 245 Creating View Manager with
storage type :REMOTE_FIRST
2023-08-21 23:56:32.037 [] pool-5111-thread-1 INFO flink
hudi.common.table.view.FileSystemViewManager 265 Creating remote first table
view
2023-08-21 23:56:32.297 [] pool-5111-thread-1 INFO flink
apache.hudi.common.fs.FSUtils 732 Removed directory at
abfs://fl...@test.dfs.core.windows.net/hudi/mor/.hoodie/.temp/20230821235630450
2023-08-21 23:56:32.297 [] pool-5111-thread-1 INFO flink
org.apache.hudi.client.BaseHoodieWriteClient 235 Committed 20230821235630450
2023-08-21 23:56:32.298 [] pool-5111-thread-1 INFO flink
org.apache.hudi.client.BaseHoodieWriteClient 571 Start to archive synchronously.
2023-08-21 23:56:32.321 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 171 Loaded instants upto :
Option{val=[20230821235630450__deltacommit__COMPLETED]}
2023-08-21 23:56:32.321 [] pool-5111-thread-1 INFO flink
org.apache.hudi.client.HoodieTimelineArchiver 179 No Instants to archive
2023-08-21 23:56:32.426 [] pool-5111-thread-1 INFO flink
org.apache.hudi.sink.StreamWriteOperatorCoordinator 542 Commit instant
[20230821235630450] success!
2023-08-21 23:56:32.431 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 130 Loading
HoodieTableMetaClient from abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:32.445 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableConfig 268 Loading table properties from
abfs://fl...@test.dfs.core.windows.net/hudi/mor/.hoodie/hoodie.properties
2023-08-21 23:56:32.475 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 149 Finished Loading Table of
type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from
abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:32.475 [] pool-5111-thread-1 INFO flink
apache.hudi.common.table.HoodieTableMetaClient 152 Loading Active commit
timeline for abfs://fl...@test.dfs.core.windows.net/hudi/mor/
2023-08-21 23:56:32.497 [] pool-5111-thread-1 INFO flink
hudi.common.table.timeline.HoodieActiveTimeline 171 Loaded instants upto :
Option{val=[20230821235630450__deltacommit__COMPLETED]}
2023-08-21 23:56:32.501 [] pool-5111-thread-1 ERROR flink
org.apache.hudi.sink.StreamWriteOperatorCoordinator 140 Executor executes
action [handle end input event for instant 20230821235630450] error
java.lang.NoClassDefFoundError: org/apache/calcite/plan/RelOptRule
at java.lang.Class.getDeclaredMethods0(Native Method) ~[?:?]
at java.lang.Class.privateGetDeclaredMethods(Unknown Source) ~[?:?]
at java.lang.Class.getMethodsRecursive(Unknown Source) ~[?:?]
at java.lang.Class.getMethod0(Unknown Source) ~[?:?]
at java.lang.Class.getMethod(Unknown Source) ~[?:?]
at
org.apache.hudi.hive.ddl.HMSDDLExecutor.<init>(HMSDDLExecutor.java:86)
~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-cd8c32c0bf152df1c672bb269a800586:0.13.0]
at
org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:87)
~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-cd8c32c0bf152df1c672bb269a800586:0.13.0]
at
org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:119)
~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-cd8c32c0bf152df1c672bb269a800586:0.13.0]
at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:113)
~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-cd8c32c0bf152df1c672bb269a800586:0.13.0]
at
org.apache.hudi.sink.utils.HiveSyncContext.hiveSyncTool(HiveSyncContext.java:81)
~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-cd8c32c0bf152df1c672bb269a800586:0.13.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:338)
~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-cd8c32c0bf152df1c672bb269a800586:0.13.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.syncHive(StreamWriteOperatorCoordinator.java:329)
~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-cd8c32c0bf152df1c672bb269a800586:0.13.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.handleEndInputEvent(StreamWriteOperatorCoordinator.java:433)
~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-cd8c32c0bf152df1c672bb269a800586:0.13.0]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$handleEventFromOperator$3(StreamWriteOperatorCoordinator.java:281)
~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-cd8c32c0bf152df1c672bb269a800586:0.13.0]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)
~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-cd8c32c0bf152df1c672bb269a800586:0.13.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.ClassNotFoundException:
org.apache.calcite.plan.RelOptRule
at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
~[?:?]
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown
Source) ~[?:?]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
... 20 more
2023-08-21 23:56:32.501 [] flink-akka.actor.default-dispatcher-15 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 stream_write: mor
(1/1) (1ce0c00a771c0214a3141bd885bcfdb0_de00fca8057a992b9689a76776a87e88_0_0)
switched from RUNNING to FINISHED.
2023-08-21 23:56:32.501 [] flink-akka.actor.default-dispatcher-15 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435
compact_plan_generate (1/1)
(1ce0c00a771c0214a3141bd885bcfdb0_454d170138cbd567f9c11b29e690b935_0_0)
switched from RUNNING to FINISHED.
2023-08-21 23:56:32.501 [] flink-akka.actor.default-dispatcher-15 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 compact_task ->
Sink: compact_commit (1/1)
(1ce0c00a771c0214a3141bd885bcfdb0_848dfd76319f29eb5fbacd6c182b1d35_0_0)
switched from RUNNING to FINISHED.
2023-08-21 23:56:32.502 [] flink-akka.actor.default-dispatcher-14 INFO
flink flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager 292
Clearing resource requirements of job b38f5c29d67259e28847e178a6145495
2023-08-21 23:56:32.502 [] flink-akka.actor.default-dispatcher-15 INFO
flink apache.flink.runtime.executiongraph.ExecutionGraph 1152 Job
insert-into_hudi_catalog.hudicatalogdb.mor (b38f5c29d67259e28847e178a6145495)
switched from state RUNNING to FINISHED.
2023-08-21 23:56:32.502 [] flink-akka.actor.default-dispatcher-15 INFO
flink apache.flink.runtime.checkpoint.CheckpointCoordinator 411 Stopping
checkpoint coordinator for job b38f5c29d67259e28847e178a6145495.
2023-08-21 23:56:32.503 [] flink-akka.actor.default-dispatcher-15 INFO
flink apache.flink.runtime.jobmaster.JobMaster 297 Trying to recover from a
global failure.
org.apache.flink.util.FlinkException: Global failure triggered by
OperatorCoordinator for 'stream_write: mor' (operator
de00fca8057a992b9689a76776a87e88).
at
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:617)
~[flink-dist-1.16.0-0.0.18.jar:1.16.0-0.0.18]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:190)
~[?:?]
at
org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:142)
~[?:?]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:133)
~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: org.apache.hudi.exception.HoodieException: Executor executes
action [handle end input event for instant 20230821235630450] error
... 8 more
Caused by: java.lang.NoClassDefFoundError: org/apache/calcite/plan/RelOptRule
at java.lang.Class.getDeclaredMethods0(Native Method) ~[?:?]
at java.lang.Class.privateGetDeclaredMethods(Unknown Source) ~[?:?]
at java.lang.Class.getMethodsRecursive(Unknown Source) ~[?:?]
at java.lang.Class.getMethod0(Unknown Source) ~[?:?]
at java.lang.Class.getMethod(Unknown Source) ~[?:?]
at
org.apache.hudi.hive.ddl.HMSDDLExecutor.<init>(HMSDDLExecutor.java:86) ~[?:?]
at
org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:87)
~[?:?]
at
org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:119) ~[?:?]
at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:113)
~[?:?]
at
org.apache.hudi.sink.utils.HiveSyncContext.hiveSyncTool(HiveSyncContext.java:81)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:338)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.syncHive(StreamWriteOperatorCoordinator.java:329)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.handleEndInputEvent(StreamWriteOperatorCoordinator.java:433)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$handleEventFromOperator$3(StreamWriteOperatorCoordinator.java:281)
~[?:?]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)
~[?:?]
... 5 more
Caused by: java.lang.ClassNotFoundException:
org.apache.calcite.plan.RelOptRule
at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
~[?:?]
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown
Source) ~[?:?]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
at java.lang.Class.getDeclaredMethods0(Native Method) ~[?:?]
at java.lang.Class.privateGetDeclaredMethods(Unknown Source) ~[?:?]
at java.lang.Class.getMethodsRecursive(Unknown Source) ~[?:?]
at java.lang.Class.getMethod0(Unknown Source) ~[?:?]
at java.lang.Class.getMethod(Unknown Source) ~[?:?]
at
org.apache.hudi.hive.ddl.HMSDDLExecutor.<init>(HMSDDLExecutor.java:86) ~[?:?]
at
org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:87)
~[?:?]
at
org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:119) ~[?:?]
at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:113)
~[?:?]
at
org.apache.hudi.sink.utils.HiveSyncContext.hiveSyncTool(HiveSyncContext.java:81)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:338)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.syncHive(StreamWriteOperatorCoordinator.java:329)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.handleEndInputEvent(StreamWriteOperatorCoordinator.java:433)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$handleEventFromOperator$3(StreamWriteOperatorCoordinator.java:281)
~[?:?]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)
~[?:?]
... 5 more
2023-08-21 23:56:32.510 [] flink-akka.actor.default-dispatcher-15 INFO
flink apache.flink.runtime.dispatcher.StandaloneDispatcher 1113 Job
b38f5c29d67259e28847e178a6145495 reached terminal state FINISHED.
2023-08-21 23:56:32.671 [] cluster-io-thread-4 INFO flink
apache.flink.runtime.history.FsJobArchivist 91 Job
b38f5c29d67259e28847e178a6145495 has been archived at
abfs://fl...@test.dfs.core.windows.net/completed-jobs/b38f5c29d67259e28847e178a6145495.
2023-08-21 23:56:32.820 [] cluster-io-thread-3 INFO flink
apache.flink.runtime.dispatcher.StandaloneDispatcher 1161 Job
b38f5c29d67259e28847e178a6145495 has been registered for cleanup in the
JobResultStore after reaching a terminal state.
2023-08-21 23:56:32.820 [] flink-akka.actor.default-dispatcher-13 INFO
flink apache.flink.runtime.jobmaster.JobMaster 434 Stopping the JobMaster for
job 'insert-into_hudi_catalog.hudicatalogdb.mor'
(b38f5c29d67259e28847e178a6145495).
2023-08-21 23:56:32.821 [] flink-akka.actor.default-dispatcher-13 INFO
flink org.apache.hudi.client.BaseHoodieClient 111 Stopping Timeline service !!
2023-08-21 23:56:32.821 [] flink-akka.actor.default-dispatcher-13 INFO
flink apache.hudi.client.embedded.EmbeddedTimelineService 151 Closing Timeline
server
2023-08-21 23:56:32.821 [] flink-akka.actor.default-dispatcher-13 INFO
flink apache.hudi.timeline.service.TimelineService 399 Closing Timeline Service
2023-08-21 23:56:32.821 [] flink-akka.actor.default-dispatcher-13 INFO
flink io.javalin.Javalin 22 Stopping Javalin ...
2023-08-21 23:56:32.826 [] flink-akka.actor.default-dispatcher-13 INFO
flink io.javalin.Javalin 22 Javalin has stopped
2023-08-21 23:56:32.826 [] flink-akka.actor.default-dispatcher-13 INFO
flink apache.hudi.timeline.service.TimelineService 408 Closed Timeline Service
2023-08-21 23:56:32.826 [] flink-akka.actor.default-dispatcher-13 INFO
flink apache.hudi.client.embedded.EmbeddedTimelineService 155 Closed Timeline
server
2023-08-21 23:56:32.827 [] flink-akka.actor.default-dispatcher-13 INFO
flink org.apache.hudi.client.BaseHoodieClient 111 Stopping Timeline service !!
2023-08-21 23:56:32.827 [] flink-akka.actor.default-dispatcher-13 INFO
flink apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore 182
Shutting down
2023-08-21 23:56:32.836 [] flink-akka.actor.default-dispatcher-13 INFO
flink apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter 82
Shutting down.
2023-08-21 23:56:32.836 [] flink-akka.actor.default-dispatcher-13 INFO
flink apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter 84
Removing counter from ConfigMap
flink-d064232b26594088ae78032391934433-b38f5c29d67259e28847e178a6145495-config-map
2023-08-21 23:56:32.845 [] flink-akka.actor.default-dispatcher-13 INFO
flink apache.flink.runtime.jobmaster.JobMaster 525 Disconnect TaskExecutor
10.244.7.3:6122-d80422 because: Stopping JobMaster for job
'insert-into_hudi_catalog.hudicatalogdb.mor' (b38f5c29d67259e28847e178a6145495).
```
Adding jar to the sql-client
my directory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
