[GitHub] [hudi] guyuqi commented on pull request #4617: HUDI-1657: build failed on AArch64, Fedora 33
guyuqi commented on pull request #4617:
URL: https://github.com/apache/hudi/pull/4617#issuecomment-1034692908
I put everything in a Fedora-33 docker container for I have no Fedora-33
host.
And then follow the [Quick Start (demo)
guide](https://github.com/apache/hudi/blob/master/hudi-kafka-connect/README.md):
But when add the Hudi Sink to the Connector:
`curl -X GET -H "Content-Type:application/json"
http://localhost:8083/connectors/hudi-sink/status | jq`
`hudi-sink` is running but `tasks `failed:
```
[builder@f7b3d84dbcab kafka_2.12-3.1.0]$ curl -X GET -H
"Content-Type:application/json"
http://localhost:8083/connectors/hudi-sink/status | jq
% Total% Received % Xferd Average Speed TimeTime Time
Current
Dload Upload Total SpentLeft Speed
100 7650 100 76500 0 373k 0 --:--:-- --:--:-- --:--:--
373k
{
"name": "hudi-sink",
"connector": {
"state": "RUNNING",
"worker_id": "172.17.0.3:8083"
},
"tasks": [
{
"id": 0,
"state": "FAILED",
"worker_id": "172.17.0.3:8083",
"trace": "org.apache.kafka.common.KafkaException: Failed to construct
kafka producer\n\tat
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:442)\n\tat
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:292)\n\tat
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:319)\n\tat
org.apache.hudi.connect.kafka.KafkaControlProducer.start(KafkaControlProducer.java:59)\n\tat
org.apache.hudi.connect.kafka.KafkaControlProducer.(KafkaControlProducer.java:50)\n\tat
org.apache.hudi.connect.kafka.KafkaConnectControlAgent.(KafkaConnectControlAgent.java:77)\n\tat
org.apache.hudi.connect.kafka.KafkaConnectControlAgent.createKafkaControlManager(KafkaConnectControlAgent.java:86)\n\tat
org.apache.hudi.connect.HoodieSinkTask.start(HoodieSinkTask.java:81)\n\tat
org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:312)\n\tat
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask
.java:186)\n\tat
org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:243)\n\tat
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
java.lang.Thread.run(Thread.java:748)\nCaused by:
org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls
given in bootstrap.servers\n\tat
org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:89)\n\tat
org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48)\n\tat
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:416)\n\t...
15 more\n"
},
{
"id": 1,
"state": "FAILED",
"worker_id": "172.17.0.3:8083",
"trace": "org.apache.kafka.common.KafkaException: Failed to construct
kafka producer\n\tat
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:442)\n\tat
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:292)\n\tat
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:319)\n\tat
org.apache.hudi.connect.kafka.KafkaControlProducer.start(KafkaControlProducer.java:59)\n\tat
org.apache.hudi.connect.kafka.KafkaControlProducer.(KafkaControlProducer.java:50)\n\tat
org.apache.hudi.connect.kafka.KafkaConnectControlAgent.(KafkaConnectControlAgent.java:77)\n\tat
org.apache.hudi.connect.kafka.KafkaConnectControlAgent.createKafkaControlManager(KafkaConnectControlAgent.java:86)\n\tat
org.apache.hudi.connect.HoodieSinkTask.start(HoodieSinkTask.java:81)\n\tat
org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:312)\n\tat
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask
.java:186)\n\tat
org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:243)\n\tat
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
java.lang.Thread.run(Thread.java:748)\nCaused by:
org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls
given in bootstrap.servers\n\tat
org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:89)\n\tat
org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48)\n\tat
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:416)\n\t...
15 more\n"
},
{
"id": 2,
[GitHub] [hudi] guyuqi commented on pull request #4617: HUDI-1657: build failed on AArch64, Fedora 33
guyuqi commented on pull request #4617: URL: https://github.com/apache/hudi/pull/4617#issuecomment-1026493615 > @guyuqi : can you respond to @yihua 's clarification above. Sorry for the late reply. I’m on Chinese New Year vacation and limited to access the PC. I’ll update the PR at the end of this week. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] guyuqi commented on pull request #4617: HUDI-1657: build failed on AArch64, Fedora 33
guyuqi commented on pull request #4617:
URL: https://github.com/apache/hudi/pull/4617#issuecomment-1018298182
> LGTM. @guyuqi Could you run the [Quick Start
Guide](https://github.com/apache/hudi/tree/master/hudi-kafka-connect) for Kafka
Connect Sink for Hudi to make sure the Sink functionality is not affected?
Thanks for your comments.
From [Quick Start
Guide](https://github.com/apache/hudi/tree/master/hudi-kafka-connect),
**Enviroments:**
```
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk/
export CONFLUENT_DIR=/home/builder/confluent-7.0.1
export PATH=${CONFLUENT_DIR}/bin:${PATH}
export KAFKA_HOME=/home/builder/kafka_2.12-3.0.0
export HUDI_DIR=/home/builder/hudi
Linux fdr33-test-vm 5.11.0-43-generic #47~20.04.2-Ubuntu SMP Mon Dec 13
11:10:13 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
```
**1. Successfully Create the Hudi Topic for the Sink and insert data into
the topic:**
```
[builder@fdr33-test-vm demo]$ bash setupKafka.sh -n 3
Argument num-kafka-records is 3
Delete Kafka topic hudi-test-topic ...
Create Kafka topic hudi-test-topic ...
Created topic hudi-test-topic.
{"id":1}{"subject":"hudi-test-topic","version":1,"id":1,"schema":"{\"type\":\"record\",\"name\":\"stock_ticks\",\"fields\":[{\"name\":\"volume\",\"type\":\"long\"},{\"name\":\"ts\",\"type\":\"string\"},{\"name\":\"symbol\",\"type\":\"string\"},{\"name\":\"year\",\"type\":\"int\"},{\"name\":\"month\",\"type\":\"string\"},{\"name\":\"high\",\"type\":\"double\"},{\"name\":\"low\",\"type\":\"double\"},{\"name\":\"key\",\"type\":\"string\"},{\"name\":\"date\",\"type\":\"string\"},{\"name\":\"close\",\"type\":\"double\"},{\"name\":\"open\",\"type\":\"double\"},{\"name\":\"day\",\"type\":\"string\"}]}"}Fri
Jan 21 08:25:52 UTC 2022
Start batch 1 ...
Fri Jan 21 08:25:53 UTC 2022
Record key until 3
publish to Kafka ...
```
```
[builder@fdr33-test-vm demo]$ bash setupKafka.sh -n 3 -b 3
Argument num-kafka-records is 3
Argument num-batch is 3
Delete Kafka topic hudi-test-topic ...
Create Kafka topic hudi-test-topic ...
Created topic hudi-test-topic.
{"id":1}{"subject":"hudi-test-topic","version":1,"id":1,"schema":"{\"type\":\"record\",\"name\":\"stock_ticks\",\"fields\":[{\"name\":\"volume\",\"type\":\"long\"},{\"name\":\"ts\",\"type\":\"string\"},{\"name\":\"symbol\",\"type\":\"string\"},{\"name\":\"year\",\"type\":\"int\"},{\"name\":\"month\",\"type\":\"string\"},{\"name\":\"high\",\"type\":\"double\"},{\"name\":\"low\",\"type\":\"double\"},{\"name\":\"key\",\"type\":\"string\"},{\"name\":\"date\",\"type\":\"string\"},{\"name\":\"close\",\"type\":\"double\"},{\"name\":\"open\",\"type\":\"double\"},{\"name\":\"day\",\"type\":\"string\"}]}"}Fri
Jan 21 08:27:04 UTC 2022
Start batch 1 ...
Fri Jan 21 08:27:04 UTC 2022
Record key until 3
publish to Kafka ...
Fri Jan 21 08:27:24 UTC 2022
Start batch 2 ...
Fri Jan 21 08:27:25 UTC 2022
Record key until 6
publish to Kafka ...
Fri Jan 21 08:27:45 UTC 2022
Start batch 3 ...
Fri Jan 21 08:27:45 UTC 2022
Record key until 9
publish to Kafka ...
```
**2. Run the Sink connector worker**
```
[builder@fdr33-test-vm kafka_2.12-3.0.0]$ ./bin/connect-distributed.sh
$HUDI_DIR/hudi-kafka-connect/demo/connect-distributed.properties
[2022-01-21 08:30:48,524] INFO WorkerInfo values:
jvm.args = -Xms256M, -Xmx2G, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20,
-XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent,
-XX:MaxInlineLevel=15, -Djava.awt.headless=true,
-Dcom.sun.management.jmxremote,
-Dcom.sun.management.jmxremote.authenticate=false,
-Dcom.sun.management.jmxremote.ssl=false,
-Dkafka.logs.dir=/home/builder/kafka_2.12-3.0.0/bin/../logs,
-Dlog4j.configuration=file:./bin/../config/connect-log4j.properties
jvm.spec = Red Hat, Inc., OpenJDK 64-Bit Server VM, 1.8.0_312,
25.312-b07
jvm.classpath =
/home/builder/kafka_2.12-3.0.0/bin/../libs/activation-1.1.1.jar:/home/builder/kafka_2.12-3.0.0/bin/../libs/aopalliance-repackaged-2.6.1.jar:/home/builder/kafka_2.12-3.0.0/bin/../libs/argparse4j-0.7.0.jar:/home/builder/kafka_2.12-3.0.0/bin/../libs/audience-annotations-0.5.0.jar:/home/builder/kafka_2.12-3.0.0/bin/../libs/commons-cli-1.4.jar:/home/builder/kafka_2.12-3.0.0/bin/../libs/commons-lang3-3.8.1.jar:/home/builder/kafka_2.12-3.0.0/bin/../libs/connect-api-3.0.0.jar:/home/builder/kafka_2.12-3.0.0/bin/../libs/connect-basic-auth-extension-3.0.0.jar:/home/builder/kafka_2.12-3.0.0/bin/../libs/connect-file-3.0.0.jar:/home/builder/kafka_2.12-3.0.0/bin/../libs/connect-json-3.0.0.jar:/home/builder/kafka_2.12-3.0.0/bin/../libs/connect-mirror-3.0.0.jar:/home/builder/kafka_2.12-3.0.0/bin/../libs/connect-mirror-client-3.0.0.jar:/home/builder/kafka_2.12-3.0.0/bin/../libs/connect-runtime-3.0.0.jar:/home/builder/kafka_2.12-3.0.0/bin/../libs/connect-transforms-3.0.0.jar:
/home/bui
[GitHub] [hudi] guyuqi commented on pull request #4617: HUDI-1657: build failed on AArch64, Fedora 33
guyuqi commented on pull request #4617: URL: https://github.com/apache/hudi/pull/4617#issuecomment-1014243001 Successfully build Hudi on Arm64 Fedora33/Ubuntu20: ``` INFO] Dependency-reduced POM written at: /home/builder/hudi/packaging/hudi-kafka-connect-bundle/target/dependency-reduced-pom.xml [INFO] [INFO] Reactor Summary for Hudi 0.11.0-SNAPSHOT: [INFO] [INFO] Hudi ... SUCCESS [ 3.737 s] [INFO] hudi-common SUCCESS [ 28.122 s] [INFO] hudi-aws ... SUCCESS [ 3.771 s] [INFO] hudi-timeline-service .. SUCCESS [ 3.532 s] [INFO] hudi-client SUCCESS [ 0.186 s] [INFO] hudi-client-common . SUCCESS [ 18.025 s] [INFO] hudi-hadoop-mr . SUCCESS [ 6.800 s] [INFO] hudi-spark-client .. SUCCESS [ 41.667 s] [INFO] hudi-sync-common ... SUCCESS [ 2.059 s] [INFO] hudi-hive-sync . SUCCESS [ 8.411 s] [INFO] hudi-spark-datasource .. SUCCESS [ 0.161 s] [INFO] hudi-spark-common_2.11 . SUCCESS [ 43.698 s] [INFO] hudi-spark2_2.11 ... SUCCESS [ 25.266 s] [INFO] hudi-spark2-common . SUCCESS [ 0.220 s] [INFO] hudi-spark_2.11 SUCCESS [01:04 min] [INFO] hudi-utilities_2.11 SUCCESS [ 13.561 s] [INFO] hudi-utilities-bundle_2.11 . SUCCESS [ 24.301 s] [INFO] hudi-cli ... SUCCESS [ 26.444 s] [INFO] hudi-java-client ... SUCCESS [ 5.383 s] [INFO] hudi-flink-client .. SUCCESS [ 13.235 s] [INFO] hudi-spark3-common . SUCCESS [ 15.734 s] [INFO] hudi-spark3_2.12 ... SUCCESS [ 11.753 s] [INFO] hudi-spark3.1.x_2.12 ... SUCCESS [ 7.994 s] [INFO] hudi-dla-sync .. SUCCESS [ 3.061 s] [INFO] hudi-sync .. SUCCESS [ 0.142 s] [INFO] hudi-hadoop-mr-bundle .. SUCCESS [ 10.599 s] [INFO] hudi-hive-sync-bundle .. SUCCESS [ 3.295 s] [INFO] hudi-spark-bundle_2.11 . SUCCESS [ 20.337 s] [INFO] hudi-presto-bundle . SUCCESS [ 12.757 s] [INFO] hudi-timeline-server-bundle SUCCESS [ 7.888 s] [INFO] hudi-trino-bundle .. SUCCESS [ 8.714 s] [INFO] hudi-hadoop-docker . SUCCESS [ 3.496 s] [INFO] hudi-hadoop-base-docker SUCCESS [ 2.160 s] [INFO] hudi-hadoop-base-java11-docker . SUCCESS [ 2.255 s] [INFO] hudi-hadoop-namenode-docker SUCCESS [ 2.243 s] [INFO] hudi-hadoop-datanode-docker SUCCESS [ 2.236 s] [INFO] hudi-hadoop-history-docker . SUCCESS [ 2.381 s] [INFO] hudi-hadoop-hive-docker SUCCESS [ 2.848 s] [INFO] hudi-hadoop-sparkbase-docker ... SUCCESS [ 2.314 s] [INFO] hudi-hadoop-sparkmaster-docker . SUCCESS [ 2.235 s] [INFO] hudi-hadoop-sparkworker-docker . SUCCESS [ 2.236 s] [INFO] hudi-hadoop-sparkadhoc-docker .. SUCCESS [ 2.234 s] [INFO] hudi-hadoop-presto-docker .. SUCCESS [ 2.369 s] [INFO] hudi-hadoop-trinobase-docker ... SUCCESS [ 2.505 s] [INFO] hudi-hadoop-trinocoordinator-docker SUCCESS [ 2.232 s] [INFO] hudi-hadoop-trinoworker-docker . SUCCESS [ 2.228 s] [INFO] hudi-integ-test SUCCESS [ 24.196 s] [INFO] hudi-integ-test-bundle . SUCCESS [01:02 min] [INFO] hudi-examples .. SUCCESS [ 13.839 s] [INFO] hudi-flink_2.11 SUCCESS [ 8.843 s] [INFO] hudi-kafka-connect . SUCCESS [ 6.963 s] [INFO] hudi-flink-bundle_2.11 . SUCCESS [ 26.518 s] [INFO] hudi-kafka-connect-bundle .. SUCCESS [ 30.123 s] [INFO] [INF
