homesickjava commented on issue #1576: URL: https://github.com/apache/incubator-paimon/issues/1576#issuecomment-1653478928
I mean, when I use paimon-flink-action.jar to run a paimon job in flink, I can use parameters to turning performance, for example : ./flink run -Dscan.mode=latest -Dstreaming-read-mode=file -Dexecution.checkpointing.interval=1s -Dclassloader.check-leaked-classloader=false paimon-flink-action-0.5-SNAPSHOT.jar mysql-sync-table --warehouse hdfs://pcosmo-gyos03:8020/user/root/paimon --database paimon_test --table paimon_sync_wangke5 --computed-columns 'current_login_time=avg(current_login_time)' --mysql-conf hostname=10.206.32.3 --mysql-conf port=9006 --mysql-conf username=root --mysql-conf password=123456 --mysql-conf database-name=paimontest --mysql-conf table-name='wangke' --catalog-conf metastore=hive --catalog-conf uri=thrift://10.206.32.3:9083 --table-conf write-manifest-cache=1024mb --table-conf manifest.full-compaction-threshold-size=256mb **--table-conf scan.manifest.parallelism=6** --table-conf scan.split-enumerator.batch-size=6 --table-conf bucket=6 --table-conf changelog-producer=input --table-conf sink.parallelism=6 --table-conf sort-spill-threshold=1024mb **--table-conf execution.checkpoint ing.timeout=60min** --table-**conf execution.checkpointing.interval=5min** **--table-conf full-compaction.delta-commits=3** **--table-conf changelog-producer=full-compaction** --table-conf scan.mode=compacted-full --table-conf num-sorted-run.stop-trigger=2147483647 --table-conf sort-spill-threshold=10 **--table-conf write-buffer-size=4096mb --table-conf taskmanager.memory.managed.size=1024m --table-conf write-buffer-spillable=true** --table-conf file.format=avro --table-conf metadata.stats-mode=none I can use these BOND FONT parameters to turning performance, meanwhile, if I use flink-connector-pulsar.jar to sync data from pulsar to paimon, I can use following steps: 1. create a table with connector of 'pulsar', name tableA 2. pulsar cdc will auto sync data from pulsar to flink 3. create a paimon catalog 4. within this paimon catalog, create a table name tableB 5. set checkpoint.interval = 1s 6. insert into tableB select * from tableA with doing this, how can i turning performance like I use paimon-flink-action.jar above? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
