Re: [PR] [Fix][Plugin]Add remote plugin discovery for flink yarn applicion deploy mode [seatunnel]

Tue, 17 Sep 2024 17:37:04 -0700 


hanhanzhang commented on PR #7649:
URL: https://github.com/apache/seatunnel/pull/7649#issuecomment-2357259168

   > `${FLINK_HOME}/bin/flink run-application 
-Dyarn.ship-files='./config/v2.streaming.conf.template' 
-Dyarn.classpath.include-user-jar=DISABLED 
-Dyarn.provided.usrlib.dir='hdfs://HDFS4001185/user/flink/usrlib/' --target 
yarn-application -c org.apache.seatunnel.core.starter.flink.SeaTunnelFlink 
/usr/local/service/seatunnel/starter/seatunnel-flink-15-starter.jar --config 
v2.streaming.conf.template --name SeaTunnel`
   > 
   > Using the combination of yarn.provided.usrlib.dir along with 
yarn.ship-files seems to be a good approach. 
![image](https://private-user-images.githubusercontent.com/67902676/367509950-a9e277dc-2fa3-4084-b60c-ad4e980be276.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjY2MTcyMDQsIm5iZiI6MTcyNjYxNjkwNCwicGF0aCI6Ii82NzkwMjY3Ni8zNjc1MDk5NTAtYTllMjc3ZGMtMmZhMy00MDg0LWI2MGMtYWQ0ZTk4MGJlMjc2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTE3VDIzNDgyNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTZiZDBhMGI3N2M2NGExMWE5YmE3ZmYyNDk5MWMyYzQ2M2UwZmExNjk5ZWVjYjU5MzEzN2FlMTIyMDViN2QzYWMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.3VngA-VuB2ieA8xzHz2iMXir0HQ6-4Ys32dsJqTCe08)
 ![image](https://private-user-images.githubusercontent.com
 
/67902676/367510043-53742690-9b3f-4962-97d5-7783a33d2493.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjY2MTcyMDQsIm5iZiI6MTcyNjYxNjkwNCwicGF0aCI6Ii82NzkwMjY3Ni8zNjc1MTAwNDMtNTM3NDI2OTAtOWIzZi00OTYyLTk3ZDUtNzc4M2EzM2QyNDkzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTE3VDIzNDgyNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQ2NGU1MmQ3ZjM2ZmRjMzg4MjNjMTIxMTJmMDIwNzgwY2Q2NTFlNmYzZDk4MzkwMmQ4NWRjNjI5YWQwODVjMTgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.wc4E-X0j3mMYUZAG962vy3PU2pnDPDeCChLl5irbRr0)
 
![image](https://private-user-images.githubusercontent.com/67902676/367505857-a3ee682c-868d-4d80-be7d-e55e7a3fd90e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50Lm
 
NvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjY2MTcyMDQsIm5iZiI6MTcyNjYxNjkwNCwicGF0aCI6Ii82NzkwMjY3Ni8zNjc1MDU4NTctYTNlZTY4MmMtODY4ZC00ZDgwLWJlN2QtZTU1ZTdhM2ZkOTBlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTE3VDIzNDgyNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTVjYjU3ZDA5NjdiYmJkY2JiZjkxZTE4MmY1ZGExYzY3NDcxYWYxMWI0ZDBmYzg3MDBjOTJiNzg3NTkwZTA0MGEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.0JAdC8HjoJyt6BrppRDKwxvVCeM1wEo_mEOje43ob0Y)
   
   Yes, I've thought about it this way. I wouldn't recommend putting SeaTunnel 
lib in yarn.provided.lib.dirs. This would require SeaTunnel and Flink 
dependency packages to be placed together. I think it might be a good way to 
use SeaTunnel Connector as a UDF similar to SQL.
   
   In addition, the yarn.provided.usrlib.dirs means all the jars in the 
classpath, the initial plugin design should be to put the plugins needed for 
the task in the classpath. What do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to