Hi community,
At present, the command executed by the task in the python Api is to pass
the specific content through the parameter. For example, shell task:
Shell(name="task_parent", command="echo hello pydolphinscheduler").
File is a good choice when the command executed by the task is very long.
Teacher Zhongjiajie guided me, we called it New mechanism file plugins to
Python API. Resource plugins can be used in workflows and tasks. When both
have resource plugins, the resource plugin in the task is used first.
Plugins I will implement include but are not limited to local file system,
GitHub, GitLab, Amazon S3, Alibaba Cloud OSS.
This is a tutorial for the local resource plugin, you can find it in PR[1],
it is the local resource plugin.
with ProcessDefinition(
name="tutorial_resource_plugin",
schedule="0 0 0 * * ? *",
start_time="2022-01-01",
tenant="tenant_exists",
resource_plugin=Local("/tmp"),
) as process_definition:
file = "resource.sh"
path = Path("/tmp").joinpath(file)
with open(str(path), "w") as f:
f.write("echo tutorial resource plugin")
task_parent = Shell(
name="local-resource-example",
command=file,
)
print(task_parent.task_params)
os.remove(path)
process_definition.run()
Below is my draft design for the shell task.
You can see the GitLab resource plugin in PR[2] to understand my design.
I abstracted a Resource class and created an abstract function read_file
for it. All resource plugins need to inherit from it and implement the
abstract function read_file.
Add property in shell
ext: set = {".sh", ".zsh"}
ext is used to detect that shell tasks can only accept files ending in .sh
and .zsh
ext_attr: str = "_raw_script"
At the same time, it is added in the init function
self._raw_script = command
ext_attr is used to save the file path,
Add a new parameter resource_plugin to the init function of the task to
accept resource plug-ins, add a new method to obtain specific resource
plug-ins, and also need to add attributes ext and ext_attr.
Additional details can be found in pr[1] and pr[2].
I'm very sorry, it's my first time to contribute and I didn't follow the
DSIP process, sorry.
I already add a GitHub Issue for my proposal, which you could see in
https://github.com/apache/dolphinscheduler/issues/10911
<https://github.com/apache/dolphinscheduler/issues/10911>.
Looking forward any feedback for this thread.
[1]: https://github.com/apache/dolphinscheduler/pull/11360 <
https://github.com/apache/dolphinscheduler/pull/11360>
[2]: https://github.com/apache/dolphinscheduler/pull/11831 <
https://github.com/apache/dolphinscheduler/pull/11831>
