Just merge the dependent third-party packages into the same directory, and each service directory is still separate. Just modify the directory specified in the startup script for dependencies
Jay Chung <zhongjiajie...@gmail.com> 于2023年9月11日周一 09:49写道： > Not yet, but we should not merge them into one single directory, cause > we take time to separate our services into multiple directories for > easy access and use > > On Sun, Sep 10, 2023 at 12:15 PM 顾泽民 <guzemin2...@gmail.com> wrote: > > > > I have an idea, can we merge libs from a single directory into one > > directory? > > > > Jay Chung <zhongjiajie...@gmail.com> 于2023年9月7日周四 17:14写道： > > > > > Hi guys, > > > > > > !!! We need some hero for the tarball size of dolphinscheduler. !!! > > > ASF infra team disagrees with the Jira ticket for bumping our SVN > > > limit[1], we need to find some way to make our binary tarball smaller > > > than 950MB. and the current size of our tarball is almost 1.2GB which > > > means we should reduce at least 278MB for the tarball. > > > > > > We have a DSIP to separate and release plugins in the future[2], but > > > it is still WIP, maybe we have to find another way to reduce the size > > > of binary tarball > > > > > > Please let me know if you have any good ideas about our tarball size, > > > thanks a lot > > > > > > [1]: https://issues.apache.org/jira/browse/INFRA-24944 > > > [2]: https://github.com/apache/dolphinscheduler/issues/1252 > > > > > > On Thu, Aug 31, 2023 at 9:58 AM Jay Chung <zhongjiajie...@gmail.com> > > > wrote: > > > > > > > > And if you find some bugfix still WIP and nearly to merge, please let > > > > me know in this thread, maybe we could add them to 3.2.0 too, cause > we > > > > block by SVN limit now > > > > > > > > Some of our community found out about two of them, and thank > Gallardot > > > > for bringing those up > > > > > > > > https://github.com/apache/dolphinscheduler/pull/14328 > > > > https://github.com/apache/dolphinscheduler/pull/14829 > > > > > > > > On Thu, Aug 31, 2023 at 9:54 AM Jay Chung <zhongjiajie...@gmail.com> > > > wrote: > > > > > > > > > > FYI > > > > > > > > > > I use `gh` cli to batch bump all issues and prs with milestone > `3.2.0` > > > > > but still open to `3.3.0`, maybe some of then should tag in 3.2.1 > > > > > (especially bug). so please take care of the bug with milestone > > > > > `3.3.0` and still open since Aug 31, 2023 > > > > > > > > > > On Thu, Aug 31, 2023 at 9:40 AM Jay Chung < > zhongjiajie...@gmail.com> > > > wrote: > > > > > > > > > > > > Hi all, > > > > > > > > > > > > Version 3.2.0 is nearly close and released during those days, > but we > > > > > > can not upload it to Apache SVN because too big for our tarball, > We > > > > > > created a Jira ticket to bump the svn limit for dolphinscheduler > and > > > > > > it is still in discussion. I will follow up when it has a further > > > > > > update > > > > > > > > > > > > On Thu, Jul 20, 2023 at 12:01 PM Jay Chung < > zhongjiajie...@gmail.com> > > > wrote: > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I had cherry-picked some recent commits to the branch > > > 3.2.0-prepare, > > > > > > > anyone who interested in checking 3.2.0 features could clone > the > > > > > > > latest branch to have a look > > > > > > > > > > > > > > On Tue, Jun 20, 2023 at 3:34 PM Jay Chung < > > > zhongjiajie...@gmail.com> wrote: > > > > > > > > > > > > > > > > Hi guys, > > > > > > > > > > > > > > > > I submit new PR[1] to remove test datasource in version > 3.2.0, > > > in [2] > > > > > > > > we introduce the test task and test datasource. It is a good > > > feature > > > > > > > > to allow user run in test environment. > > > > > > > > > > > > > > > > But I find out we have to create test datasource before > create > > > the > > > > > > > > prod one. I think it make user confuse especial some one who > do > > > not > > > > > > > > use test environment. So I think we should remove it in > version > > > 3.2.0. > > > > > > > > > > > > > > > > If you have different idea, please let me know in this mail > > > thread > > > > > > > > > > > > > > > > [1]: https://github.com/apache/dolphinscheduler/pull/14381 > > > > > > > > [2]: https://github.com/apache/dolphinscheduler/pull/11670 > > > > > > > > > > > > > > > > On Mon, Jun 19, 2023 at 2:59 PM Jay Chung < > > > zhongjiajie...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > Hi Community, > > > > > > > > > > > > > > > > > > I added a checklist for the version 3.2.0 release, > everyone who > > > > > > > > > interested in the incoming version can participate in our > > > test, the > > > > > > > > > detail checklist issue is in > > > > > > > > > https://github.com/apache/dolphinscheduler/issues/14372 > > > > > > > > > > > > > > > > > > On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang < > > > zhuang.ker...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > Thanks Jay. > > > > > > > > > > > > > > > > > > > > Looking forward to the 3.2.0 release. > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Best Wish > > > > > > > > > > — Kerwin (zhuangchong) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jay Chung <zhongjia...@apache.org> 于2023年6月3日周六 20:01写道： > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > I am here to propose Apache Dolphinscheduler version > 3.2.0 > > > plan to > > > > > > > > > > > release in July 2023 or August 2023. > > > > > > > > > > > > > > > > > > > > > > Actually, I already created a new branch named > > > `3.2.0-prepare` for > > > > > > > > > > > it[1], including all features merged to branch dev > after > > > we release > > > > > > > > > > > 3.1.0(date Sep 28, 2022), including many interesting > > > functional > > > > > > > > > > > additions and code base refactoring. > > > > > > > > > > > > > > > > > > > > > > Version 3.2.0 is WIP in testing and bug fixing now, > anyone > > > interested > > > > > > > > > > > to Dolphinscheduler or want to contribute to the latest > > > version is > > > > > > > > > > > welcome to test it. You can checkout the > 3.2.0-prepare[1] > > > branch and > > > > > > > > > > > start the standalone server, or build tarball for > cluster > > > mode test > > > > > > > > > > > based on your wish. In this version, we need to test > the > > > core function > > > > > > > > > > > of dolphinscheduler like CURD of project, workflow, > task > > > and execute > > > > > > > > > > > them. we allow need to test the resource center > carefully > > > because it > > > > > > > > > > > changes a lot in this version. > > > > > > > > > > > > > > > > > > > > > > You can report dolpinscheduler issue[2] if you find any > > > bugs. We have > > > > > > > > > > > a specific version for this incoming version name > > > `3.2.0-prepare` in > > > > > > > > > > > the bugs reported template. > > > > > > > > > > > > > > > > > > > > > > And feel free to submit PR to fix bugs, whatever is > > > reported by others or > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > Some of the functional features list for 3.2.0: > > > > > > > > > > > > > > > > > > > > > > * User friendly > > > > > > > > > > > * UI only: More hyperlinks jump between resource, > > > adjustable column > > > > > > > > > > > width > > > > > > > > > > > * Default tenant: Out of box run workflow by > default > > > tenant > > > > > > > > > > > * Run or rerun task in task instance > > > > > > > > > > > * Human readable json export result > > > > > > > > > > > * Cloud native > > > > > > > > > > > * Kubernetes Real-time log and custom label support > > > > > > > > > > > * Support KEDA autoscaler worker > > > > > > > > > > > * Support Terraform deploy to AWS > > > > > > > > > > > * Multi arch for zookeper and postgresql(all > services > > > are supported > > > > > > > > > > > now) > > > > > > > > > > > * Resource center > > > > > > > > > > > * Refactor resource center, we use provider API to > > > list and manage > > > > > > > > > > > resource now > > > > > > > > > > > * Allow resource re-upload now > > > > > > > > > > > * Datasource > > > > > > > > > > > * New datasource: > > > > > > > > > > > > > > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS > > > > > > > > > > > Athena > > > > > > > > > > > * Support access key for redshift datasource > > > > > > > > > > > * Tasks > > > > > > > > > > > * New tasks: azure datafactory、apache > > > > > > > > > > > linkis、Kubeflow、datasync、remote-shell、AWS Database > > > Migration、Java Task > > > > > > > > > > > * Improve for exists: Add presto for datax, > parameter > > > passing for > > > > > > > > > > > python and http, sql task multiple sql delimiter, > > > dependent depend > > > > > > > > > > > itself > > > > > > > > > > > * Introduce task cache for MLops or other long-run > > > tasks > > > > > > > > > > > * Parameter > > > > > > > > > > > * Add new rule of build-in function > > > > > > > > > > > * Introduce file type parameter > > > > > > > > > > > * Startup parameters have the highest priority > > > > > > > > > > > * Logs: Introduce remote logs mechanism and support > Google > > > Cloud > > > > > > > > > > > Storage、S3、OSS up to now > > > > > > > > > > > * API: > > > > > > > > > > > * Introduce some new restful API(experimental), > > > including > > > > > > > > > > > taskInstance、workflow state、workflowInstance、workflow > and > > > > > > > > > > > schedule、task relation > > > > > > > > > > > * Can get task instance id when triggering workflow > > > via API > > > > > > > > > > > * Registry: Introduce etcd, jdbc supported > > > > > > > > > > > > > > > > > > > > > > [1]: > > > https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare > > > > > > > > > > > [2]: > > > https://github.com/apache/dolphinscheduler/issues/new/choose > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Best Wish > > > > > > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Best Wish > > > > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Best Wish > > > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Best Wish > > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best Wish > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > -- > > > > > Best Wish > > > > > —Jay Chung > > > > > > > > > > > > > > > > -- > > > > Best Wish > > > > —Jay Chung > > > > > > > > > > > > -- > > > Best Wish > > > —Jay Chung > > > > > > > -- > Best Wish > —Jay Chung >
