Thanks eye, It is one solution to do that, but we should consider reducing the number of duplicate libs first instead of making them get together
On Mon, Sep 11, 2023 at 10:17 AM eye gu <guzemin2...@gmail.com> wrote: > > Just merge the dependent third-party packages into the same directory, and > each service directory is still separate. Just modify the directory > specified in the startup script for dependencies > > Jay Chung <zhongjiajie...@gmail.com> 于2023年9月11日周一 09:49写道： > > > Not yet, but we should not merge them into one single directory, cause > > we take time to separate our services into multiple directories for > > easy access and use > > > > On Sun, Sep 10, 2023 at 12:15 PM 顾泽民 <guzemin2...@gmail.com> wrote: > > > > > > I have an idea, can we merge libs from a single directory into one > > > directory? > > > > > > Jay Chung <zhongjiajie...@gmail.com> 于2023年9月7日周四 17:14写道： > > > > > > > Hi guys, > > > > > > > > !!! We need some hero for the tarball size of dolphinscheduler. !!! > > > > ASF infra team disagrees with the Jira ticket for bumping our SVN > > > > limit[1], we need to find some way to make our binary tarball smaller > > > > than 950MB. and the current size of our tarball is almost 1.2GB which > > > > means we should reduce at least 278MB for the tarball. > > > > > > > > We have a DSIP to separate and release plugins in the future[2], but > > > > it is still WIP, maybe we have to find another way to reduce the size > > > > of binary tarball > > > > > > > > Please let me know if you have any good ideas about our tarball size, > > > > thanks a lot > > > > > > > > [1]: https://issues.apache.org/jira/browse/INFRA-24944 > > > > [2]: https://github.com/apache/dolphinscheduler/issues/1252 > > > > > > > > On Thu, Aug 31, 2023 at 9:58 AM Jay Chung <zhongjiajie...@gmail.com> > > > > wrote: > > > > > > > > > > And if you find some bugfix still WIP and nearly to merge, please let > > > > > me know in this thread, maybe we could add them to 3.2.0 too, cause > > we > > > > > block by SVN limit now > > > > > > > > > > Some of our community found out about two of them, and thank > > Gallardot > > > > > for bringing those up > > > > > > > > > > https://github.com/apache/dolphinscheduler/pull/14328 > > > > > https://github.com/apache/dolphinscheduler/pull/14829 > > > > > > > > > > On Thu, Aug 31, 2023 at 9:54 AM Jay Chung <zhongjiajie...@gmail.com> > > > > wrote: > > > > > > > > > > > > FYI > > > > > > > > > > > > I use `gh` cli to batch bump all issues and prs with milestone > > `3.2.0` > > > > > > but still open to `3.3.0`, maybe some of then should tag in 3.2.1 > > > > > > (especially bug). so please take care of the bug with milestone > > > > > > `3.3.0` and still open since Aug 31, 2023 > > > > > > > > > > > > On Thu, Aug 31, 2023 at 9:40 AM Jay Chung < > > zhongjiajie...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > Version 3.2.0 is nearly close and released during those days, > > but we > > > > > > > can not upload it to Apache SVN because too big for our tarball, > > We > > > > > > > created a Jira ticket to bump the svn limit for dolphinscheduler > > and > > > > > > > it is still in discussion. I will follow up when it has a further > > > > > > > update > > > > > > > > > > > > > > On Thu, Jul 20, 2023 at 12:01 PM Jay Chung < > > zhongjiajie...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I had cherry-picked some recent commits to the branch > > > > 3.2.0-prepare, > > > > > > > > anyone who interested in checking 3.2.0 features could clone > > the > > > > > > > > latest branch to have a look > > > > > > > > > > > > > > > > On Tue, Jun 20, 2023 at 3:34 PM Jay Chung < > > > > zhongjiajie...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > Hi guys, > > > > > > > > > > > > > > > > > > I submit new PR[1] to remove test datasource in version > > 3.2.0, > > > > in [2] > > > > > > > > > we introduce the test task and test datasource. It is a good > > > > feature > > > > > > > > > to allow user run in test environment. > > > > > > > > > > > > > > > > > > But I find out we have to create test datasource before > > create > > > > the > > > > > > > > > prod one. I think it make user confuse especial some one who > > do > > > > not > > > > > > > > > use test environment. So I think we should remove it in > > version > > > > 3.2.0. > > > > > > > > > > > > > > > > > > If you have different idea, please let me know in this mail > > > > thread > > > > > > > > > > > > > > > > > > [1]: https://github.com/apache/dolphinscheduler/pull/14381 > > > > > > > > > [2]: https://github.com/apache/dolphinscheduler/pull/11670 > > > > > > > > > > > > > > > > > > On Mon, Jun 19, 2023 at 2:59 PM Jay Chung < > > > > zhongjiajie...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > Hi Community, > > > > > > > > > > > > > > > > > > > > I added a checklist for the version 3.2.0 release, > > everyone who > > > > > > > > > > interested in the incoming version can participate in our > > > > test, the > > > > > > > > > > detail checklist issue is in > > > > > > > > > > https://github.com/apache/dolphinscheduler/issues/14372 > > > > > > > > > > > > > > > > > > > > On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang < > > > > zhuang.ker...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > Thanks Jay. > > > > > > > > > > > > > > > > > > > > > > Looking forward to the 3.2.0 release. > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Best Wish > > > > > > > > > > > — Kerwin (zhuangchong) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jay Chung <zhongjia...@apache.org> 于2023年6月3日周六 20:01写道： > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > I am here to propose Apache Dolphinscheduler version > > 3.2.0 > > > > plan to > > > > > > > > > > > > release in July 2023 or August 2023. > > > > > > > > > > > > > > > > > > > > > > > > Actually, I already created a new branch named > > > > `3.2.0-prepare` for > > > > > > > > > > > > it[1], including all features merged to branch dev > > after > > > > we release > > > > > > > > > > > > 3.1.0(date Sep 28, 2022), including many interesting > > > > functional > > > > > > > > > > > > additions and code base refactoring. > > > > > > > > > > > > > > > > > > > > > > > > Version 3.2.0 is WIP in testing and bug fixing now, > > anyone > > > > interested > > > > > > > > > > > > to Dolphinscheduler or want to contribute to the latest > > > > version is > > > > > > > > > > > > welcome to test it. You can checkout the > > 3.2.0-prepare[1] > > > > branch and > > > > > > > > > > > > start the standalone server, or build tarball for > > cluster > > > > mode test > > > > > > > > > > > > based on your wish. In this version, we need to test > > the > > > > core function > > > > > > > > > > > > of dolphinscheduler like CURD of project, workflow, > > task > > > > and execute > > > > > > > > > > > > them. we allow need to test the resource center > > carefully > > > > because it > > > > > > > > > > > > changes a lot in this version. > > > > > > > > > > > > > > > > > > > > > > > > You can report dolpinscheduler issue[2] if you find any > > > > bugs. We have > > > > > > > > > > > > a specific version for this incoming version name > > > > `3.2.0-prepare` in > > > > > > > > > > > > the bugs reported template. > > > > > > > > > > > > > > > > > > > > > > > > And feel free to submit PR to fix bugs, whatever is > > > > reported by others or > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > > Some of the functional features list for 3.2.0: > > > > > > > > > > > > > > > > > > > > > > > > * User friendly > > > > > > > > > > > > * UI only: More hyperlinks jump between resource, > > > > adjustable column > > > > > > > > > > > > width > > > > > > > > > > > > * Default tenant: Out of box run workflow by > > default > > > > tenant > > > > > > > > > > > > * Run or rerun task in task instance > > > > > > > > > > > > * Human readable json export result > > > > > > > > > > > > * Cloud native > > > > > > > > > > > > * Kubernetes Real-time log and custom label support > > > > > > > > > > > > * Support KEDA autoscaler worker > > > > > > > > > > > > * Support Terraform deploy to AWS > > > > > > > > > > > > * Multi arch for zookeper and postgresql(all > > services > > > > are supported > > > > > > > > > > > > now) > > > > > > > > > > > > * Resource center > > > > > > > > > > > > * Refactor resource center, we use provider API to > > > > list and manage > > > > > > > > > > > > resource now > > > > > > > > > > > > * Allow resource re-upload now > > > > > > > > > > > > * Datasource > > > > > > > > > > > > * New datasource: > > > > > > > > > > > > > > > > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS > > > > > > > > > > > > Athena > > > > > > > > > > > > * Support access key for redshift datasource > > > > > > > > > > > > * Tasks > > > > > > > > > > > > * New tasks: azure datafactory、apache > > > > > > > > > > > > linkis、Kubeflow、datasync、remote-shell、AWS Database > > > > Migration、Java Task > > > > > > > > > > > > * Improve for exists: Add presto for datax, > > parameter > > > > passing for > > > > > > > > > > > > python and http, sql task multiple sql delimiter, > > > > dependent depend > > > > > > > > > > > > itself > > > > > > > > > > > > * Introduce task cache for MLops or other long-run > > > > tasks > > > > > > > > > > > > * Parameter > > > > > > > > > > > > * Add new rule of build-in function > > > > > > > > > > > > * Introduce file type parameter > > > > > > > > > > > > * Startup parameters have the highest priority > > > > > > > > > > > > * Logs: Introduce remote logs mechanism and support > > Google > > > > Cloud > > > > > > > > > > > > Storage、S3、OSS up to now > > > > > > > > > > > > * API: > > > > > > > > > > > > * Introduce some new restful API(experimental), > > > > including > > > > > > > > > > > > taskInstance、workflow state、workflowInstance、workflow > > and > > > > > > > > > > > > schedule、task relation > > > > > > > > > > > > * Can get task instance id when triggering workflow > > > > via API > > > > > > > > > > > > * Registry: Introduce etcd, jdbc supported > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > > https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare > > > > > > > > > > > > [2]: > > > > https://github.com/apache/dolphinscheduler/issues/new/choose > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Best Wish > > > > > > > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Best Wish > > > > > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Best Wish > > > > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Best Wish > > > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Best Wish > > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best Wish > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > -- > > > > > Best Wish > > > > > —Jay Chung > > > > > > > > > > > > > > > > -- > > > > Best Wish > > > > —Jay Chung > > > > > > > > > > > > -- > > Best Wish > > —Jay Chung > > -- Best Wish —Jay Chung
