Hi guys, !!! We need some hero for the tarball size of dolphinscheduler. !!! ASF infra team disagrees with the Jira ticket for bumping our SVN limit[1], we need to find some way to make our binary tarball smaller than 950MB. and the current size of our tarball is almost 1.2GB which means we should reduce at least 278MB for the tarball.
We have a DSIP to separate and release plugins in the future[2], but it is still WIP, maybe we have to find another way to reduce the size of binary tarball Please let me know if you have any good ideas about our tarball size, thanks a lot [1]: https://issues.apache.org/jira/browse/INFRA-24944 [2]: https://github.com/apache/dolphinscheduler/issues/1252 On Thu, Aug 31, 2023 at 9:58 AM Jay Chung <zhongjiajie...@gmail.com> wrote: > > And if you find some bugfix still WIP and nearly to merge, please let > me know in this thread, maybe we could add them to 3.2.0 too, cause we > block by SVN limit now > > Some of our community found out about two of them, and thank Gallardot > for bringing those up > > https://github.com/apache/dolphinscheduler/pull/14328 > https://github.com/apache/dolphinscheduler/pull/14829 > > On Thu, Aug 31, 2023 at 9:54 AM Jay Chung <zhongjiajie...@gmail.com> wrote: > > > > FYI > > > > I use `gh` cli to batch bump all issues and prs with milestone `3.2.0` > > but still open to `3.3.0`, maybe some of then should tag in 3.2.1 > > (especially bug). so please take care of the bug with milestone > > `3.3.0` and still open since Aug 31, 2023 > > > > On Thu, Aug 31, 2023 at 9:40 AM Jay Chung <zhongjiajie...@gmail.com> wrote: > > > > > > Hi all, > > > > > > Version 3.2.0 is nearly close and released during those days, but we > > > can not upload it to Apache SVN because too big for our tarball, We > > > created a Jira ticket to bump the svn limit for dolphinscheduler and > > > it is still in discussion. I will follow up when it has a further > > > update > > > > > > On Thu, Jul 20, 2023 at 12:01 PM Jay Chung <zhongjiajie...@gmail.com> > > > wrote: > > > > > > > > Hi all, > > > > > > > > I had cherry-picked some recent commits to the branch 3.2.0-prepare, > > > > anyone who interested in checking 3.2.0 features could clone the > > > > latest branch to have a look > > > > > > > > On Tue, Jun 20, 2023 at 3:34 PM Jay Chung <zhongjiajie...@gmail.com> > > > > wrote: > > > > > > > > > > Hi guys, > > > > > > > > > > I submit new PR[1] to remove test datasource in version 3.2.0, in [2] > > > > > we introduce the test task and test datasource. It is a good feature > > > > > to allow user run in test environment. > > > > > > > > > > But I find out we have to create test datasource before create the > > > > > prod one. I think it make user confuse especial some one who do not > > > > > use test environment. So I think we should remove it in version 3.2.0. > > > > > > > > > > If you have different idea, please let me know in this mail thread > > > > > > > > > > [1]: https://github.com/apache/dolphinscheduler/pull/14381 > > > > > [2]: https://github.com/apache/dolphinscheduler/pull/11670 > > > > > > > > > > On Mon, Jun 19, 2023 at 2:59 PM Jay Chung <zhongjiajie...@gmail.com> > > > > > wrote: > > > > > > > > > > > > Hi Community, > > > > > > > > > > > > I added a checklist for the version 3.2.0 release, everyone who > > > > > > interested in the incoming version can participate in our test, the > > > > > > detail checklist issue is in > > > > > > https://github.com/apache/dolphinscheduler/issues/14372 > > > > > > > > > > > > On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang > > > > > > <zhuang.ker...@gmail.com> wrote: > > > > > > > > > > > > > > Thanks Jay. > > > > > > > > > > > > > > Looking forward to the 3.2.0 release. > > > > > > > > > > > > > > -- > > > > > > > Best Wish > > > > > > > — Kerwin (zhuangchong) > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jay Chung <zhongjia...@apache.org> 于2023年6月3日周六 20:01写道： > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I am here to propose Apache Dolphinscheduler version 3.2.0 plan > > > > > > > > to > > > > > > > > release in July 2023 or August 2023. > > > > > > > > > > > > > > > > Actually, I already created a new branch named `3.2.0-prepare` > > > > > > > > for > > > > > > > > it[1], including all features merged to branch dev after we > > > > > > > > release > > > > > > > > 3.1.0(date Sep 28, 2022), including many interesting functional > > > > > > > > additions and code base refactoring. > > > > > > > > > > > > > > > > Version 3.2.0 is WIP in testing and bug fixing now, anyone > > > > > > > > interested > > > > > > > > to Dolphinscheduler or want to contribute to the latest version > > > > > > > > is > > > > > > > > welcome to test it. You can checkout the 3.2.0-prepare[1] > > > > > > > > branch and > > > > > > > > start the standalone server, or build tarball for cluster mode > > > > > > > > test > > > > > > > > based on your wish. In this version, we need to test the core > > > > > > > > function > > > > > > > > of dolphinscheduler like CURD of project, workflow, task and > > > > > > > > execute > > > > > > > > them. we allow need to test the resource center carefully > > > > > > > > because it > > > > > > > > changes a lot in this version. > > > > > > > > > > > > > > > > You can report dolpinscheduler issue[2] if you find any bugs. > > > > > > > > We have > > > > > > > > a specific version for this incoming version name > > > > > > > > `3.2.0-prepare` in > > > > > > > > the bugs reported template. > > > > > > > > > > > > > > > > And feel free to submit PR to fix bugs, whatever is reported by > > > > > > > > others or > > > > > > > > you > > > > > > > > > > > > > > > > Some of the functional features list for 3.2.0: > > > > > > > > > > > > > > > > * User friendly > > > > > > > > * UI only: More hyperlinks jump between resource, > > > > > > > > adjustable column > > > > > > > > width > > > > > > > > * Default tenant: Out of box run workflow by default tenant > > > > > > > > * Run or rerun task in task instance > > > > > > > > * Human readable json export result > > > > > > > > * Cloud native > > > > > > > > * Kubernetes Real-time log and custom label support > > > > > > > > * Support KEDA autoscaler worker > > > > > > > > * Support Terraform deploy to AWS > > > > > > > > * Multi arch for zookeper and postgresql(all services are > > > > > > > > supported > > > > > > > > now) > > > > > > > > * Resource center > > > > > > > > * Refactor resource center, we use provider API to list and > > > > > > > > manage > > > > > > > > resource now > > > > > > > > * Allow resource re-upload now > > > > > > > > * Datasource > > > > > > > > * New datasource: > > > > > > > > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS > > > > > > > > Athena > > > > > > > > * Support access key for redshift datasource > > > > > > > > * Tasks > > > > > > > > * New tasks: azure datafactory、apache > > > > > > > > linkis、Kubeflow、datasync、remote-shell、AWS Database > > > > > > > > Migration、Java Task > > > > > > > > * Improve for exists: Add presto for datax, parameter > > > > > > > > passing for > > > > > > > > python and http, sql task multiple sql delimiter, dependent > > > > > > > > depend > > > > > > > > itself > > > > > > > > * Introduce task cache for MLops or other long-run tasks > > > > > > > > * Parameter > > > > > > > > * Add new rule of build-in function > > > > > > > > * Introduce file type parameter > > > > > > > > * Startup parameters have the highest priority > > > > > > > > * Logs: Introduce remote logs mechanism and support Google Cloud > > > > > > > > Storage、S3、OSS up to now > > > > > > > > * API: > > > > > > > > * Introduce some new restful API(experimental), including > > > > > > > > taskInstance、workflow state、workflowInstance、workflow and > > > > > > > > schedule、task relation > > > > > > > > * Can get task instance id when triggering workflow via API > > > > > > > > * Registry: Introduce etcd, jdbc supported > > > > > > > > > > > > > > > > [1]: > > > > > > > > https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare > > > > > > > > [2]: > > > > > > > > https://github.com/apache/dolphinscheduler/issues/new/choose > > > > > > > > > > > > > > > > -- > > > > > > > > Best Wish > > > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best Wish > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > -- > > > > > Best Wish > > > > > —Jay Chung > > > > > > > > > > > > > > > > -- > > > > Best Wish > > > > —Jay Chung > > > > > > > > > > > > -- > > > Best Wish > > > —Jay Chung > > > > > > > > -- > > Best Wish > > —Jay Chung > > > > -- > Best Wish > —Jay Chung -- Best Wish —Jay Chung
