I have an idea, can we merge libs from a single directory into one directory?
Jay Chung <[email protected]> 于2023年9月7日周四 17:14写道： > Hi guys, > > !!! We need some hero for the tarball size of dolphinscheduler. !!! > ASF infra team disagrees with the Jira ticket for bumping our SVN > limit[1], we need to find some way to make our binary tarball smaller > than 950MB. and the current size of our tarball is almost 1.2GB which > means we should reduce at least 278MB for the tarball. > > We have a DSIP to separate and release plugins in the future[2], but > it is still WIP, maybe we have to find another way to reduce the size > of binary tarball > > Please let me know if you have any good ideas about our tarball size, > thanks a lot > > [1]: https://issues.apache.org/jira/browse/INFRA-24944 > [2]: https://github.com/apache/dolphinscheduler/issues/1252 > > On Thu, Aug 31, 2023 at 9:58 AM Jay Chung <[email protected]> > wrote: > > > > And if you find some bugfix still WIP and nearly to merge, please let > > me know in this thread, maybe we could add them to 3.2.0 too, cause we > > block by SVN limit now > > > > Some of our community found out about two of them, and thank Gallardot > > for bringing those up > > > > https://github.com/apache/dolphinscheduler/pull/14328 > > https://github.com/apache/dolphinscheduler/pull/14829 > > > > On Thu, Aug 31, 2023 at 9:54 AM Jay Chung <[email protected]> > wrote: > > > > > > FYI > > > > > > I use `gh` cli to batch bump all issues and prs with milestone `3.2.0` > > > but still open to `3.3.0`, maybe some of then should tag in 3.2.1 > > > (especially bug). so please take care of the bug with milestone > > > `3.3.0` and still open since Aug 31, 2023 > > > > > > On Thu, Aug 31, 2023 at 9:40 AM Jay Chung <[email protected]> > wrote: > > > > > > > > Hi all, > > > > > > > > Version 3.2.0 is nearly close and released during those days, but we > > > > can not upload it to Apache SVN because too big for our tarball, We > > > > created a Jira ticket to bump the svn limit for dolphinscheduler and > > > > it is still in discussion. I will follow up when it has a further > > > > update > > > > > > > > On Thu, Jul 20, 2023 at 12:01 PM Jay Chung <[email protected]> > wrote: > > > > > > > > > > Hi all, > > > > > > > > > > I had cherry-picked some recent commits to the branch > 3.2.0-prepare, > > > > > anyone who interested in checking 3.2.0 features could clone the > > > > > latest branch to have a look > > > > > > > > > > On Tue, Jun 20, 2023 at 3:34 PM Jay Chung < > [email protected]> wrote: > > > > > > > > > > > > Hi guys, > > > > > > > > > > > > I submit new PR[1] to remove test datasource in version 3.2.0, > in [2] > > > > > > we introduce the test task and test datasource. It is a good > feature > > > > > > to allow user run in test environment. > > > > > > > > > > > > But I find out we have to create test datasource before create > the > > > > > > prod one. I think it make user confuse especial some one who do > not > > > > > > use test environment. So I think we should remove it in version > 3.2.0. > > > > > > > > > > > > If you have different idea, please let me know in this mail > thread > > > > > > > > > > > > [1]: https://github.com/apache/dolphinscheduler/pull/14381 > > > > > > [2]: https://github.com/apache/dolphinscheduler/pull/11670 > > > > > > > > > > > > On Mon, Jun 19, 2023 at 2:59 PM Jay Chung < > [email protected]> wrote: > > > > > > > > > > > > > > Hi Community, > > > > > > > > > > > > > > I added a checklist for the version 3.2.0 release, everyone who > > > > > > > interested in the incoming version can participate in our > test, the > > > > > > > detail checklist issue is in > > > > > > > https://github.com/apache/dolphinscheduler/issues/14372 > > > > > > > > > > > > > > On Mon, Jun 5, 2023 at 8:59 AM Kerwin Zhuang < > [email protected]> wrote: > > > > > > > > > > > > > > > > Thanks Jay. > > > > > > > > > > > > > > > > Looking forward to the 3.2.0 release. > > > > > > > > > > > > > > > > -- > > > > > > > > Best Wish > > > > > > > > — Kerwin (zhuangchong) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jay Chung <[email protected]> 于2023年6月3日周六 20:01写道： > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > I am here to propose Apache Dolphinscheduler version 3.2.0 > plan to > > > > > > > > > release in July 2023 or August 2023. > > > > > > > > > > > > > > > > > > Actually, I already created a new branch named > `3.2.0-prepare` for > > > > > > > > > it[1], including all features merged to branch dev after > we release > > > > > > > > > 3.1.0(date Sep 28, 2022), including many interesting > functional > > > > > > > > > additions and code base refactoring. > > > > > > > > > > > > > > > > > > Version 3.2.0 is WIP in testing and bug fixing now, anyone > interested > > > > > > > > > to Dolphinscheduler or want to contribute to the latest > version is > > > > > > > > > welcome to test it. You can checkout the 3.2.0-prepare[1] > branch and > > > > > > > > > start the standalone server, or build tarball for cluster > mode test > > > > > > > > > based on your wish. In this version, we need to test the > core function > > > > > > > > > of dolphinscheduler like CURD of project, workflow, task > and execute > > > > > > > > > them. we allow need to test the resource center carefully > because it > > > > > > > > > changes a lot in this version. > > > > > > > > > > > > > > > > > > You can report dolpinscheduler issue[2] if you find any > bugs. We have > > > > > > > > > a specific version for this incoming version name > `3.2.0-prepare` in > > > > > > > > > the bugs reported template. > > > > > > > > > > > > > > > > > > And feel free to submit PR to fix bugs, whatever is > reported by others or > > > > > > > > > you > > > > > > > > > > > > > > > > > > Some of the functional features list for 3.2.0: > > > > > > > > > > > > > > > > > > * User friendly > > > > > > > > > * UI only: More hyperlinks jump between resource, > adjustable column > > > > > > > > > width > > > > > > > > > * Default tenant: Out of box run workflow by default > tenant > > > > > > > > > * Run or rerun task in task instance > > > > > > > > > * Human readable json export result > > > > > > > > > * Cloud native > > > > > > > > > * Kubernetes Real-time log and custom label support > > > > > > > > > * Support KEDA autoscaler worker > > > > > > > > > * Support Terraform deploy to AWS > > > > > > > > > * Multi arch for zookeper and postgresql(all services > are supported > > > > > > > > > now) > > > > > > > > > * Resource center > > > > > > > > > * Refactor resource center, we use provider API to > list and manage > > > > > > > > > resource now > > > > > > > > > * Allow resource re-upload now > > > > > > > > > * Datasource > > > > > > > > > * New datasource: > > > > > > > > > > snowflake、databend、kyuubi、oceanbase、dameng、azuresql、starrocks、AWS > > > > > > > > > Athena > > > > > > > > > * Support access key for redshift datasource > > > > > > > > > * Tasks > > > > > > > > > * New tasks: azure datafactory、apache > > > > > > > > > linkis、Kubeflow、datasync、remote-shell、AWS Database > Migration、Java Task > > > > > > > > > * Improve for exists: Add presto for datax, parameter > passing for > > > > > > > > > python and http, sql task multiple sql delimiter, > dependent depend > > > > > > > > > itself > > > > > > > > > * Introduce task cache for MLops or other long-run > tasks > > > > > > > > > * Parameter > > > > > > > > > * Add new rule of build-in function > > > > > > > > > * Introduce file type parameter > > > > > > > > > * Startup parameters have the highest priority > > > > > > > > > * Logs: Introduce remote logs mechanism and support Google > Cloud > > > > > > > > > Storage、S3、OSS up to now > > > > > > > > > * API: > > > > > > > > > * Introduce some new restful API(experimental), > including > > > > > > > > > taskInstance、workflow state、workflowInstance、workflow and > > > > > > > > > schedule、task relation > > > > > > > > > * Can get task instance id when triggering workflow > via API > > > > > > > > > * Registry: Introduce etcd, jdbc supported > > > > > > > > > > > > > > > > > > [1]: > https://github.com/apache/dolphinscheduler/tree/3.2.0-prepare > > > > > > > > > [2]: > https://github.com/apache/dolphinscheduler/issues/new/choose > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Best Wish > > > > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Best Wish > > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best Wish > > > > > > —Jay Chung > > > > > > > > > > > > > > > > > > > > -- > > > > > Best Wish > > > > > —Jay Chung > > > > > > > > > > > > > > > > -- > > > > Best Wish > > > > —Jay Chung > > > > > > > > > > > > -- > > > Best Wish > > > —Jay Chung > > > > > > > > -- > > Best Wish > > —Jay Chung > > > > -- > Best Wish > —Jay Chung >
