I have no permissions so I can't do it but I'm happy to help (although I am more familiar with Gitlab CICD than Github Actions). Is there some point of contact that can provide me needed context and permissions? I'd also love to see why the costs are high and see how we can reduce them...
Thanks, Nimrod On Wed, May 8, 2024 at 8:26 AM Holden Karau <holden.ka...@gmail.com> wrote: > I think signing the artifacts produced from a secure CI sounds like a good > idea. I know we’ve been asked to reduce our GitHub action usage but perhaps > someone interested could volunteer to set that up. > > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > > > On Tue, May 7, 2024 at 9:43 PM Nimrod Ofek <ofek.nim...@gmail.com> wrote: > >> Hi, >> Thanks for the reply. >> >> From my experience, a build on a build server would be much more >> predictable and less error prone than building on some laptop- and of >> course much faster to have builds, snapshots, release candidates, early >> previews releases, release candidates or final releases. >> It will enable us to have a preview version with current changes- >> snapshot version, either automatically every day or if we need to save >> costs (although build is really not expensive) - with a click of a button. >> >> Regarding keys for signing. - that's what vaults are for, all across the >> industry we are using vaults (such as hashicorp vault)- but if the build >> will be automated and the only thing which will be manual is to sign the >> release for security reasons that would be reasonable. >> >> Thanks, >> Nimrod >> >> >> בתאריך יום ד׳, 8 במאי 2024, 00:54, מאת Holden Karau < >> holden.ka...@gmail.com>: >> >>> Indeed. We could conceivably build the release in CI/CD but the final >>> verification / signing should be done locally to keep the keys safe (there >>> was some concern from earlier release processes). >>> >>> Twitter: https://twitter.com/holdenkarau >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> >>> >>> On Tue, May 7, 2024 at 10:55 AM Nimrod Ofek <ofek.nim...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> Sorry for the novice question, Wenchen - the release is done manually >>>> from a laptop? Not using a CI CD process on a build server? >>>> >>>> Thanks, >>>> Nimrod >>>> >>>> On Tue, May 7, 2024 at 8:50 PM Wenchen Fan <cloud0...@gmail.com> wrote: >>>> >>>>> UPDATE: >>>>> >>>>> Unfortunately, it took me quite some time to set up my laptop and get >>>>> it ready for the release process (docker desktop doesn't work anymore, my >>>>> pgp key is lost, etc.). I'll start the RC process at my tomorrow. Thanks >>>>> for your patience! >>>>> >>>>> Wenchen >>>>> >>>>> On Fri, May 3, 2024 at 7:47 AM yangjie01 <yangji...@baidu.com> wrote: >>>>> >>>>>> +1 >>>>>> >>>>>> >>>>>> >>>>>> *发件人**: *Jungtaek Lim <kabhwan.opensou...@gmail.com> >>>>>> *日期**: *2024年5月2日 星期四 10:21 >>>>>> *收件人**: *Holden Karau <holden.ka...@gmail.com> >>>>>> *抄送**: *Chao Sun <sunc...@apache.org>, Xiao Li <gatorsm...@gmail.com>, >>>>>> Tathagata Das <tathagata.das1...@gmail.com>, Wenchen Fan < >>>>>> cloud0...@gmail.com>, Cheng Pan <pan3...@gmail.com>, Nicholas >>>>>> Chammas <nicholas.cham...@gmail.com>, Dongjoon Hyun < >>>>>> dongjoon.h...@gmail.com>, Cheng Pan <cheng...@apache.org>, Spark dev >>>>>> list <dev@spark.apache.org>, Anish Shrigondekar < >>>>>> anish.shrigonde...@databricks.com> >>>>>> *主题**: *Re: [DISCUSS] Spark 4.0.0 release >>>>>> >>>>>> >>>>>> >>>>>> +1 love to see it! >>>>>> >>>>>> >>>>>> >>>>>> On Thu, May 2, 2024 at 10:08 AM Holden Karau <holden.ka...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> +1 :) yay previews >>>>>> >>>>>> >>>>>> >>>>>> On Wed, May 1, 2024 at 5:36 PM Chao Sun <sunc...@apache.org> wrote: >>>>>> >>>>>> +1 >>>>>> >>>>>> >>>>>> >>>>>> On Wed, May 1, 2024 at 5:23 PM Xiao Li <gatorsm...@gmail.com> wrote: >>>>>> >>>>>> +1 for next Monday. >>>>>> >>>>>> >>>>>> >>>>>> We can do more previews when the other features are ready for preview. >>>>>> >>>>>> >>>>>> >>>>>> Tathagata Das <tathagata.das1...@gmail.com> 于2024年5月1日周三 08:46写道： >>>>>> >>>>>> Next week sounds great! Thank you Wenchen! >>>>>> >>>>>> >>>>>> >>>>>> On Wed, May 1, 2024 at 11:16 AM Wenchen Fan <cloud0...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> Yea I think a preview release won't hurt (without a branch cut). We >>>>>> don't need to wait for all the ongoing projects to be ready. How about we >>>>>> do a 4.0 preview release based on the current master branch next Monday? >>>>>> >>>>>> >>>>>> >>>>>> On Wed, May 1, 2024 at 11:06 PM Tathagata Das < >>>>>> tathagata.das1...@gmail.com> wrote: >>>>>> >>>>>> Hey all, >>>>>> >>>>>> >>>>>> >>>>>> Reviving this thread, but Spark master has already accumulated a huge >>>>>> amount of changes. As a downstream project maintainer, I want to really >>>>>> start testing the new features and other breaking changes, and it's hard >>>>>> to >>>>>> do that without a Preview release. So the sooner we make a Preview >>>>>> release, >>>>>> the faster we can start getting feedback for fixing things for a great >>>>>> Spark 4.0 final release. >>>>>> >>>>>> >>>>>> >>>>>> So I urge the community to produce a Spark 4.0 Preview soon even if >>>>>> certain features targeting the Delta 4.0 release are still incomplete. >>>>>> >>>>>> >>>>>> >>>>>> Thanks! >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Apr 17, 2024 at 8:35 AM Wenchen Fan <cloud0...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> Thank you all for the replies! >>>>>> >>>>>> >>>>>> >>>>>> To @Nicholas Chammas <nicholas.cham...@gmail.com> : Thanks for >>>>>> cleaning up the error terminology and documentation! I've merged the >>>>>> first >>>>>> PR and let's finish others before the 4.0 release. >>>>>> >>>>>> To @Dongjoon Hyun <dongjoon.h...@gmail.com> : Thanks for driving the >>>>>> ANSI on by default effort! Now the vote has passed, let's flip the config >>>>>> and finish the DataFrame error context feature before 4.0. >>>>>> >>>>>> To @Jungtaek Lim <kabhwan.opensou...@gmail.com> : Ack. We can treat >>>>>> the Streaming state store data source as completed for 4.0 then. >>>>>> >>>>>> To @Cheng Pan <cheng...@apache.org> : Yea we definitely should have >>>>>> a preview release. Let's collect more feedback on the ongoing projects >>>>>> and >>>>>> then we can propose a date for the preview release. >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Apr 17, 2024 at 1:22 PM Cheng Pan <pan3...@gmail.com> wrote: >>>>>> >>>>>> will we have preview release for 4.0.0 like we did for 2.0.0 and >>>>>> 3.0.0? >>>>>> >>>>>> Thanks, >>>>>> Cheng Pan >>>>>> >>>>>> >>>>>> > On Apr 15, 2024, at 09:58, Jungtaek Lim < >>>>>> kabhwan.opensou...@gmail.com> wrote: >>>>>> > >>>>>> > W.r.t. state data source - reader (SPARK-45511), there are several >>>>>> follow-up tickets, but we don't plan to address them soon. The current >>>>>> implementation is the final shape for Spark 4.0.0, unless there are >>>>>> demands >>>>>> on the follow-up tickets. >>>>>> > >>>>>> > We may want to check the plan for transformWithState - my >>>>>> understanding is that we want to release the feature to 4.0.0, but there >>>>>> are several remaining works to be done. While the tentative timeline for >>>>>> releasing is June 2024, what would be the tentative timeline for the RC >>>>>> cut? >>>>>> > (cc. Anish to add more context on the plan for transformWithState) >>>>>> > >>>>>> > On Sat, Apr 13, 2024 at 3:15 AM Wenchen Fan <cloud0...@gmail.com> >>>>>> wrote: >>>>>> > Hi all, >>>>>> > >>>>>> > It's close to the previously proposed 4.0.0 release date (June >>>>>> 2024), and I think it's time to prepare for it and discuss the ongoing >>>>>> projects: >>>>>> > • >>>>>> > ANSI by default >>>>>> > • Spark Connect GA >>>>>> > • Structured Logging >>>>>> > • Streaming state store data source >>>>>> > • new data type VARIANT >>>>>> > • STRING collation support >>>>>> > • Spark k8s operator versioning >>>>>> > Please help to add more items to this list that are missed here. I >>>>>> would like to volunteer as the release manager for Apache Spark 4.0.0 if >>>>>> there is no objection. Thank you all for the great work that fills Spark >>>>>> 4.0! >>>>>> > >>>>>> > Wenchen Fan >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Twitter: https://twitter.com/holdenkarau >>>>>> <https://mailshield.baidu.com/check?q=9DewFnOIsK%2bK64Uu60Jx4QkcL9rDgnApD6spzOBjk%2fa2KQxn> >>>>>> >>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>> https://amzn.to/2MaRAG9 >>>>>> <https://mailshield.baidu.com/check?q=D34Ozfkj%2bFrnkuu9ci%2b4FcMkreOvMZ3jO85bIw%3d%3d> >>>>>> >>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>> <https://mailshield.baidu.com/check?q=nadOZCZjNeU0qOVGCJesf8dvH4OrsWdKamKIxnJncPneWoN8%2bsIqc2DWow8%3d> >>>>>> >>>>>>
