Thanks Zoltan. Makes sense. Also, we should definitely strive to release
within 180 days especially when there are lots of commits to a branch.

-Vihang

On Fri, May 26, 2023 at 12:04 AM Zoltan Haindrich <k...@rxd.hu> wrote:

> On 5/25/23 19:58, vihang karajgaonkar wrote:
> > I just tried the job and it worked as expected. Thanks! If I understand
> > correctly, the job retains builds for 180 days. Does it mean if there
> were
> > no commits to a branch for more than 180 days, we will lose the build
> > artifacts eventually?
>
> not entirely - the removal of old builds is a post-build action; which
> means - if there are no builds; the removal logic will never run
> https://plugins.jenkins.io/discard-old-build/
>
> on the other hand I wonder how much value a nightly build can still
> provide after 180 days :)
> preferably - a real release should be done after some time :)
>
> cheers,
> Zoltan
>
> >
> > On Thu, May 25, 2023 at 1:50 AM Zoltan Haindrich <k...@rxd.hu> wrote:
> >
> >> Hey Vihang,
> >>
> >> I've added you as an admin; and I've copied the job as
> >> http://ci.hive.apache.org/job/hive-nightly-branch-3/
> >> other option could be to trigger the original job or use
> >> parameterized-scheduler  but that would configure a real unconditional
> >> nightly build - which will just build the
> >> same version over-and-over again if there are no changes...
> >> ...the current nighly is SCM triggered ; but only once-a-day it makes a
> >> check which creates the desired results.
> >>
> >> the least painfull was to copy the job; I guess no-one touched the
> >> pipeline script ever since it was introduced :D
> >>
> >> cheers,
> >> Zoltan
> >>
> >> On 5/25/23 01:26, vihang karajgaonkar wrote:
> >>> I created https://issues.apache.org/jira/browse/HIVE-27371 to have
> >> nightly
> >>> builds for branch-3. Once that is merged, I think we can have scheduled
> >>> builds for branch-3 as well. Although, I don't have permissions to
> >> create a
> >>> new job for branch-3. Does anyone know how to do it?
> >>>
> >>> Thanks,
> >>> Vihang
> >>>
> >>> On Wed, May 24, 2023 at 10:07 AM vihang karajgaonkar <
> >> vihan...@apache.org>
> >>> wrote:
> >>>
> >>>> The nightly job http://ci.hive.apache.org/job/hive-nightly/ is great.
> >> Can
> >>>> we have this for branch-3 as well since we have been backporting a lot
> >> of
> >>>> PRs to branch-3 lately.
> >>>>
> >>>> Thanks,
> >>>> Vihang
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Wed, May 24, 2023 at 6:56 AM Zoltan Haindrich <k...@rxd.hu> wrote:
> >>>>
> >>>>> Hey,
> >>>>>
> >>>>>    > We already have nightly builds for Hive [1].
> >>>>>    > [1] http://ci.hive.apache.org/job/hive-nightly/
> >>>>>
> >>>>> ...and hive-dev-box can launch such archives; either by using it like
> >>>>> this:
> >>>>> https://www.mail-archive.com/dev@hive.apache.org/msg142420.html
> >>>>>
> >>>>> or with a somewhat longer command you could launch hdb in bazaar
> mode;
> >>>>> and have an HS2 running with a nightly version:
> >>>>>
> >>>>> docker run --rm -d -p 10000:10000 -v hive-dev-box_work:/work -e
> >>>>> HIVE_VERSION=
> >>>>>
> >>
> http://ci.hive.apache.org/job/hive-nightly/lastSuccessfulBuild/artifact/archive/apache-hive-4.0.0-nightly-b0b3fde70c-20230524_014711-bin.tar.gz
> >>>>> --name hive
> >>>>> kgyrtkirk/hive-dev-box:bazaar
> >>>>>
> >>>>> cheers,
> >>>>> Zoltan
> >>>>>
> >>>>> On 5/24/23 09:15, Stamatis Zampetakis wrote:
> >>>>>> Hey all,
> >>>>>>
> >>>>>> We already have nightly builds for Hive [1].
> >>>>>>
> >>>>>> Do we need something more than that?
> >>>>>>
> >>>>>> Best,
> >>>>>> Stamatis
> >>>>>>
> >>>>>> [1] http://ci.hive.apache.org/job/hive-nightly/
> >>>>>>
> >>>>>>
> >>>>>> On Tue, May 23, 2023 at 9:03 AM vihang karajgaonkar <
> >>>>> vihan...@apache.org> wrote:
> >>>>>>>
> >>>>>>> I think there are many benefits like others in this thread
> suggested
> >>>>> which
> >>>>>>> can be built on top of nightly builds. Having docker images is
> great
> >>>>> but
> >>>>>>> for now I think we can start simple and publish the jars. Many
> users
> >>>>> still
> >>>>>>> just deploy using jars and it would be useful to them. Once we
> have a
> >>>>>>> docker environment we can add a docker image too to the nightly
> >> builds
> >>>>> so
> >>>>>>> that users can choose their preferred way.
> >>>>>>>
> >>>>>>> On Mon, May 22, 2023 at 11:07 PM Sungwoo Park <glap...@gmail.com>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> I think such nightly builds will be useful for testing and
> debugging
> >>>>> in the
> >>>>>>>> future.
> >>>>>>>>
> >>>>>>>> I also wonder if we can somehow create builds even from previous
> >>>>> commits
> >>>>>>>> (e.g., for the past few years). Such builds from previous commits
> >>>>> don't
> >>>>>>>> have to be daily builds, and I think weekly builds (or even
> monthly
> >>>>> builds)
> >>>>>>>> would also be very useful.
> >>>>>>>>
> >>>>>>>> The reason I wish such builds were available is to facilitate
> >>>>> debugging and
> >>>>>>>> testing. When tested against the TPC-DS benchmark, the current
> >> master
> >>>>>>>> branch has several correctness problems that were introduced after
> >> the
> >>>>>>>> release of Hive 3.1.2. We have reported all problems known to us
> in
> >>>>> [1] and
> >>>>>>>> also submitted several patches. If such nightly builds had been
> >>>>> available,
> >>>>>>>> we would have saved quite a bit of time for implementing the
> patches
> >>>>> by
> >>>>>>>> quickly finding offending commits that introduced new correctness
> >>>>> bugs.
> >>>>>>>>
> >>>>>>>> In addition, you can find quite a few commits in the master branch
> >>>>> that
> >>>>>>>> report bugs which are not reproduced in Hive 3.1.2. Examples:
> >>>>> HIVE-19990,
> >>>>>>>> HIVE-14557, HIVE-21132, HIVE-21188, HIVE-21544, HIVE-22114,
> >>>>>>>> HIVE-22227, HIVE-22236, HIVE-23911, HIVE-24198, HIVE-22777,
> >>>>>>>> HIVE-25170, HIVE-25864, HIVE-26671.
> >>>>>>>> (There may be some errors in this list because we compared against
> >>>>> Hive
> >>>>>>>> 3.1.2 with many patches backported.) Such nightly builds can be
> >>>>> useful for
> >>>>>>>> finding root causes of such bugs.
> >>>>>>>>
> >>>>>>>> Ideally I wish there was an automated procedure to create nightly
> >>>>> builds,
> >>>>>>>> run TPC-DS benchmark, and report correctness/performance results,
> >>>>> although
> >>>>>>>> this would be quite hard to implement. (I remember Spark
> implemented
> >>>>> this
> >>>>>>>> procedure in the era of Spark 2, but my memory could be wrong.)
> >>>>>>>>
> >>>>>>>> [1] https://issues.apache.org/jira/browse/HIVE-26654
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, May 23, 2023 at 10:44 AM Ayush Saxena <ayush...@gmail.com
> >
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Vihang,
> >>>>>>>>> +1, We were even exploring publishing the docker images of the
> >>>>> snapshot
> >>>>>>>>> version as well per commit or maybe weekly, so just shoot 2
> docker
> >>>>>>>> commands
> >>>>>>>>> and you get a Hive cluster running with master code.
> >>>>>>>>>
> >>>>>>>>> Sai, I think to spin up an env via Docker with all these things
> >>>>> should be
> >>>>>>>>> doable for sure, but would require someone with real good
> expertise
> >>>>> with
> >>>>>>>>> docker as well as setting up these services with Hive.
> Obviously, I
> >>>>> am
> >>>>>>>> not
> >>>>>>>>> that guy :-)
> >>>>>>>>>
> >>>>>>>>> @Simhadri has a PR which publishes docker images once a release
> tag
> >>>>> is
> >>>>>>>>> pushed, you can explore to have similar stuff for the Snapshot
> >>>>> version,
> >>>>>>>>> maybe if that sounds cool
> >>>>>>>>>
> >>>>>>>>> -Ayush
> >>>>>>>>>
> >>>>>>>>> On Tue, 23 May 2023 at 04:26, Sai Hemanth Gantasala
> >>>>>>>>> <saihema...@cloudera.com.invalid> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Vihang,
> >>>>>>>>>>
> >>>>>>>>>> +1 on the idea.
> >>>>>>>>>>
> >>>>>>>>>> This is a great idea to quickly test if a certain feature is
> >>>>> working as
> >>>>>>>>>> expected on a certain branch.
> >>>>>>>>>> This way we test data loss, correctness, or any other unexpected
> >>>>>>>>> scenarios
> >>>>>>>>>> that are Hive specific only. However, I'm wondering if it is
> >>>>> possible
> >>>>>>>> to
> >>>>>>>>>> deploy/test in a kerberized environment or issues involving
> >>>>>>>> authorization
> >>>>>>>>>> services like sentry/ranger.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Sai.
> >>>>>>>>>>
> >>>>>>>>>> On Mon, May 22, 2023 at 11:15 AM vihang karajgaonkar <
> >>>>>>>>> vihan...@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hello Team,
> >>>>>>>>>>>
> >>>>>>>>>>> I have observed that it is a common use-case where users would
> >> like
> >>>>>>>> to
> >>>>>>>>>> test
> >>>>>>>>>>> out unreleased features/bug fixes either to unblock them or
> test
> >>>>> out
> >>>>>>>> if
> >>>>>>>>>> the
> >>>>>>>>>>> bug fixes really work as intended in their environments. Today
> in
> >>>>> the
> >>>>>>>>>> case
> >>>>>>>>>>> of Apache Hive, this is not very user friendly because it
> >> requires
> >>>>>>>> the
> >>>>>>>>>> end
> >>>>>>>>>>> user to build the binaries directly from the hive source code.
> >>>>>>>>>>>
> >>>>>>>>>>> I found that Apache Spark has a very useful infrastructure [1]
> >>>>> which
> >>>>>>>>>>> deploys nightly snapshots [2] [3] from the branch using github
> >>>>>>>> actions.
> >>>>>>>>>>> This is super useful for any user who wants to try out the
> latest
> >>>>> and
> >>>>>>>>>>> greatest using the nightly builds.
> >>>>>>>>>>>
> >>>>>>>>>>> I was wondering if we should also adopt this. We can use github
> >>>>>>>> actions
> >>>>>>>>>> to
> >>>>>>>>>>> upload the snapshot jars to the public repository (e.g github
> >>>>>>>> packages)
> >>>>>>>>>> and
> >>>>>>>>>>> schedule it as a nightly job.
> >>>>>>>>>>>
> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/INFRA-21167
> >>>>>>>>>>> [2]
> >>>>>>>>>
> >> https://github.com/apache/spark/pkgs/container/apache-spark-ci-image
> >>>>>>>>>>> [3] https://github.com/apache/spark/pull/30623
> >>>>>>>>>>>
> >>>>>>>>>>> I can take a stab at this if the community thinks that this is
> a
> >>>>> nice
> >>>>>>>>>> thing
> >>>>>>>>>>> to have.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Vihang
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>

Reply via email to