Re: emacs-app-devel build fail on master not detected by buildbot

[Nathaniel W Griswold]

> 
> I'll just weigh in to say that the purpose of our buildbot setup (the way 
> that it is currently conceived) is to build binary packages that are 
> distributed to users when they install ports. The purpose of the buildbot is 
> not to periodically rebuild ports to discover breakages.
> 
> In this case, the problem occurred because the maintainer of the port did not 
> update it before it broke. The maintainer should get into the habit of 
> updating the port before it breaks and/or the maintainer should set the fetch 
> depth such that the port will not break before the next time they update the 
> port.
> 

Ok, thank you for letting my know, Ryan. I did not know this.

Nate

Re: emacs-app-devel build fail on master not detected by buildbot

[Ryan Schmidt]

On Apr 22, 2021, at 21:48, Dan Ports wrote:

> To be clear, the problem here was that MacPorts uses the xcode-provided
> git, and certain older systems have a version of git that's too old to
> support the --shallow-since option, causing the port to break on
> systems where it's buildable today.

MacPorts base does not always use Xcode's git. The code is here:

https://github.com/macports/macports-base/blob/master/src/port1.0/portfetch.tcl#L192-L198

It currently uses Xcode git for OS X 10.10 Yosemite and later and MacPorts git 
for OS X 10.9 Mavericks and earlier.

Re: emacs-app-devel build fail on master not detected by buildbot

[Joshua Root]

On 2021-4-23 15:35 , Christopher Jones wrote:




On 23 Apr 2021, at 3:48 am, Dan Ports  wrote:

To be clear, the problem here was that MacPorts uses the xcode-provided
git, and certain older systems have a version of git that's too old to
support the --shallow-since option, causing the port to break on
systems where it's buildable today.

Now, those systems are pretty old at this point (IIRC, it was El
Capitan and older) so perhaps that is worth reconsidering. We could
also install and use our own version of git on older systems as we do
for some other fetch/extract dependencies, but that would require a
base update.


why a base update ? Just adding port:git as a build dep would do it, as port 
would then use that version instead.


Should be depends_fetch, and might also need to set git.cmd to 
${prefix}/bin/git, but yes, it's certainly possible without a base update.


- Josh

Re: emacs-app-devel build fail on master not detected by buildbot

[Christopher Jones]

> On 23 Apr 2021, at 3:48 am, Dan Ports  wrote:
> 
> On Thu, Apr 22, 2021 at 11:05:34PM +0900, Aaron Madlon-Kay wrote:
>> However it was rejected by the maintainer because he *wants* the current 
>> setup. If the port no longer builds because the referenced commit is more 
>> than 1,000 commits in the past, then the port is ripe for a bump. Increasing 
>> the depth or using a date-based strategy will just balloon the amount of 
>> data fetched.
> 
> To be clear, the problem here was that MacPorts uses the xcode-provided
> git, and certain older systems have a version of git that's too old to
> support the --shallow-since option, causing the port to break on
> systems where it's buildable today.
> 
> Now, those systems are pretty old at this point (IIRC, it was El
> Capitan and older) so perhaps that is worth reconsidering. We could
> also install and use our own version of git on older systems as we do
> for some other fetch/extract dependencies, but that would require a
> base update.

why a base update ? Just adding port:git as a build dep would do it, as port 
would then use that version instead.

Chris

> 
> Dan
> 
> -- 
> Dan R. K. Ports  https://drkp.net/



smime.p7s
Description: S/MIME cryptographic signature

Re: emacs-app-devel build fail on master not detected by buildbot

[Aaron Madlon-Kay]

For the general case an update to MacPorts base would be required, but
for the purposes of any given port shouldn't it be sufficient to do
`depends_fetch-append port:git`?

And my apologies; it wasn't you, Dan, who took the position I
described earlier, it was another commentator:
https://github.com/macports/macports-ports/pull/9295#discussion_r532780239

-Aaron

Re: emacs-app-devel build fail on master not detected by buildbot

[Dan Ports]

On Thu, Apr 22, 2021 at 11:05:34PM +0900, Aaron Madlon-Kay wrote:
> However it was rejected by the maintainer because he *wants* the current 
> setup. If the port no longer builds because the referenced commit is more 
> than 1,000 commits in the past, then the port is ripe for a bump. Increasing 
> the depth or using a date-based strategy will just balloon the amount of data 
> fetched.

To be clear, the problem here was that MacPorts uses the xcode-provided
git, and certain older systems have a version of git that's too old to
support the --shallow-since option, causing the port to break on
systems where it's buildable today.

Now, those systems are pretty old at this point (IIRC, it was El
Capitan and older) so perhaps that is worth reconsidering. We could
also install and use our own version of git on older systems as we do
for some other fetch/extract dependencies, but that would require a
base update.

Dan

-- 
Dan R. K. Ports  https://drkp.net/

Re: emacs-app-devel build fail on master not detected by buildbot

[Ryan Schmidt]

On Apr 22, 2021, at 18:22, Nathaniel W Griswold wrote:

> So are you guys just gonna point to the new commit and otherwise keep it the 
> same? I think maybe it should change.

I'll just weigh in to say that the purpose of our buildbot setup (the way that 
it is currently conceived) is to build binary packages that are distributed to 
users when they install ports. The purpose of the buildbot is not to 
periodically rebuild ports to discover breakages.

In this case, the problem occurred because the maintainer of the port did not 
update it before it broke. The maintainer should get into the habit of updating 
the port before it breaks and/or the maintainer should set the fetch depth such 
that the port will not break before the next time they update the port.

Re: emacs-app-devel build fail on master not detected by buildbot

[Nathaniel W Griswold]

So are you guys just gonna point to the new commit and otherwise keep it the 
same? I think maybe it should change.

Nate

> On Apr 22, 2021, at 4:55 PM, Christopher Jones  
> wrote:
> 
> 
> 
>> On 22 Apr 2021, at 10:44 pm, Aaron Madlon-Kay  wrote:
>> 
>>> all it does is save a bit of bandwidth during the fetch
>> 
>> In the case of Emacs, it saves *gigabytes* during fetch.
> 
> OK, fair enough. I didn’t realise emacs had quite that much history ;)…
> 
>> 
>> The depth restriction was a lifesaver when I worked on the nativecomp 
>> variant a while back. 
>> 
>> -Aaron
> 

Re: emacs-app-devel build fail on master not detected by buildbot

[Christopher Jones]

> On 22 Apr 2021, at 10:44 pm, Aaron Madlon-Kay  wrote:
> 
> > all it does is save a bit of bandwidth during the fetch
> 
> In the case of Emacs, it saves *gigabytes* during fetch.

OK, fair enough. I didn’t realise emacs had quite that much history ;)…

> 
> The depth restriction was a lifesaver when I worked on the nativecomp variant 
> a while back. 
> 
> -Aaron



smime.p7s
Description: S/MIME cryptographic signature

Re: emacs-app-devel build fail on master not detected by buildbot

[Aaron Madlon-Kay]

> all it does is save a bit of bandwidth during the fetch

In the case of Emacs, it saves *gigabytes* during fetch.

The depth restriction was a lifesaver when I worked on the nativecomp
variant a while back.

-Aaron

Re: emacs-app-devel build fail on master not detected by buildbot

[Christopher Jones]

> On 22 Apr 2021, at 10:04 pm, Christopher Jones  
> wrote:
> 
> 
> 
>> On 22 Apr 2021, at 9:59 pm, Nathaniel W Griswold  
>> wrote:
>> 
>> Thank you, Christopher.
>> 
>> Are you saying the date-style depth would be the right way forward? That 
>> seems fine and then the maintainers could either keep up or not. The current 
>> idea of using the breakage event as a signal to update the port file is 
>> kinda bad IMO.
> 
> I was simply stating that updating the port is anyway a good thing to do, and 
> would fix the current issue.
> 
> But yes, on the 1000 depth thing I agree that doesn’t seem like a great thing 
> to be doing in the port file, but thats the maintainers decision to change ….

… but changing it to 3000 doesn’t do anything but delay the inevitable… I would 
just remove it, as we don’t really do this sort of thing elsewhere where we 
perform git fetches, and as fas as I can see all it does is save a bit of 
bandwidth during the fetch…..

> 
> Chris
> 
> 
>> 
>> Nate
>> 
>>> On Apr 22, 2021, at 3:55 PM, Christopher Jones  
>>> wrote:
>>> 
>>> 
>>> 
 On 22 Apr 2021, at 3:05 pm, Aaron Madlon-Kay  wrote:
 
 I proposed in a past PR to emacs-app-devel to use a modern git flag that 
 lets you specify a depth based on commit date. That would be the “real” 
 solution in the direction you’re going.
 
 However it was rejected by the maintainer because he *wants* the current 
 setup. If the port no longer builds because the referenced commit is more 
 than 1,000 commits in the past, then the port is ripe for a bump. 
 Increasing the depth or using a date-based strategy will just balloon the 
 amount of data fetched.
 
 So rather than increasing the depth to 3,000, I recommend you either:
 
 - bump the commit to a recent one, or
 - file a Trac ticket so that someone else is prompted to do so
>>> 
>>> Indeed that is the correct way forward really…
>>> 
>>> https://github.com/macports/macports-ports/commit/6fb61146fb988bd75fe7bc5a209544b30b560692
>>> 
 
 Thanks,
 Aaron
 
 
> On Apr 22, 2021, at 22:29, Nathaniel W Griswold  
> wrote:
> 
> I use the subport emacs-app-devel (subport of emacs) on my 10.15 Catalina 
> system (with variants +imagemagick, +rsvg). The build failed during my 
> last port upgrade outdated and i investigated why.
> 
> The external git mirror (https://github.com/emacs-mirror/emacs.git) has 
> exceeded 1000 new commits since the commit referenced by the Portfile 
> (80e26472206cc44837521ba594cd50e724d9af5c). Since the clone produced from 
> the Portfile uses depth 1000, This means that port cannot check out that 
> commit in its local checkout and the port build fails on that step.
> 
> I thought about it a bit and i feel like if the logic to trigger a build 
> is already Portfile-aware this could be detected with a small change to 
> the system. If a git clone with a —depth=${val} is found in the Portfile 
> for a port or subport, then the build system could trigger a build 
> periodically at some rate that doesn’t stress the build setup too much. I 
> don’t know how many Portfiles have `git clone —depth=${val} ${repo}` 
> git.url values but if there aren’t that many you could trigger these 
> builds quite often.
> 
> I will increase the depth to 3000 for now and submit my updated Portfile.
> 
> Thank you
> 
> Nate
>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: emacs-app-devel build fail on master not detected by buildbot

[Christopher Jones]

> On 22 Apr 2021, at 9:59 pm, Nathaniel W Griswold  wrote:
> 
> Thank you, Christopher.
> 
> Are you saying the date-style depth would be the right way forward? That 
> seems fine and then the maintainers could either keep up or not. The current 
> idea of using the breakage event as a signal to update the port file is kinda 
> bad IMO.

I was simply stating that updating the port is anyway a good thing to do, and 
would fix the current issue.

But yes, on the 1000 depth thing I agree that doesn’t seem like a great thing 
to be doing in the port file, but thats the maintainers decision to change ….

Chris


> 
> Nate
> 
>> On Apr 22, 2021, at 3:55 PM, Christopher Jones  
>> wrote:
>> 
>> 
>> 
>>> On 22 Apr 2021, at 3:05 pm, Aaron Madlon-Kay  wrote:
>>> 
>>> I proposed in a past PR to emacs-app-devel to use a modern git flag that 
>>> lets you specify a depth based on commit date. That would be the “real” 
>>> solution in the direction you’re going.
>>> 
>>> However it was rejected by the maintainer because he *wants* the current 
>>> setup. If the port no longer builds because the referenced commit is more 
>>> than 1,000 commits in the past, then the port is ripe for a bump. 
>>> Increasing the depth or using a date-based strategy will just balloon the 
>>> amount of data fetched.
>>> 
>>> So rather than increasing the depth to 3,000, I recommend you either:
>>> 
>>> - bump the commit to a recent one, or
>>> - file a Trac ticket so that someone else is prompted to do so
>> 
>> Indeed that is the correct way forward really…
>> 
>> https://github.com/macports/macports-ports/commit/6fb61146fb988bd75fe7bc5a209544b30b560692
>> 
>>> 
>>> Thanks,
>>> Aaron
>>> 
>>> 
 On Apr 22, 2021, at 22:29, Nathaniel W Griswold  
 wrote:
 
 I use the subport emacs-app-devel (subport of emacs) on my 10.15 Catalina 
 system (with variants +imagemagick, +rsvg). The build failed during my 
 last port upgrade outdated and i investigated why.
 
 The external git mirror (https://github.com/emacs-mirror/emacs.git) has 
 exceeded 1000 new commits since the commit referenced by the Portfile 
 (80e26472206cc44837521ba594cd50e724d9af5c). Since the clone produced from 
 the Portfile uses depth 1000, This means that port cannot check out that 
 commit in its local checkout and the port build fails on that step.
 
 I thought about it a bit and i feel like if the logic to trigger a build 
 is already Portfile-aware this could be detected with a small change to 
 the system. If a git clone with a —depth=${val} is found in the Portfile 
 for a port or subport, then the build system could trigger a build 
 periodically at some rate that doesn’t stress the build setup too much. I 
 don’t know how many Portfiles have `git clone —depth=${val} ${repo}` 
 git.url values but if there aren’t that many you could trigger these 
 builds quite often.
 
 I will increase the depth to 3000 for now and submit my updated Portfile.
 
 Thank you
 
 Nate
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: emacs-app-devel build fail on master not detected by buildbot

[Nathaniel W Griswold]

Thank you, Christopher.

Are you saying the date-style depth would be the right way forward? That seems 
fine and then the maintainers could either keep up or not. The current idea of 
using the breakage event as a signal to update the port file is kinda bad IMO.

Nate

> On Apr 22, 2021, at 3:55 PM, Christopher Jones  
> wrote:
> 
> 
> 
>> On 22 Apr 2021, at 3:05 pm, Aaron Madlon-Kay  wrote:
>> 
>> I proposed in a past PR to emacs-app-devel to use a modern git flag that 
>> lets you specify a depth based on commit date. That would be the “real” 
>> solution in the direction you’re going.
>> 
>> However it was rejected by the maintainer because he *wants* the current 
>> setup. If the port no longer builds because the referenced commit is more 
>> than 1,000 commits in the past, then the port is ripe for a bump. Increasing 
>> the depth or using a date-based strategy will just balloon the amount of 
>> data fetched.
>> 
>> So rather than increasing the depth to 3,000, I recommend you either:
>> 
>> - bump the commit to a recent one, or
>> - file a Trac ticket so that someone else is prompted to do so
> 
> Indeed that is the correct way forward really…
> 
> https://github.com/macports/macports-ports/commit/6fb61146fb988bd75fe7bc5a209544b30b560692
> 
>> 
>> Thanks,
>> Aaron
>> 
>> 
>>> On Apr 22, 2021, at 22:29, Nathaniel W Griswold  
>>> wrote:
>>> 
>>> I use the subport emacs-app-devel (subport of emacs) on my 10.15 Catalina 
>>> system (with variants +imagemagick, +rsvg). The build failed during my last 
>>> port upgrade outdated and i investigated why.
>>> 
>>> The external git mirror (https://github.com/emacs-mirror/emacs.git) has 
>>> exceeded 1000 new commits since the commit referenced by the Portfile 
>>> (80e26472206cc44837521ba594cd50e724d9af5c). Since the clone produced from 
>>> the Portfile uses depth 1000, This means that port cannot check out that 
>>> commit in its local checkout and the port build fails on that step.
>>> 
>>> I thought about it a bit and i feel like if the logic to trigger a build is 
>>> already Portfile-aware this could be detected with a small change to the 
>>> system. If a git clone with a —depth=${val} is found in the Portfile for a 
>>> port or subport, then the build system could trigger a build periodically 
>>> at some rate that doesn’t stress the build setup too much. I don’t know how 
>>> many Portfiles have `git clone —depth=${val} ${repo}` git.url values but if 
>>> there aren’t that many you could trigger these builds quite often.
>>> 
>>> I will increase the depth to 3000 for now and submit my updated Portfile.
>>> 
>>> Thank you
>>> 
>>> Nate

Re: emacs-app-devel build fail on master not detected by buildbot

[Christopher Jones]

> On 22 Apr 2021, at 3:05 pm, Aaron Madlon-Kay  wrote:
> 
> I proposed in a past PR to emacs-app-devel to use a modern git flag that lets 
> you specify a depth based on commit date. That would be the “real” solution 
> in the direction you’re going.
> 
> However it was rejected by the maintainer because he *wants* the current 
> setup. If the port no longer builds because the referenced commit is more 
> than 1,000 commits in the past, then the port is ripe for a bump. Increasing 
> the depth or using a date-based strategy will just balloon the amount of data 
> fetched.
> 
> So rather than increasing the depth to 3,000, I recommend you either:
> 
> - bump the commit to a recent one, or
> - file a Trac ticket so that someone else is prompted to do so

Indeed that is the correct way forward really…

https://github.com/macports/macports-ports/commit/6fb61146fb988bd75fe7bc5a209544b30b560692
 


> 
> Thanks,
> Aaron
> 
> 
>> On Apr 22, 2021, at 22:29, Nathaniel W Griswold  wrote:
>> 
>> I use the subport emacs-app-devel (subport of emacs) on my 10.15 Catalina 
>> system (with variants +imagemagick, +rsvg). The build failed during my last 
>> port upgrade outdated and i investigated why.
>> 
>> The external git mirror (https://github.com/emacs-mirror/emacs.git) has 
>> exceeded 1000 new commits since the commit referenced by the Portfile 
>> (80e26472206cc44837521ba594cd50e724d9af5c). Since the clone produced from 
>> the Portfile uses depth 1000, This means that port cannot check out that 
>> commit in its local checkout and the port build fails on that step.
>> 
>> I thought about it a bit and i feel like if the logic to trigger a build is 
>> already Portfile-aware this could be detected with a small change to the 
>> system. If a git clone with a —depth=${val} is found in the Portfile for a 
>> port or subport, then the build system could trigger a build periodically at 
>> some rate that doesn’t stress the build setup too much. I don’t know how 
>> many Portfiles have `git clone —depth=${val} ${repo}` git.url values but if 
>> there aren’t that many you could trigger these builds quite often.
>> 
>> I will increase the depth to 3000 for now and submit my updated Portfile.
>> 
>> Thank you
>> 
>> Nate
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: emacs-app-devel build fail on master not detected by buildbot

[Aaron Madlon-Kay]

I proposed in a past PR to emacs-app-devel to use a modern git flag that lets 
you specify a depth based on commit date. That would be the “real” solution in 
the direction you’re going.

However it was rejected by the maintainer because he *wants* the current setup. 
If the port no longer builds because the referenced commit is more than 1,000 
commits in the past, then the port is ripe for a bump. Increasing the depth or 
using a date-based strategy will just balloon the amount of data fetched.

So rather than increasing the depth to 3,000, I recommend you either:

- bump the commit to a recent one, or
- file a Trac ticket so that someone else is prompted to do so

Thanks,
Aaron


> On Apr 22, 2021, at 22:29, Nathaniel W Griswold  wrote:
> 
> I use the subport emacs-app-devel (subport of emacs) on my 10.15 Catalina 
> system (with variants +imagemagick, +rsvg). The build failed during my last 
> port upgrade outdated and i investigated why.
> 
> The external git mirror (https://github.com/emacs-mirror/emacs.git) has 
> exceeded 1000 new commits since the commit referenced by the Portfile 
> (80e26472206cc44837521ba594cd50e724d9af5c). Since the clone produced from the 
> Portfile uses depth 1000, This means that port cannot check out that commit 
> in its local checkout and the port build fails on that step.
> 
> I thought about it a bit and i feel like if the logic to trigger a build is 
> already Portfile-aware this could be detected with a small change to the 
> system. If a git clone with a —depth=${val} is found in the Portfile for a 
> port or subport, then the build system could trigger a build periodically at 
> some rate that doesn’t stress the build setup too much. I don’t know how many 
> Portfiles have `git clone —depth=${val} ${repo}` git.url values but if there 
> aren’t that many you could trigger these builds quite often.
> 
> I will increase the depth to 3000 for now and submit my updated Portfile.
> 
> Thank you
> 
> Nate

Re: emacs-app-devel build fail on master not detected by buildbot

[Nathaniel W Griswold]

While i agree with the maintainer that the port should be kept up to date, i 
did spend quite a bit of time investigating this issue on my machine. I hadn’t 
worked on a port in a while so i had to get the rust off and spend time on 
digging to find out what was wrong. While i wasn’t too busy today, and that’s 
the reason i was able to dig into it, it remains that this was an issue that 
has already come up and so this is actually time that i could have spent on my 
actual work instead. I’m sure it will happen to other people. We need a 
solution so it doesn’t happen again. Your recommendation to me in keeping with 
the maintainer’s solutions does not seem it will prevent the problem from 
reocurring. Additionally, the solution you previously proposed to the 
maintainer does not keep the package up to date with git.

Adding the ability to explicitly flag the port as ‘volatile’ or some such 
either directly in the Portfile or in any build meta info that exists would 
prevent people who are mostly just users like myself from suffering lost time 
in their day because of broken builds. Even if we all go to macports support 
when it happens and the dev team does all the work, it’s still a problem. I 
think this should be automated in the buildbot, or all maintainers of such 
ports should get some kind of regular reminder to update their port, or such 
ports should just have an explicit shelf life to compute future expiration 
date, or pretty much anything so users don’t suffer the consequences.

Do you agree?

Nate

> On Apr 22, 2021, at 9:05 AM, Aaron Madlon-Kay  wrote:
> 
> I proposed in a past PR to emacs-app-devel to use a modern git flag that lets 
> you specify a depth based on commit date. That would be the “real” solution 
> in the direction you’re going.
> 
> However it was rejected by the maintainer because he *wants* the current 
> setup. If the port no longer builds because the referenced commit is more 
> than 1,000 commits in the past, then the port is ripe for a bump. Increasing 
> the depth or using a date-based strategy will just balloon the amount of data 
> fetched.
> 
> So rather than increasing the depth to 3,000, I recommend you either:
> 
> - bump the commit to a recent one, or
> - file a Trac ticket so that someone else is prompted to do so
> 
> Thanks,
> Aaron
> 
> 
>> On Apr 22, 2021, at 22:29, Nathaniel W Griswold  wrote:
>> 
>> I use the subport emacs-app-devel (subport of emacs) on my 10.15 Catalina 
>> system (with variants +imagemagick, +rsvg). The build failed during my last 
>> port upgrade outdated and i investigated why.
>> 
>> The external git mirror (https://github.com/emacs-mirror/emacs.git) has 
>> exceeded 1000 new commits since the commit referenced by the Portfile 
>> (80e26472206cc44837521ba594cd50e724d9af5c). Since the clone produced from 
>> the Portfile uses depth 1000, This means that port cannot check out that 
>> commit in its local checkout and the port build fails on that step.
>> 
>> I thought about it a bit and i feel like if the logic to trigger a build is 
>> already Portfile-aware this could be detected with a small change to the 
>> system. If a git clone with a —depth=${val} is found in the Portfile for a 
>> port or subport, then the build system could trigger a build periodically at 
>> some rate that doesn’t stress the build setup too much. I don’t know how 
>> many Portfiles have `git clone —depth=${val} ${repo}` git.url values but if 
>> there aren’t that many you could trigger these builds quite often.
>> 
>> I will increase the depth to 3000 for now and submit my updated Portfile.
>> 
>> Thank you
>> 
>> Nate
>