Re: request for port create command, to build a portfile from a URL

[Rainer Müller]

On 2018-03-11 23:54, Enrico Maria Crisostomo wrote:
>> However, when we are going to add more templates, lots of options will
>> be repeated in each of them. If we are later going to change any of
>> these, we have to do this in all templates. I am not sure how much of a
>> problem that will be.
>>
>> Also, such a template approach will never allow to combine different
>> features, like using github port group for fetching, but the python port
>> group for building the port.
>>
> 
> The prototype uses just one monolithic template, but I'd expect the following 
> to be true:
> 
>   * Templates may not need to be one per possible output, they could be finer 
> grained and get concatenated.
> 
>   * Dynamic content could be generated and interleaved with template output, 
> if necessary.
> 
> I think I share your feelings about templates, and I have little idea about 
> how complex the output can/should be.

I just wanted to point out what the limitations of this approach could
be. Maybe we should not worry too much about it.  It is totally fine if
some ports are too complex for this script, because it is meant to cover
the most common cases and will never work for all software projects out
there.

Another thing this script could do would be to fetch and extract the
distfile and try to guess the build system by checking for the existence
of "configure", "Makefile", "CMakeLists.txt", etc. and set up the
configure/build/destroot phases accordingly.

Maybe at some later point we could even try to merge this with scripts
like pypi2port, that use the language-specific package manager to fill
the list of dependencies of modules. Then we could have a single
interface for all these generators. But that is really for much later,
let's start with the easy tasks.

Rainer

Re: request for port create command, to build a portfile from a URL

[Enrico Maria Crisostomo]

> On 12 Mar 2018, at 08:28, Ryan Schmidt  wrote:
> 
> 
> On Mar 9, 2018, at 13:22, Enrico Maria Crisostomo wrote:
> 
>> I pushed to GitHub a skeleton of the idea:
>> 
>>   https://github.com/emcrisostomo/macports-utils
>> 
>> I've moved what I'm using to a new script to see what the end result looked 
>> like. If you want to try it, just grab the release tarball here (if you 
>> don't have the Autotools installed):
>> 
>>   ./configure && make install
>> 
>> otherwise just clone the repo, bootstrap it and use it:
>> 
>>   ./autogen.sh && ./configure && make install
>> 
>> An example:
>> 
>>   $ port-gen --url 
>> https://github.com/emcrisostomo/semver-utils/releases/download/1.1.3/semver-utils-1.1.3.tar.gz
>> 
>> outputs:
>> 
>>   # -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; 
>> c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4
>> 
>>   PortSystem  1.0
>>   PortGroup   github 1.0
>> 
>>   github.setupemcrisostomo semver-utils 1.1.3
>>   github.tarball_from releases
>> 
>>   categories  changeme
>>   platforms   darwin
>>   license GPL-3
>> 
>>   maintainers obfuscated-maintainer-mail \
>>   openmaintainer
>>   description Add a short port description here.
>>   long_descriptionAdd a long port description here.
>> 
>>   homepagehttps://github.com/emcrisostomo/semver-utils
>> 
>>   checksums   md5e65be62dc9e25af8aa467aa99cde1e00 \
>>   rmd160 71cf46420315edd8019d6974062033480b5c79a0 \
>>   sha256 
>> 888a688feabc82ce59abc754c63fd2babff5747f0463fb1a3f8fffaf50d5d982 \
>>   size   514429
>> 
>>   livecheck.url   ${github.homepage}/releases/latest
> 
> The github portgroup takes care of setting the homepage and livecheck for you.
> 
> 

Thanks Ryan: that's why the expected output must be reviewed and/or designed by 
the experts.

Re: request for port create command, to build a portfile from a URL

[Ryan Schmidt]

On Mar 9, 2018, at 13:22, Enrico Maria Crisostomo wrote:

> I pushed to GitHub a skeleton of the idea:
> 
>https://github.com/emcrisostomo/macports-utils
> 
> I've moved what I'm using to a new script to see what the end result looked 
> like. If you want to try it, just grab the release tarball here (if you don't 
> have the Autotools installed):
> 
>./configure && make install
> 
> otherwise just clone the repo, bootstrap it and use it:
> 
>./autogen.sh && ./configure && make install
> 
> An example:
> 
>$ port-gen --url 
> https://github.com/emcrisostomo/semver-utils/releases/download/1.1.3/semver-utils-1.1.3.tar.gz
> 
> outputs:
> 
># -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; 
> c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4
> 
>PortSystem  1.0
>PortGroup   github 1.0
> 
>github.setupemcrisostomo semver-utils 1.1.3
>github.tarball_from releases
> 
>categories  changeme
>platforms   darwin
>license GPL-3
> 
>maintainers obfuscated-maintainer-mail \
>openmaintainer
>description Add a short port description here.
>long_descriptionAdd a long port description here.
> 
>homepagehttps://github.com/emcrisostomo/semver-utils
> 
>checksums   md5e65be62dc9e25af8aa467aa99cde1e00 \
>rmd160 71cf46420315edd8019d6974062033480b5c79a0 \
>sha256 
> 888a688feabc82ce59abc754c63fd2babff5747f0463fb1a3f8fffaf50d5d982 \
>size   514429
> 
>livecheck.url   ${github.homepage}/releases/latest

The github portgroup takes care of setting the homepage and livecheck for you.

Re: request for port create command, to build a portfile from a URL

[Enrico Maria Crisostomo]

> On 11 Mar 2018, at 16:40, Rainer Müller  wrote:
> 
> On 2018-03-09 20:22, Enrico Maria Crisostomo wrote:
>> I pushed to GitHub a skeleton of the idea:
>> 
>>https://github.com/emcrisostomo/macports-utils
>> 
>> I've moved what I'm using to a new script to see what the end result looked 
>> like. If you want to try it, just grab the release tarball here (if you 
>> don't have the Autotools installed):
>> 
>>./configure && make install
>> 
>> otherwise just clone the repo, bootstrap it and use it:
>> 
>>./autogen.sh && ./configure && make install
>> 
>> An example:
>> 
>>$ port-gen --url 
>> https://github.com/emcrisostomo/semver-utils/releases/download/1.1.3/semver-utils-1.1.3.tar.gz
> 
> Nice work! Looks already very polished for a prototype.

Thanks Rainer.

> 
> I think it would be even nicer to just specify the URL to the project
> (or to the tag) and it will find the latest distfile to download itself.

Yes.

> 
> However, when we are going to add more templates, lots of options will
> be repeated in each of them. If we are later going to change any of
> these, we have to do this in all templates. I am not sure how much of a
> problem that will be.
> 
> Also, such a template approach will never allow to combine different
> features, like using github port group for fetching, but the python port
> group for building the port.
> 

The prototype uses just one monolithic template, but I'd expect the following 
to be true:

  * Templates may not need to be one per possible output, they could be finer 
grained and get concatenated.

  * Dynamic content could be generated and interleaved with template output, if 
necessary.

I think I share your feelings about templates, and I have little idea about how 
complex the output can/should be.


> Rainer

Re: request for port create command, to build a portfile from a URL

[Rainer Müller]

On 2018-03-09 20:22, Enrico Maria Crisostomo wrote:
> I pushed to GitHub a skeleton of the idea:
> 
> https://github.com/emcrisostomo/macports-utils
> 
> I've moved what I'm using to a new script to see what the end result looked 
> like. If you want to try it, just grab the release tarball here (if you don't 
> have the Autotools installed):
> 
> ./configure && make install
> 
> otherwise just clone the repo, bootstrap it and use it:
> 
> ./autogen.sh && ./configure && make install
> 
> An example:
> 
> $ port-gen --url 
> https://github.com/emcrisostomo/semver-utils/releases/download/1.1.3/semver-utils-1.1.3.tar.gz

Nice work! Looks already very polished for a prototype.

I think it would be even nicer to just specify the URL to the project
(or to the tag) and it will find the latest distfile to download itself.

However, when we are going to add more templates, lots of options will
be repeated in each of them. If we are later going to change any of
these, we have to do this in all templates. I am not sure how much of a
problem that will be.

Also, such a template approach will never allow to combine different
features, like using github port group for fetching, but the python port
group for building the port.

Rainer

Re: request for port create command, to build a portfile from a URL

[Enrico Maria Crisostomo]

Ken,

I pushed to GitHub a skeleton of the idea:

https://github.com/emcrisostomo/macports-utils

I've moved what I'm using to a new script to see what the end result looked 
like. If you want to try it, just grab the release tarball here (if you don't 
have the Autotools installed):

./configure && make install

otherwise just clone the repo, bootstrap it and use it:

./autogen.sh && ./configure && make install

An example:

$ port-gen --url 
https://github.com/emcrisostomo/semver-utils/releases/download/1.1.3/semver-utils-1.1.3.tar.gz

outputs:

# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; 
c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

PortSystem  1.0
PortGroup   github 1.0

github.setupemcrisostomo semver-utils 1.1.3
github.tarball_from releases

categories  changeme
platforms   darwin
license GPL-3

maintainers obfuscated-maintainer-mail \
openmaintainer
description Add a short port description here.
long_descriptionAdd a long port description here.

homepagehttps://github.com/emcrisostomo/semver-utils

checksums   md5e65be62dc9e25af8aa467aa99cde1e00 \
rmd160 71cf46420315edd8019d6974062033480b5c79a0 \
sha256 
888a688feabc82ce59abc754c63fd2babff5747f0463fb1a3f8fffaf50d5d982 \
size   514429

livecheck.url   ${github.homepage}/releases/latest


The template is intuitive:

$ cat /usr/local/share/macports-utils/templates/github.in
# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; 
c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

PortSystem  1.0
PortGroup   github 1.0

github.setup${GITHUB_AUTHOR} ${PORT_NAME} ${PORT_VERSION}
github.tarball_from releases
   
categories  changeme
platforms   darwin
license GPL-3

maintainers obfuscated-maintainer-mail 
openmaintainer
description Add a short port description here.
long_descriptionAdd a long port description here.

homepage${GITHUB_REPO_URL}

checksums   md5${CHECKSUM_MD5} 
rmd160 ${CHECKSUM_RMD160} 
sha256 ${CHECKSUM_SHA256} 
size   ${CHECKSUM_SIZE}

livecheck.url   \${github.homepage}/releases/latest

> On 7 Mar 2018, at 10:23, Enrico Maria Crisostomo 
>  wrote:
> 
> Hi Ken,
> 
> I think it's a great idea. I'm maintaining a lot of software both on my 
> personal machines and at work using MacPorts and I've setup some local 
> Makefile targets to do some of this work. Perhaps I could contribute some 
> work on this area, but first I'd like to know what is it that we'd like to 
> have.
> 
> What is currently working for me is a very simple, template-based approach. 
> Since 90% of the time I'm dealing with ports using the github portgroup, I'm 
> basically calculating the relevant variables (port name, version, checksums, 
> etc.) and fill in a template. The template is a file where shell expansion is 
> performed (i.e.: variables in the form ${VAR}) are expanded.) I think this 
> approach could be both very useful and simple to maintain if we had a list of 
> pre-defined templates. It doesn't scale well with the number of different 
> templates to support and their complexity. If we needed templates to have 
> logic in it (flow control, expressions, etc.),then my guts say we should move 
> to a suitable templating engine (or a different kind of solution altogether).
> 
> I've had a look at portfile-gen and it's a nice source of inspiration. Yet, 
> I'm not convinced that outputting text directly from the program flow is 
> either easy to read or easily maintainable, especially if complexity goes up.
> 
> portfile-gen, if I understand correctly, supports the following groups:
> 
>  * perl5
>  * php 
>  * python 
>  * ruby
>  * github
> 
> If we extracted a template for each one of those groups (or more templates in 
> case a group may have multiple templates), then the effort required to come 
> up with a working prototype would be implementing the functions required to 
> populate the environment (i.e.: downloading a file if needed, calculating 
> checksums, examining it). Perhaps we would have to split a Portfile in 
> multiple segments, and generate each segment from a specific template (e.g.: 
> if a port requires GNU Autotools).
> 
> What are your gut feelings about a template-based solution? Do you think it 
> would be able to manage the level of Portfile complexity we desire for this 
> tool?
> 
> Cheers,
> -- 
> Enrico
> 
> 
>> On 7 Mar 2018, at 01:27, Ken Cunningham  
>> wrote:
>> 

Re: request for port create command, to build a portfile from a URL

[Enrico Maria Crisostomo]

Thanks Ryan for the explanation.

> On 7 Mar 2018, at 19:51, Ryan Schmidt  wrote:
> 
> 
> On Mar 7, 2018, at 07:32, Rainer Müller wrote:
> 
>> On 2018-03-07 11:32, Enrico Maria Crisostomo wrote:
>>> This is a little bit offtopic IMHO, anyway: I found surprising too that
>>> the github portgroup was not documented, given the high percentage of
>>> software that I build out of github repositories, and that's why I
>>> recently contributed the documentation of the github portgroup:
>>> 
>>> https://github.com/macports/macports-guide/pull/12
>>> 
>>> It has been merged some days ago. Yet, I don't see it online yet.
>> 
>> That is because we still have not any automatic deployment on
>> build.macports.org for any of our websites. I am irregularly running
>> these jobs from my local buildbot setup on my personal MacBook every few
>> weeks, which is the only reason they are not completely outdated.
>> 
>> I triggered another run of these jobs, so these changes should be
>> available shortly online.
> 
> We now have one of the new "jobs" running on the buildbot, so hopefully 
> adding the others will be easy.
> 

Re: request for port create command, to build a portfile from a URL

[Rainer Müller]

On 2018-03-07 11:32, Enrico Maria Crisostomo wrote:
> This is a little bit offtopic IMHO, anyway: I found surprising too that
> the github portgroup was not documented, given the high percentage of
> software that I build out of github repositories, and that's why I
> recently contributed the documentation of the github portgroup:
> 
> https://github.com/macports/macports-guide/pull/12
> 
> It has been merged some days ago. Yet, I don't see it online yet.

That is because we still have not any automatic deployment on
build.macports.org for any of our websites. I am irregularly running
these jobs from my local buildbot setup on my personal MacBook every few
weeks, which is the only reason they are not completely outdated.

I triggered another run of these jobs, so these changes should be
available shortly online.

Rainer

Re: request for port create command, to build a portfile from a URL

[G Alexander]

I play in a different playground and primarily use either master, dev or 
develop branches as I run with latest codebase available.  ffmpeg, imagemagick, 
opencv, clang6, openblas, etc.  These get a lot of PRs everyday. 

Having to update the Portfiles manually with latest hash and date/version is 
not optimal.  

Why can't a script find the repo, verify hash and date automatically then 
archive or if you have a GitHub account verify your SSH, and just grab the repo 
without creating an archive.

> On Mar 7, 2018, at 02:32, Enrico Maria Crisostomo 
>  wrote:
> 
> This is a little bit offtopic IMHO, anyway: I found surprising too that the 
> github portgroup was not documented, given the high percentage of software 
> that I build out of github repositories, and that's why I recently 
> contributed the documentation of the github portgroup:
> 
> https://github.com/macports/macports-guide/pull/12
> 
> It has been merged some days ago. Yet, I don't see it online yet.
> 
>> On 7 Mar 2018, at 11:29, db  wrote:
>> 
>>> On 7 Mar 2018, at 01:53, Rainer Müller  wrote:
 On 2018-03-06 23:00, db wrote:
 [...] an *overview* of how to write a portfile is much needed.
>>> Isn't this what this chapter in the guide is supposed to provide?
>>> https://guide.macports.org/#development
>> 
>> Yes, supposed. When you're in it's difficult to say, but AFAIR I was 
>> probably trying to write a portfile for something hosted on GitHub without 
>> knowing about the relative portgroup and its documentation being buried 
>> somewhere under the prefix in the tcl file itself.
> 

Re: request for port create command, to build a portfile from a URL

[Enrico Maria Crisostomo]

This is a little bit offtopic IMHO, anyway: I found surprising too that the 
github portgroup was not documented, given the high percentage of software that 
I build out of github repositories, and that's why I recently contributed the 
documentation of the github portgroup:

https://github.com/macports/macports-guide/pull/12 


It has been merged some days ago. Yet, I don't see it online yet.

> On 7 Mar 2018, at 11:29, db  wrote:
> 
> On 7 Mar 2018, at 01:53, Rainer Müller  wrote:
>> On 2018-03-06 23:00, db wrote:
>>> [...] an *overview* of how to write a portfile is much needed.
>> Isn't this what this chapter in the guide is supposed to provide?
>> https://guide.macports.org/#development
> 
> Yes, supposed. When you're in it's difficult to say, but AFAIR I was probably 
> trying to write a portfile for something hosted on GitHub without knowing 
> about the relative portgroup and its documentation being buried somewhere 
> under the prefix in the tcl file itself.

Re: request for port create command, to build a portfile from a URL

[db]

On 7 Mar 2018, at 01:53, Rainer Müller  wrote:
> On 2018-03-06 23:00, db wrote:
>> [...] an *overview* of how to write a portfile is much needed.
> Isn't this what this chapter in the guide is supposed to provide?
> https://guide.macports.org/#development

Yes, supposed. When you're in it's difficult to say, but AFAIR I was probably 
trying to write a portfile for something hosted on GitHub without knowing about 
the relative portgroup and its documentation being buried somewhere under the 
prefix in the tcl file itself.

Re: request for port create command, to build a portfile from a URL

[Rainer Müller]

On 2018-03-06 23:00, db wrote:
> [...] an *overview* of how to write a portfile is much needed.

Isn't this what this chapter in the guide is supposed to provide?

https://guide.macports.org/#development

Rainer

Re: request for port create command, to build a portfile from a URL

[Ken Cunningham]

On 2018-03-06, at 7:19 AM, Joshua Root wrote:

> 
> There's a fairly basic tool called portfile-gen in contrib. Improvements
> are certainly welcome.

That is indeed a step towards what I was thinking of.

I'll look at this. 

Ken

Re: request for port create command, to build a portfile from a URL

[db]

On 6 Mar 2018, at 16:19, Joshua Root  wrote:
> On 2018-3-7 01:58 , Ken Cunningham wrote:
>> 
>> port create URL
>> Might make things faster and easier for people to get started and up and 
>> running.
> There's a fairly basic tool called portfile-gen in contrib.

I took a bit to find it, so here's the link 
https://github.com/macports/macports-contrib/tree/master/portfile-gen.

I learned to write a portfile following some article in the wild, then I went 
through the whole guide, landed eventually in the mailing list.

While portfile-gen might help (homebrew has a similar feature), an *overview* 
of how to write a portfile is much needed.

Re: request for port create command, to build a portfile from a URL

[Ken Cunningham]

On 2018-03-06, at 7:22 AM, G Alexander wrote:

> I like the auto calculate hashes
> 
> For grabbing something from url sounds like having a lot of switches, 
> 
> for archive type and depedencies
> 

I was kinda hoping that the archive type might be figured out from the tail of 
the URL. All the use_bzip2 yes business if plenty confusing for people. Even 
the whole idea of master_sites and distnames takes getting used to.

dependencies would have to be manually specified. A basic template could be 
laid down for them, however, in the generated Portfile.


> what if link is a redirect to cdn, or something like Dropbox or symbolic
> 

some things would have to be manually done, unless AI can sort this out in ways 
I don't know how to do...

K

Re: request for port create command, to build a portfile from a URL

[Mojca Miklavec]

Hi,

On 6 March 2018 at 15:58, Ken Cunningham wrote:
> It would be nice to have a command like
>
> port create URL
>
> that would download the URL, calc the checksums for it, and build a basic 
> Portfile for it.
>
> It could ask for a category to use for the Port.
>
> In a more advance version, it could extract the URL and do some quick 
> analysis on it.
>
> if there's a CMakeLists.txt file in the root, do the right things to make it 
> a cmake based port.
>
> If there's a configure file in the root, do the right things for that.
>
> Might make things faster and easier for people to get started and up and 
> running.
>
> Certainly would make it easier for people who want to use MacPorts 
> infrastructure to build their own files, as many don't know how to properly 
> set up the include and lib directories, etc.

I would make a GSOC project proposal for this :)

This would certainly be nice functionality, it's just that someone
needs to write it.

Mojca

Re: request for port create command, to build a portfile from a URL

[G Alexander]

I like the auto calculate hashes

For grabbing something from url sounds like having a lot of switches, 

for archive type and depedencies

what if link is a redirect to cdn, or something like Dropbox or symbolic

> On Mar 6, 2018, at 06:58, Ken Cunningham  
> wrote:
> 
> Best,

Re: request for port create command, to build a portfile from a URL

[Joshua Root]

On 2018-3-7 01:58 , Ken Cunningham wrote:
> It would be nice to have a command like
> 
> port create URL
> 
> that would download the URL, calc the checksums for it, and build a basic 
> Portfile for it.
> 
> It could ask for a category to use for the Port.
> 
> In a more advance version, it could extract the URL and do some quick 
> analysis on it.
> 
> if there's a CMakeLists.txt file in the root, do the right things to make it 
> a cmake based port.
> 
> If there's a configure file in the root, do the right things for that.
> 
> Might make things faster and easier for people to get started and up and 
> running.
> 
> Certainly would make it easier for people who want to use MacPorts 
> infrastructure to build their own files, as many don't know how to properly 
> set up the include and lib directories, etc.

There's a fairly basic tool called portfile-gen in contrib. Improvements
are certainly welcome.

I'd advise not just blindly setting whatever checksums the downloaded
file happens to have though; the maintainer should be getting those from
a secure source (e.g. https web site) and verifying that they match, or
verifying the distfile against a gpg signature. If that can be automated
somehow then great, but let's not encourage skipping it.

- Josh

Re: request for port create command, to build a portfile from a URL

[Michael Dickens]

Hi Ken - I think that's a great idea. In "GNU Radio" land, we have such a tool 
for creating out-of-tree GR modules, called "gr_modtool"; it's great for 
setting up the skeleton for such OOT GR scripts. The end-user still has to fill 
in many "blanks", but this tool provides the starting point. I could see 
something similar for MacPorts, much as you describe. Maybe this could be added 
to the GSoC wiki page? It shouldn't be too difficult to implement something 
like what you propose, whether directly in 'port' or as a tool separate from 
'port'. Cheers! - MLD

On Tue, Mar 6, 2018, at 9:58 AM, Ken Cunningham wrote:
> It would be nice to have a command like
> 
> port create URL
> 
> that would download the URL, calc the checksums for it, and build a 
> basic Portfile for it.
> 
> It could ask for a category to use for the Port.
> 
> In a more advance version, it could extract the URL and do some quick 
> analysis on it.
> 
> if there's a CMakeLists.txt file in the root, do the right things to 
> make it a cmake based port.
> 
> If there's a configure file in the root, do the right things for that.
> 
> Might make things faster and easier for people to get started and up and 
> running.
> 
> Certainly would make it easier for people who want to use MacPorts 
> infrastructure to build their own files, as many don't know how to 
> properly set up the include and lib directories, etc.