People have responded in thoughtful ways but it seems to me that I may
not have been clear in explaining my intent. I will respond in-line
below to this particular post because it included so much material.
I want to thank everyone who has replied to my request. The comments
have helped me visualize some simple scripts that I can write to start
testing.
I think that I can safely conclude that the sort of script I envision
does not exist yet.
Cheers!
Steve Herber [EMAIL PROTECTED] work: 206-221-7262
Security Engineer, UW Medicine, IT Services home: 425-454-2399
On Fri, 28 Apr 2006, Duncan wrote:
[GRR!! Insane quote ordering for proper contextual replies rearranged to
conventional.]
Dice R. Random posted
<[EMAIL PROTECTED]>, excerpted
below, on Fri, 28 Apr 2006 02:43:44 -0700:
On 4/28/06, Steve Herber <[EMAIL PROTECTED]> wrote:
Portage is great but... it depends upon a person to create a new ebuild
when a new version is released.
There are two times when I would like a script to create an ebuild
automatically. I wonder if anyone has such a script or if someone is
working on such a script.
Often a new software release is available from the home site but
portage has not been updated. I would like a script that would:
check the source repository for a package
find the latest version
if the repository has a newer version
create an overlay with an ebuild for the new version
try to build the new package
This would be relatively easy to automate, due to the standard bump
procedure below. It wouldn't always work, and the risk for even trying it
would be entirely the user's, because in some cases such a procedure WOULD
break the system, due to changes between versions that weren't
accounted for by the automated process, but it would be relatively easy to
write a script that would work for most minor bumps anyway, a reasonable
percentage of the time, risky tho it would indeed be to just blindly run
it.
I assume that installing a package based on an automatically
created ebuild may break a system. But how else can you
determine that a new software package release is good on not?
You need to try it. I just want a script that lowers the bar
for the person that is willing to try.
Other times I will hear about a new software package and I want to try
it out. I would like a script that would take a URL for the home site
as an argument and then would operate like the script above.
This simply couldn't be automated in a sane manner. There's too much
Sane? A crude script that can create an ebuild with just
the minimum fields may not work very often, but if it works
for the package YOU want, then you have just saved YOUR time.
If it partially works, then you are that much closer to testing
the package. That extra time could then be used to improve the
ebuild and maybe get it working. That seems pretty sane to me.
variability in the way various things build, and too much risk in simply
assuming the normal otherwise automatable procedure will work, and will
not do something strange enough that it could be harmful to a working
system. An initial build for an application isn't a simple matter. One
I disagree. Many applications need very simple ebuilds.
The feature I love about portage is that if you do end up
putting files in less than optimum places, you can just remove
the package and update the ebuild to use better locations.
I wish /usr/portage were split into two parts. One for the core
packages that you need when Gentoo is used as your operating
system and another set for applications. I expect this issue
has been discussed by the portage on Mac OS X people but I have
not looked at their solution yet. The reason I mention this is
that I would generally be interested in automatically creating
ebuilds for the application set and would tend to leave the core
OS side alone to avoid the sort of catastrophic events that you
point out could happen.
must decide where things should go and supply the appropriate parameters
to have it install there. One must check that the licensing is suitable
for Gentoo, and further, whether Gentoo can mirror the sources or whether
I am not suggesting Gentoo ever use an automated systems
to put new software into the official portage tree. I would
always want a person to be involved for the many reasons you
mention here. That does not stop us from having a system that
jump starts the ebuild creation process.
they have to be fetched directly from upstream due to licencing, possibly
manually, if the licensing is strict enough. One must verify the
setup/unpack/configure/compile/install/qmerge steps and create
The script I am looking for will in fact have to create the
overlay area and run through all the steps you mention. The
nice thing is that emerge will do each step for you and it keeps
multiple logs, so we can see if the build works or not. It is
nice to not have to rewrite emerge.
non-defaults for anything that doesn't work normally. One must ensure
that the sequence doesn't violate the non-interactive rules of portage.
One must ensure the merged application is sufficiently configured to "just
work" out of the box, with no additional configuration, if at all possible
The script I want may well create an ebuild that will fully
succeed with emerge but then NOT just work. Ebuilds in the
official Portage ebuilds need to "just work", but experimental
ebuilds don't need to fully work.
(obviously it's not going to be possible with things like cups, where the
user must configure the printers they use as appropriate). One must make
some attempt to verify sane compile settings and that the package will use
those supplied by portage over its default settings. One must further
verify that "normal" CFLAGS/CXXFLAGS actually work. One must evaluate
dependencies both optional and required, setting the correct build and
runtime dependencies, and setting up USE flags as appropriate for the
optional ones. One must... One must... One must... Finally, appropriate
patches must be devised if the package doesn't by default pass any of
these steps, such that it does.
Really, the devs do more for you by supplying Gentoo than picking their
nose all day. Don't you think if it were that easy (or even /possible/)
to automate creating an ebuild for a any random product out there, that
they would have such a system by now? I'm not a dev, but if I was, I'd be
insulted by just how lazy or stupid you obviously think they are, lazy if
you think they have such an approach and are simply hiding it as some big
secret, so they can do the easy stuff and make it look hard, stupid if you
think it's possible to automate the thing, and they haven't done it.
This is a bit harsh. The devs do a great job.
I don't know why there is not a an automated system to create
ebuilds for random software, that is why I asked the list.
Maybe the developers are busy improving the core system. That
is what it looks like to me.
But it is a closed system in the sense that there is not a
public web site where a person like you or I can go, point to an
interesting project, and have a first cut at an overlay created
within the limits of the script. The web site could even try to
compile and install in a virtual machine and give you a report
before you ever tried it on your system. I would also want the
web to have a way for people to register their success and
failure and have a place where you could upload a fixed ebuild.
Allowing non portage developers a chance to try their hand on a
new ebuild and get feedback on their efforts seems to be taking
advantage of the Open Source methods.
Really, if there's no ebuild available at all, the user is best advised to
read the documentation to learn of any gotchas, then follow thru on the
steps manually, such that he can watch what's going on and see where any
breakage occurs, and attempt to fix it at that point.
Or use a script that generates a skeleton that you can start
with.
If such scripts were available, it would be much easier to check out
new software releases. I could imagine a system that would scan
listings from freshmeat or other non-Gentoo site, build the ebuild and
try the package. In the simplest case you would get a go/no-go answer.
For the packages that download and build, you could try it out. If it
failed you could ignore it or try to figure out the problem.
If it failed, you could indeed try to figure out the problem --
potentially try to figure out why nothing you try to run works any more,
and your system won't boot, because the automated script created an ebuild
that blindly deleted /usr/bin/*, or something equally dangerous to a
system, because someone thought it was a bright idea to trust the build to
an automated script, sight entirely unseen, README and INSTALL entirely
unread!
This sounds like the sort of problem you would get if you tried
to update one of the core OS packages. I am thinking more along
the lines of Asterisk. The portage tree is generally out of
date. I think there is an alternate overlay site that is much
more current but just bumping the version has a good chance of
working.
Or, I just saw a new package, http://www.getdemocracy.com/, and
I was interested in trying it. It is a video player. I don't
expect it to totally trash my system. But if it did and I had a
web site where I could report on it, someone else might know how
to fix the ebuild.
There's simply no sane nor safe way to automate that stuff. For something
with an existing ebuild that might just need bumped, sure. For something
without an existing ebuild, I must conclude you really hadn't thought the
implications of what you were asking for thru AT ALL!!
This just seems harsh.
We can decode the human genome with a Perl script but we can't
figure out how to use a few heuristics and a sandbox to get
close to a working ebuild. I think we can and I am looking for
existing scripts and ideas from people that will help us get to
the point that most open source packages have an ebuild.
I have thought about this a little bit.
I don't know about an automated process, there are a lot of variables
that can factor into a version bump. However, for many simple
packages, i.e. things that are not heavily patched or that do not have
changing software requirements, it is usually sufficient to simply
copy the old ebuild over to the new version name.
Example, suppose net-www/fooapp has recently been bumped from version
1.001 to 1.002. You could do the following:
mkdir -p /usr/local/portage/net-www
cp -r /usr/portage/net-www/fooapp /usr/local/portage/net-www
cd /usr/local/portage/net-www/fooapp
cp fooapp-1.001.ebuild fooapp-1.002.ebuild
ebuild fooapp-1.002.ebuild digest
emerge -auv fooapp
Thanks for this suggestion. I will create a shell script to
automate this part of the process and test it out. Thanks.
The reason for copying the entire /usr/portage/net-www/fooapp
directory tree over is that there may be some patches or configuration
files or such under the files/ subdirectory which are required by the
ebuild. As I said, this procedure works a good 8 or 9 times out of 10
with simple ebuilds, sometimes some slight modifications are necessary
and other times the modifications may be quite extensive.
A solid idea, the steps of which are done by many, but a couple
modifications make it a bit better.
Instead of copying over the entire dir, create the dir in the overlay (if
you don't already have it from a previous version), and copy over the
ebuild you intend to bump. You don't have to worry about previous
versions, or the changelog, metadata, or manifest files, since you the
former two are simply informational and the latter will be recreated.
Instead of copying over the files dir, make a symlink to the existing one
in the regular portage tree. (If you have any of your own patches or
modify any existing ones, create a different dir for those, and edit the
ebuild accordingly.)
Since the files dir is a symlink, it will automatically change as the
portage tree changes. Thus, if patches are changed or added in the Gentoo
tree, the copy in the overlay will stay current, and any remerge or the
next bump will use the new versions. In some cases that's critical, as it
could mean you get security patches added since the static creation of the
overlay.
Doing it this way, only copying the specific ebuild you'll be working on,
bumping or whatever, and using a symlink for the files dir, avoids the
risk of overwriting any work you've done on the same package previously,
that you may wish to save. It's also more space efficient.
Another good suggestion. Thanks.
...
In any case, with or without my changes, that the steps can be laid out as
a series of coommands like that indicates they can be made into a script.
It's an exercise left to the reader, then, to create a script that takes
the CPV (category/package-version) of the existing ebuild, and the version
of the the target ebuild, as command line parameters, and executes the
sequence of coommands necessary to end up with the bumped ebuild ready to
go in the overlay. Depending on how fancy you want to make the script,
with or without a --help output, with or without validity checking and
handling any possible errors, with or without the overlay location
hard-coded into the script vs. configurable using a config file or a
command line option, and depending on how good you are with bash scripting
or scripting in your language of choice (perl, python, whatever), reading
this thread could have taken you longer than creating the script will.
BTW, if you do discover a package that has a new version and you get a
good working ebuild for the new version make sure that you post it to
bugzilla so that a developer can get it into the tree.
Yep!
Always a good recommendation! =8^)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html
gentoo-amd64@gentoo.org mailing list
--
gentoo-amd64@gentoo.org mailing list
