Re: [aur-dev] Ghost repo?

[Xyne]

Doug Newgard wrote:

>This is normal. When packages are deleted, the repo remains; the author's name
>and email are from their git config, not from the AUR.

I didn't think to check if there was a package in the official repos. I assume
that is preventing the recreation of the package. If that were not the case,
could the package be recreated via a push? Or is something else required to
recreate a deleted package?

[aur-dev] Ghost repo?

[Xyne]

Hi,

A user wanted to create an AUR package for libpng12 but found that there was
already a repo when he tried to push it, even though the package does not
appear on the site [1]. The log of the existing git repo contains the author's
name and email address but there is no account with either. The last update was
May 8, 2016. Is this a bug, perhaps due to a deleted account?

I was going to try pushing an update to see if the package re-appears on the
site but it's probably better if one of the devs takes a peak to see what's
going on.

The package is libpng12.

Regards,
Xyne


[1] https://bbs.archlinux.org/viewtopic.php?id=221723

Re: [aur-dev] Making the AUR package list more useful

[Xyne]

Lukas Fleischer wrote:

>I am not in favor of adding regular expressions on the server side. I
>mentioned some of the reasons in an earlier reply and you agreed.
>However, instead of saying that we need something less powerful (like
>prefix and suffix matching) on the server side, I think that adding
>support for regular expressions (and maybe even more powerful things) on
>the client side is the way to go.
>
>Lukas

Hi ho,

I'm nearing the end of a big rewrite of powerpill/bauerbill that provides a
completely generic interface for extending existing binary repos with build
support as well as adding build-only repos such as the AUR. While working on
interfaces to generalize Pacman's various sync functionality (-Ss, -Si, -Sg,
-Sp, etc.) I have had to provided reduced information for operations that query
the full list of packages in the AUR due to limitations of the pkglist (e.g. no
versions in -Sl). I have also had to cobble together lazy lookups to get all
necessary build information given that the current AUR RPC interface does not
provide all fields in the SRCINFO (when I last checked).

Having access to a full offline database of SRCINFO would be *very* useful,
especially if it can be downloaded incrementally (e.g. with zsync as
suggested). Providing the SRCINFO directly without any additional formatting
provides a de-facto (official?) standard that is reliable and directly tied to
pacman/makepkg. AUR-specific information could be added to a second file
(either in JSON format or one that follows SRCINFO). Besides the AUR ID/URL,
votes, etc, you could also include the hash of the commit that was used to
generate the database entry. This would prevent errors due to the package on
the server leading the package in the downloaded database.

How big would such a database be? How much overhead is involved? How much
overhead would the zsync operation introduce when people begin to regularly
query updates?

Regards,
Xyne

Re: [aur-dev] rpc versions?

[Xyne]

On 2016-01-30 12:35 +0100
Lukas Fleischer wrote:

>Actually, only one more info query is required. I even wonder whether a
>search query should only return the package names by default? Making the
>amount of returned details configurable might also be a good idea...

Making it configurable would be best. "search" and "msearch" may often only
require names and description but sometimes it is useful to return the full
package information to avoid further queries.

A "tsearch" for searching by tags may also be useful. Btw, that information
should be added to the info results, which used to include the category ID.

Regards,
Xyne

Re: [aur-dev] rpc versions?

[Xyne]

On 2016-01-30 12:08 +0100
Lukas Fleischer wrote:

>The current version is documented at [1]. For all changes, you can check
>the output of `git log web/lib/aurjson.class.php` in the aurweb Git
>repository.
>
>[1] https://projects.archlinux.org/aurweb.git/plain/doc/rpc.txt

Thanks for the link. It would be very helpful if it were added to the page
along with an indication of what the current version is. Incidentally, that
document uses v=5 but that returns an invalid version error. Perhaps a link to
the current release would be best.

I have also just noticed that the results returned for type=search do not
include all of the information returned by an "info" query, regardless of the
version number. Can that be changed or parameterized? Without that information
additional info queries are necessary.

[aur-dev] rpc versions?

[Xyne]

Hi,

The RPC interface [1] returns different results depending on the requested
version via the "v" GET parameter but it isn't even mentioned on the page and
it still defaults to v=1 which lacks dependency information.

Would you please add information about that parameter along with some
information about the currently supported versions and their differences (or at
least a link to relevant information such as changelogs)?

Thanks!


[1] https://aur.archlinux.org/rpc.php

Re: [aur-dev] list of AUR packages

[Xyne]

On 2014-12-25 01:42 +
Xyne wrote:

>>I guess you want this: https://aur.archlinux.org/packages.gz
>
>Perfect. Thanks, Santa!

If anyone is interesting, python3-aur [1] now includes a module and script for
accessing the list.

[1] http://xyne.archlinux.ca/projects/python3-aur/

Re: [aur-dev] list of AUR packages

[Xyne]

>I guess you want this: https://aur.archlinux.org/packages.gz

Perfect. Thanks, Santa!

[aur-dev] list of AUR packages

[Xyne]

Hi,

What is the current position on adding a query type to the JSON interface to
return a full list of AUR packages? If creating a JSON list of that size is
prohibitive, what about providing a simple, newline-delineated text file that
is refreshed via a cronjob?

I ask because I just discovered aurlist [1] and my immediate thought was that a
simple list would save considerable bandwidth and processing vs scraping the
search results.

If this already exists, please point me to it while mocking me gently.

Happy holidays btw ;) [2]

Regards,
Xyne

[1] https://aur.archlinux.org/packages/aurlist/
[2] http://xyne.archlinux.ca/img/xmas/arch-holly_jolly.svg

Re: [aur-dev] AUR 3.5.0 released

[Xyne]

Dave Reisner wrote:

>Hi,
>
>In addition to this, I'll be releasing a new version of mkaurball today
>which ceases generation of AURINFO and creates SRCINFO instead. If
>you're using a recent enough pacman-git[0], mkaurball will just forward
>on makepkg, offering very little of its own logic.
>
>Consider this is the beginning of my efforts to deprecate mkaurball. Of
>of the pacman 4.2.0, mkaurball will be obsolete and will likely be
>replaced with a script which just yells at you to use makepkg instead.
>
>Cheers,
>dR

Will that include an option to anonymize the local user data in the generated
archive? I currently have my own wrapper named "mkanonaurball" to replace the
user with "nobody" etc.

Sorry if I've missed a relevant discussion about this. If so, please point me
to the thread.

Regards,
Xyne

Re: [aur-dev] pkgbase queries via RPC interface

[Xyne]

On 2014-08-16 16:52 +0200
Lukas Fleischer wrote:

>I'd rather not overcomplicate things. Having a "by" parameter, the
>possibility to pass one or multiple (fixed) strings and an option to
>enable exact matching is what I was thinking of. I do not think that
>combining search types gives a substantial benefit.
>
>If we really need to support very powerful queries, it might be better
>to reconsider another idea I had earlier: Replace the RPC interface with
>a static database. Basically, the result of an RPC query that matches
>every single package is computed every hour (or so) and stored in a flat
>file which can be downloaded, similar to pacman databases. AUR helpers
>can download that file and do whatever they want. Note that this file
>will probably be quite large, though (roughly 5-10MiB when compressed,
>did not check with the latest set of packages). I am not sure whether
>this is the best thing to do, since, unlike in the case of the official
>repositories, users are usually only interested in a tiny amount of AUR
>packages.

The advantage of real-time results is that they can be used to confirm uploads
and other operations. Making a compressed database available may be useful in
its own right but I would not want to see it replace the current system.

Given that the current search already searches by both name and description,
would a single "by" parameter be able to at least accept a character-delimited
list to search multiple fields (e.g. pkgname:pkgbase:pkgdesc)? If not, what
about accepting multiple "by"s for a combined OR search?

Re: [aur-dev] pkgbase queries via RPC interface

[Xyne]

On 2014-08-14 23:05 +0200
Lukas Fleischer wrote:

>> B) Add a parameter to query pkgbase data (e.g. object=pkgbase) and return a
>>JSON object with pkgbase-specific data (everything that still applies from
>>regular package data, plus an array of the included packages). Given that
>>there are AUR pages for pkgbases, it would be consistent to provide that
>>data via the RPC interface.
>> 
>
>This sounds better to me.
>
>> I believe this would be very useful. What do you think? Would it be
>> difficult?
>> 
>
>I like this idea. Should not be too hard to implement. Maybe we should
>make the whole thing more flexible by replacing everything with a single
>search method (instead of search, info and multiinfo) with different
>search modes (packages and package bases), a filter to specify the
>fields on is interested in and different query types (by name, by name
>and description).

That works for me. In that case, the method should accept a list of fields to
search (pkgname, pkgbase, pkgdesc, maintainer, deps, url?, etc.). There should
be a way to search for exact matches (e.g. to retrieve data about a specific
package or package base). That could be done either with regexes (too much
server overhead?) or an extra parameter that forces a perfect match.

The returned objects should include a "type" field to specify what kind of
object they are ("package", "package base").

A filter to reduce the returned fields may be useful in some cases but it's
easy enough to filter on the client-side. I suppose it's a matter of cpu vs
bandwidth for the server.

Search for packages "foo" and "bar" (multiple arguments -> multiple returned
objects):

https://aur.archlinux.org/search.php?by=pkgname&exact=true&arg=foo&arg=bar

Search for package base "baz":

https://aur.archlinux.org/search.php?by=pkgbase&exact=true&arg=baz

Search for all packages depending on "foo" (not exact to allow for versioned
deps):

https://aur.archlinux.org/search.php?by=depends&arg=foo

You could even split the query string by the "by" parameters to enable
multifield searches, e.g. to search for all python packages maintained by "foo":

https://aur.archlinux.org/search.php?by=pkgname&arg=python-&by=maintainer&exact=true&arg=foo

There may be multiple arguments to each "by" (e.g.
by=pkgname&arg=python-&arg=python3-) so this may tricky to do with a single
backend pass, but it would be easy with subsequent refining passes for each
"by").


I'm not sure what the best way to build in boolean logic would be ("and", "or",
"xor"?, etc.) or if it is even something that you would want to implement.
Maybe with a custom "advanced" parameter that accepts a string that the server
can parse directly (using some existing syntax?).


I'm just kicking around some ideas for the sake of discussion.

Re: [aur-dev] pkgbase queries via RPC interface

[Xyne]

On 2014-08-14 08:45 -0400
Dave Reisner wrote:

>> Incidentally, shouldn't the optdepends be returned as an object?
>> E.g.
>>
>> {"foolib":"support for foo", "bar":"pub functionality"}
>
>Why? makepkg requires that optdeps are well formed. Splitting them in the
>response here seems like an arbitrary decision. Why not also then split
>depends, in case they're versioned? What about the pkgver and pkgrel (which
>are combined)?

I see the optdeps as a set of key-value pairs that lends itself naturally to a
JSON object.

I would actually be in favor of splitting deps and the combined pkgver just for
the sake of returning a full digest of the information instead of a partial one
that leaves the rest to the user, but it's admittedly trivial and probably a
waste of CPU cycles on the server so I'm not actually suggesting it.

Thinking about it, I also realize that it wouldn't work for ranged deps, e.g.
(foo>=3.0, foo<4.0) without relatively complex objects.

Re: [aur-dev] pkgbase queries via RPC interface

[Xyne]

On 2014-08-13 16:12 -0400
Dave Reisner wrote:

>On Wed, Aug 13, 2014 at 12:28:10PM -0700, Colin Woodbury wrote:
>> (A)
>> 
>> Although, I thought `info` lookups had been converted to return Arrays as
>> well.
>
>This is only true of RPCv3:
>
>https://bugs.archlinux.org/task/40963

Incidentally, shouldn't the optdepends be returned as an object?
E.g.

{"foolib":"support for foo", "bar":"pub functionality"}

[aur-dev] pkgbase queries via RPC interface

[Xyne]

Hi,

The RPC search, info and multiinfo methods return no data when the pkgbase is
passed as the argument.

I would like to discuss the following ideas:

A) Return all of the packages in the pkgbase for search, info and multiinfo
   queries. For search and multiinfo this would be natural. For info it would be
   left to the client to detect that the result is an array instead of an
   object and thus infer that the result is from a pkgbase (or simply add that
   data to a new field in the result object). With the advent of the query
   version parameter this should not be an issue for backwards compatibility.

B) Add a parameter to query pkgbase data (e.g. object=pkgbase) and return a
   JSON object with pkgbase-specific data (everything that still applies from
   regular package data, plus an array of the included packages). Given that
   there are AUR pages for pkgbases, it would be consistent to provide that
   data via the RPC interface.

I believe this would be very useful. What do you think? Would it be difficult?

Regards,
Xyne

Re: [aur-dev] [PATCH] Make deleted comments visible to Trusted Users

[Xyne]

On 2014-07-28 11:10 +0200
Lukas Fleischer wrote:

>I am not sure about that. Users could delete their comment before anyone
>has a chance to report it and subscribers would still have the insult in
>their incoming mail. Then again, you could argue that the abuser could
>just as well have send a private email to everyone...

In that case the email can be forwarded to a TU or to aur-general.


>Let me describe another scenario that happened: A package maintainer did
>not want to update a completely broken AUR package and told the users
>that the package is fine and that he won't do anything. After some
>discussion, one of the upset users filed an orphan request. Thereupon,
>the package maintainer deleted all of his comments, claiming that he
>would never have said something like that. And indeed, it was very hard
>to tell whether the maintainer replied to the comments or not (without
>having a look at the deleted ones). Would a "Report Abuse" button help
>here?

A user could have reported a refusal to fix the broken package, yes. We could 
have a general "report" button, not just one for reporting verbal abuse against 
other users.





>Note that I will not change the comments deletion feature before the
>3.4.0 release but I am open for suggestions and discussion. Maybe we can
>reach a consensus that allows us to close FS#14840 [1] and FS#34690 [2].
>
>[1] https://bugs.archlinux.org/task/14840
>[2] https://bugs.archlinux.org/task/34690

If comments are delible then they should also be editable as both can disturb 
the flow on the page. Personally I'm not worried about the flow of comments on 
the page as it should be limited to packaging issues which in turn should be 
fixed as soon as there is a solution.

I really think that comments should be both editable and delible. The AUR is 
not a collection of holy stone tablets to be preserved for all posterity nor 
should it chronicle the full history of every package. If that was its purpose 
then we would be crying out for more advanced commenting features (e.g. 
codeblocks, quotes). If some important data goes missing someone else will 
repost it.

A report button would work well with editable comments (just store a copy of 
the actionable edit). Keeping previous versions alongside the current version 
would probably be more complicated. In either case, at some point we should 
expunge the old cruft. Persistent trolls will keep making trollish comments and 
reformed trolls can be ignored.

Re: [aur-dev] [PATCH] Make deleted comments visible to Trusted Users

[Xyne]

On 2014-07-27 11:29 +1200
David Phillips wrote:

>>So maybe we should clean them up occasionally?
>
>Makes sense. A TU's hardly going to receive complaints about abuse 12
>-- or even 6 -- months after the incident.

Even 3 months would be enough to report abuse.

Alternatively, add a "report abuse" button that stores a copy of the comment. It
can then be reviewed even if the original comment is deleted. The copy can be
deleted when the report is closed (or after some interval). This would also be
less tedious than scanning through all the duplicate comments posted due to the
absence of an edit function.

Re: [aur-dev] [PATCH] Make deleted comments visible to Trusted Users

[Xyne]

On 2014-07-25 19:49 +0200
Lukas Fleischer wrote:

>On Fri, 25 Jul 2014 at 18:42:22, Xyne wrote:
>> On 2014-07-25 11:31 +0200
>> Lukas Fleischer wrote:
>> 
>> >This allows Trusted Users to check whether a user posted a politically
>> >incorrect comment, even if he already deleted it.
>> 
>> 
>> Why keep "deleted" comments? How long are they kept?
>> 
>
>Yes. Forever.

Is there some technical reason for not deleting comments? If not, please
(re)consider actually deleting them. Keeping them is simultaneously a waste of
bits and a bit creepy.

Re: [aur-dev] [PATCH] Make deleted comments visible to Trusted Users

[Xyne]

On 2014-07-25 11:31 +0200
Lukas Fleischer wrote:

>This allows Trusted Users to check whether a user posted a politically
>incorrect comment, even if he already deleted it.


Why keep "deleted" comments? How long are they kept?

Re: [aur-dev] [PATCH] aurjson: Add a PackageBaseID field

[Xyne]

On 2014-05-31 12:04 +0200
Lukas Fleischer wrote:

>This is useful to tools that automatically generate requests to orphan,
>delete or merge packages.

Perfect, thanks! When do you plan to push the changes?

[aur-dev] mismatched AUR IDs

[Xyne]

Hi,

I'm in the process of updating python3-aur for the 3.0.0 release and I have
come across a bug. The IDs of recently updated packages reported by the RPC
interface are incorrect. For example, etmtk is currently listed in the Recent
Updates feed on the front page. The RPC interface reports the ID as 83532 but
the true ID (e.g. for the deletion form) is 80587.

Packages that have not been updated since the release of 3.0.0 do not appear to
encounter this bug.

Regards,
Xyne

Re: [aur-dev] Delete old fish shell packages

[Xyne]

On 2014-03-28 21:21 -0600
Tyler Jameson Little wrote:

>The following packages are not maintained, have been disowned and have no
>upvotes. They both have the same owner and were created within minutes of
>each other. The owner has expressed interest in the comments (first link)
>for them to be deleted.
>
>- https://aur.archlinux.org/packages/fishfish/
>- https://aur.archlinux.org/packages/fishfish-git/
>
>I'm not the owner, but it seems the owner has lost interest.
>
>Could a TU please delete these? Thanks!


Please send deletion requests to the aur-general mailing list. This list
(aur-dev) is for technical discussion of the AUR itself.

Thanks

Re: [aur-dev] [PATCH] Add support for anonymous comments

[Xyne]

Lukas Fleischer wrote:

>This allows for removing users without also removing the corresponding
>comments. Instead, all comments from deleted users will be displayed as
>"Anonymous comment".

Can you also insert the original poster's name into the comment?
E.g. "/originally posted by Foo"

In that case I would call it a "ghost comment" instead of an "anonymous 
comment".

Re: [aur-dev] [PATCH] Add an IP ban list

[Xyne]

Lukas Fleischer wrote:

>> Do the IPs need to be visible? In the case of a single IP a simple ban button
>> will suffice. A proxied IP will be completely different every time so
>> subsequent addresses are unrelated. That only leaves netmasked dynamic IPs. 
>> It
>> would be enough to have an interface button connected to a query that returns
>> all users with an IP in the netmasked range (/24?). You could even
>> automatically flag user accounts that share a range with banned IPs, again
>> without divulging the IP address.
>
>This is not the whole truth. To stop the latest spam attack, we had a
>look at the web server logs, noticed that the spammer was using Tor,
>generated a list of Tor exit nodes and added that to the IP ban list.
>How would you do that without seeing any IP addresses? How would you
>figure out if a spammer is just controlling 4-5 small subnets or using
>proxies at all?

Fair enough.

Incidentally, can a banned IP address still be used to browse the site and
download packages? There are many people who use Tor and other proxies for
various reasons and it would be a shame if they have to suffer due to one
basement-dwelling troll. Essentially only the login and post forms would need
to respect the ban.

Sorry if this has been addressed already. I haven't read through the patches.



>If you feel strongly about not showing IP addresses, we could hide IP
>addresses for TUs and only show them to the AUR administrator(s) who can
>skim through the logs anyway.

Please do. Thanks.

>Yes, they can. I did not mean to allege anything here -- I just wanted
>to make sure that banning a range of IP addresses doesn't
>(unintentionally) block any Trusted Users or developers.

That would make for a great post in the stupid computer mistakes thread... it
would be on the same level as ssh'ing into a box and killing the network.

Re: [aur-dev] [PATCH] Add an IP ban list

[Xyne]

On 2013-03-19 22:42 +0100
Lukas Fleischer wrote:

>On Tue, Mar 19, 2013 at 05:12:23PM -0400, canyonknight wrote:
>> On Tue, Mar 19, 2013 at 9:23 AM, Lukas Fleischer
>>  wrote:
>> > This allows for specifying a list of IP addresses that will no longer be
>> > able to register new accounts and login. The list of banned IP addresses
>> > can be configured in "web/lib/config.inc.php".
>> >
>> > Signed-off-by: Lukas Fleischer 
>> > ---
>> 
>> What are your thoughts on taking this a step further and adding a
>> "bans" table to the DB schema? It could eventually be extended to
>> allow for TUs and Developers to ban IP addresses directly from the web
>> interface without ever having to muck around with the config file.
>
>Exactly what I was planning to do.
>
>We should also display each user's last login IP address in his profile
>(only visible to developers and TUs) and add a "Ban this IP address"
>button next to it. The "Save last login IP address" patch I submitted
>already adds the IP address to the Users table.

Do the IPs need to be visible? In the case of a single IP a simple ban button
will suffice. A proxied IP will be completely different every time so
subsequent addresses are unrelated. That only leaves netmasked dynamic IPs. It
would be enough to have an interface button connected to a query that returns
all users with an IP in the netmasked range (/24?). You could even
automatically flag user accounts that share a range with banned IPs, again
without divulging the IP address.

>Oh, and we might want to exclude TUs and developers from IP bans.

I haven't tried it, but can't TUs disable each other's TU status? The exclusion
would be trivial if so.

Re: [aur-dev] AUR 2.0.0-rc2 released

[Xyne]

Lukas Fleischer wrote:

>2.0.0-rc2 has been released. The demo setup [1] is already up-to-date.
>
>[1] https://dev.aur.archlinux.org/

Is this still available somewhere? I would like to test changes in python3-aur
before the release.

Thanks.

[aur-dev] Python 3 AUR module

[Xyne]

Hi all,

I've written a Python 3 AUR module that some of you might find interesting or
useful: http://xyne.archlinux.ca/projects/python-aur/

Regards,
Xyne

Re: [aur-dev] [PATCH] Add smilies to comments

[Xyne]

Loui Chang wrote:

> On Sun 03 Oct 2010 12:30 +0200, Manuel Tortosa wrote:
> > The images you can get them from here:
> > wget www.chakra-project.org/temp/manutortosa/smilies.tar.gz
> > or use your own, those are licemsed under gpl2 and are exactly the same
> > you've in your forums.
> 
> I'm not really big on eyecandy to be honest.
> I prefer really simple clean interfaces.
> 
> I would like to hear more feedback about this.
> Maybe stick this in the AUR bug tracker to collect votes.
> 

I consider the AUR comments a place for serious discussion regarding packages. I
don't think smileys contribute much to that. It would be like adding sparkly
stickers to a report.

Regards,
Xyne

Re: [aur-dev] Hello all

[Xyne]

Manuel Tortosa wrote:

> All the paches will be prepared using the AUR source code and template 
> agnostic to fit the ARCH look even if this means double work for me.
> 
> Thanks a lot for your work. Keep Arch rolling!

I am not involved in AUR development but I just had to say that your attitude
is great. I love to see others make such efforts in order to contribute and
improve the situation for everyone. It is very much appreciated and it is one
of the great motivators of the entire open source movement.

Thank you!

Re: [aur-dev] Can I please get a dump of the pkgbuilds in AUR?

[Xyne]

On Tue, 03 Aug 2010 19:29:01 +0100
Nathan Wayde  wrote:

> On 03/08/10 19:14, ludovic coues wrote:
> > I don't understand what you will be able to do for pacman with AUR.
> > ABS isn't a best place to check PKGBUILD ?
> >
> 
> At the risk of sounding rude, I'm gonna come right out and say that none 
> of that made any sense.

I think he was suggesting that you collect the PKGBUILDs and local
source files from the ABS tree (which you can download via rsync using
the "abs" package) and use those in place of the AUR PKGBUILDs for
testing purposed.

Re: [aur-dev] Can I please get a dump of the pkgbuilds in AUR?

[Xyne]

> I know this has come up before with issues about scraping the files in 
> AUR but this is a different question. Is it possible to get a dump of 
> all the files in AUR, don't ask for what reason I need them[1].

Tangent: Would it be possible to either
a) add an rsync interface to the AUR
b) dump the AUR daily|weekly|monthly and add the archive to the mirrors
?

Such requests may be rare at the moment but if the files were more
readily accessible (programmatically) then it might lead to some
interesting uses. I can think of several myself.

Regards,
Xyne

Re: [aur-dev] {SPAM 01.6} Re: Safe and relatively reliable PKGBUILD parser.

[Xyne]

> Yes, there's not necessarily a problem here. The wrapper function you
> construct would be syntactically valid and would execute fine. (It would
> exit whatever shell it's executed in.)
> 
> I mention it to point out that you have to be careful regexping through
> the source (even the Bash-canonically-formatted source) of a function
> that Bash says is syntactically valid. There may be parts of that source
> that you assume to be valid Bash but which aren't, because they come
> after an exit. There may be other ways to do this too, e.g., with an
> exec.

The wrapper function is never executed so I don't see how this is an
issue. The only possible issue would be that the parser would miss the
exit and set variables that are set after it, but that's not really an
issue either. If someone creates such a PKGBUILD, the PKGBUILD itself
is invalid so it doesn't make any different if the parser assigns
variables before or after the exit. This is basically the same as
pointing out that if the PKGBUILD specifies an incorrect depends array
then the parser will also specify an incorrect depends array.

The point of the parser is to grab metadata from a PKGBUILD, not
magically determine if the PKGBUILD itself is correct in every possible
sense of the word. If the packager provides incorrect or nonsensical
data then there is little than can be done. It's not an exploit though,
because no malicious code is executed.


> E.g.:
> 
> TESTFILE=$(cat <<"EOF"
> echo normal stuff
> exit 0
> any funky stuff I want
> pkgver=#$#%$%%^&^...@#$$@^ } more funky stuff {
> EOF
> )
> WRAPPED=$'wrapper() {\n'"$TESTFILE"$'\n}'
> bash -n <<< "$WRAPPED"
> echo $? # prints 0
> bash <<< "$WRAPPED"$'\ntype wrapper'
> 
> prints:
> ===
> wrapper is a function
> wrapper ()
> {
> echo normal stuff;
> exit 0;
> any funky stuff I want;
> pkgver=#$#%$%%^ & ^...@#$$@^ } more funky stuff {
> }
> 
> The garbage comes out in the output of `set` too, as well as `type
> wrapper`.
> 
> Now if you try regexping that, without truly parsing it, you may if
> you're not careful find yourself processing the garbage at the end.
> That may make exploits possible.
>
> I like the approach you're using, but this loophole occurred to me and I
> thought you should know about it. You'll have to decide whether it's
> worth protecting against.

Can you an example of an exploit using this method? I really can't
think of any. If you are suggesting that the trailing "{" would throw
off the parser then you are wrong. The output of the set, as I've
stressed previously, is canonical. This permits the unequivocal parsing
of function blocks because they are opened and closed on their own
lines and thus code inside the block cannot both pass the validity
check and appear on its own line.



> But that makes salient the question of what advantage you're aiming to
> get by what you're doing rather than just checking the PKGBUILD with bash -n 
> and if
> that works, sourcing it with some appended echos to print out the
> variables we're interested in.
How can you source the packaging functions to get the variables nested
in the packaging functions without executing the rest of the code
inside those functions?
How can you source potentially malicious PKGBUILDs safely?


> One possible advantage would be to protect against malicious
> stuff. If that's your aim, then you should be protecting against exploits
> like this may make possible.

The only possible vulnerabilty of this method will be in the
implementation of a function whilelist (e.g. uname -r). So far I see no
other possible exploits which would allow someone to get the parser to
run malicious code.

Regards,
Xyne

Re: [aur-dev] Safe and relatively reliable PKGBUILD parser.

[Xyne]

> If you're willing to trust the variable declaration part of the
> PKGBUILD, then yeah it'd be easy to execute just that part. You don't
> even need to cut out the build() function, since executing the whole
> thing would only declare and not run that function. All you'd need to do
> is to add some "echo"s at the end of the wrapper function you've
> constructed, and execute the wrapper function.


That doesnt work for overridden variables in split packages because they
are set inside the packaging function(s). Anything which could
selectively execute blocks of code inside of functions to get the
values of the variables would be far more complicated than this
approach and probably far more exploitable.

Even without that to consider, you cannot blindly trust the variable
declaration section of PKGBUILDs uploaded to the AUR.

Re: [aur-dev] Safe and relatively reliable PKGBUILD parser.

[Xyne]

> Indeed. Perhaps Allan would be interested on this for his makepkg test  
> suite, although maybe more in the concept since the test suite us in  
> python.

It would be trivial to conver this to Python. I will probably do that
myself if there seems to be enough interest in it.

Re: [aur-dev] Safe and relatively reliable PKGBUILD parser.

[Xyne]

> I was brainstorming to think of possible exploits. It looks like this is
> valid syntax:
> 
> echo normal stuff
> exit 0
> any funky stuff I want
> pkgver=#$#%$%%^&^...@#$$@^ } more funky stuff {
> 
> Running bash -n on that gives 0. Now there's not necessarily anything
> wrong here---unless your parser doesn't stop parsing at the exit command.
> If it goes past that, then maybe exploits could be introduced, because
> we wouldn't be entitled to the assumption that the rest of the code is
> valid syntax.
> 
> -- 
> Jim Pryor

I haven't tested that but I don't think it would be an issue. As long
as it doesn't break out of the function declaration, it shoulld work
and afaik, you can include "exit" inside a function. I'm not a Bash
expert though, so correct me if I'm wrong.

[aur-dev] Incorrect URLPath Data in JSON Info

[Xyne]

Hi,

This came up in the Bauerbill thread:
http://bbs.archlinux.org/viewtopic.php?pid=686032#p686032

The URLPath returned in the JSON info for "gsm" points to
"packages/gsm/archive.tar.gz" instead of "packages/gsm/gsm.tar.gz". The
package has not been updated in two years so I suspect that the error
is due to a lazy code migration during that time, one which relied on
all packagers reuploading their packages to complete the migration.

Would it be very difficult to update the URLPaths across the whole
database? I realize that most packages which have not been updated are
probably obsolete/abandoned, but considering the sheer number of
packages in the AUR, many of them will still be valid.

Regards,
Xyne

Re: [aur-dev] Safe and relatively reliable PKGBUILD parser.

[Xyne]

Loui Chang wrote:

> Wow this is quite clever. It definitely would make the job of parsing
> much easier. Thanks for the explanation.

:)

I intend to flesh out the parser as special cases pop up. As already
mentioned, there will be limits to what it can do depending on whether
the packager uses command output to set variables, but perhaps Arch
could eventually impose a de facto standard for PKGBUILDs using the
parser itself as the standard, i.e. if the PKGBUILD metadata gets
past the parser, the PKGBUILD itself would be considered invalid. In
that case, the parser would support tricks such as

[ "$ARCH" == "x86_64" ] && depends=('foo' 'bar')

I want to be very clear that I am NOT suggesting that my parser become
the standard, only that a parser based on this approach _could_ become
one.

Also note that this is really a method on its own that just happens to
be implemented in Perl in this case. If you look at the code, you will
see that it could very quickly be adapted to Python (and thus Django),
or PHP, or just about anything.

Re: [aur-dev] Safe and relatively reliable PKGBUILD parser.

[Xyne]

> It is quite a clever idea. I haven't seen this approach before. I  
> haven't looked at it thoroughly, but it looks like you're simply  
> sourcing the PKGBUILD with some trickery not to execute the code. Why  
> then the need for further parsing? Does `set` produce "raw" bash, e.g.  
> 'source=("https://localhost/$pkgname.tgz";)'? It seems like bash should  
> be able to do it itself. If that were the case, the parser would be  
> extremely reliable (definitely more so than mine). There are still  
> some "safety" issues involved, although maybe not for your purposes.  
> One major thing is infinite loops - there's no way to break them. I'm  
> sure this parser will be very useful when such things aren't an issue.

You haven't fully understood how it works so I hope you don't mind if I
try to explain it again.

I first check the PKGBUILD with "/bin/bash -n PKGBUILD". If this
command exits without error then the PKGBUILD contains valid syntax,
most importantly it does not contain extra closing brackets ("}").

This lets me wrap the entire PKGBUILD in a function, e.g.
pkgbuild () {

}

I can then source the file with Bash without executing any code. The
previous check with "bash -n" guarantees that the PKGBUILD can not
escape the wrapping function. Because all code is inside a function,
sourcing the file does not execute any code at all.

Bash simply parses the file and stores the code itself in the
"pkgbuild" function, which itself contains other variables and
functions (e.g. package_foo, build). Because the code has not been
executed, the variables have not been expanded/interpolated and thus
still contain things such s "http://example.com/$pkgname-$pkgver.tar";,
which is why it must still be intepolated by the parser.

The advantage of this method is that "set" will print out the
"pkgbuild" function and its contents in a canonical form, e.g. all
assignments to a variable are on a single line, if/then/else statements
follow a single format, etc.

This makes it possible to easily parse the assignments themselves, in
the order that they occur, without haing to consider all variations of
valid whitespace in statements. The parser simply needs to recognize
Bash syntax for things such as string substitutions, but this is a
relatively limted set so it is not difficult to handle all such cases.
The output of "set" also guarantees that you have a representation of
all variable assignments (in sequential order, and within their local
environment) so you have all the information that you need to
interpolate them. You could even handle command output if you wish,
using a command white-list to make sure that no trickery is used to run
malicious code.

Let me repeat that my method does not run any code in the
PKGBUILD. I've tested this by including an infinite loop at the top of
the file and it was not executed. I actually believe that this method
provides a perfectly safe and potentially very reliable method of
retrieving all metadata in the PKGBUILD with very little dependencies
and considerable portability.


Regards,
Xyne

Re: [aur-dev] Safe and relatively reliable PKGBUILD parser.

[Xyne]

> What was the problem with that from Sebastian which was discussed
> earlier on the mailing lists, IRCs ? How does it know more ?
> 

I don't know. I wrote this because I needed a PKGBUILD parser in Perl
for Bauerbill. Maybe it's better, maybe it's worse. I posted it here in
case someone finds it useful per se, or wishes to take any of the ideas
from it and use them to iimprove other parsers.

Hmmm, after briefly reviewing the messages, I can mention that my
parser:
* doesn't depent on Yacc/Lex
* supports split packages already
* handles multiline assignments
* supports interpolation and string substitutions

[aur-dev] Safe and relatively reliable PKGBUILD parser.

[Xyne]

Hi,

There was no response on the pacman-dev list but someone here might
find this potentially useful:

http://mailman.archlinux.org/pipermail/pacman-dev/2010-January/010322.html

It's written in Perl but it could easily be adapted to Python. It
handles Bash variable interpolation and string substitutions so far.
I'll add support for other PKGBUILD "tricks"as I come across them, e.g.
setting dependencies by architecture. The post linked above includes
simplified JSON output from the split kernel26 package.

Regards,
Xyne

Re: [aur-dev] Sort votes by descending date.

[Xyne]

Andrea Scarpino wrote:

> patch attached

Thanks.

Re: [aur-dev] Sort votes by descending date.

[Xyne]

> Votes may be few and far between, but this is a recurring annoyance
> that should be very easy to fix.

I should add that the annoyance is needing to click through pages,
although I just noticed the GET variables in the URL. Still, descending
by date should be the default as it is the one that is likely to be the
most useful.

[aur-dev] Sort votes by descending date.

[Xyne]

HI,

I wasn't sure if it would be more appropriate to send this to this list
where the AUR devs are likely to see it, or to AUR-general where the
people most affected are likely to see it.

The TU section of the AUR displays 6 votes per page sorted in ascending
order by date. It would make more sense to sort them descending by date
so that the most recent votes are the first to be seen instead of votes
that are nearly 2 years old.

Votes may be few and far between, but this is a recurring annoyance
that should be very easy to fix.

Thanks.


Regards,
Xyne

p.s. I can call a vote on this issue to see what the others think, if
you like. :P

Re: [aur-dev] aur-general: I no longer receive messages.

[Xyne]

> > Have you tried a different mail client?

I'm using the same client (sylpheed) that I've been using since I first
subscribed to the list. It works for aur-dev and pacman-dev on the
same account so I don't see how it could be a problem with the
client. This is not connected to an update either.

Either aur-general is not bouncing my emails or the archlinux.ca host
is not passing them on. I just don't know which.

[aur-dev] aur-general: I no longer receive messages.

[Xyne]

Can someone please check if there's anything wrong with my subscription
to aur-general? I no longer recieve messages from the list. I've tried
unsubscribing and resubscring and have checked my options as well.

I even get confirmation emails when I post to the list.

I don't think it's my email client (sylpheed) either as aur-dev and
other lists still work as before (all of which use the same email
account). I have checked my junk mail but there's nothing there (from
Arch anyway).

Any help would be appreciated. Thanks.


Regards,
Xyne

Re: [aur-dev] [PATCH 2/2] Add Swedish translation

[Xyne]

> Don't sign off the Swedish translation yet. There are typos, blatant
> language errors and lots of "svengelska" (Swenglish). Half of it looks
> like it was passed through an online translator (no offense).

Here's the same patch with several changes. I would recommend getting
someone else to check it too. There are some errors in the English as
well but I didn't think the patch was meant to fix that.

Regards,
Xyne


0001-Add-Swedish-translation.patch
Description: Binary data

Re: [aur-dev] [PATCH 2/2] Add Swedish translation

[Xyne]

> Hello!
> 
> You can reach here the Swedish translation:
> http://djszapi.homelinux.net/0001-Add-Swedish-translation.patch
> 
> Best Regards,
> Laszlo Papp

Don't sign off the Swedish translation yet. There are typos, blatant
language errors and lots of "svengelska" (Swenglish). Half of it looks
like it was passed through an online translator (no offense).

Re: [aur-dev] Te agregué como amigo en Facebook

[Xyne]

On Fri, 28 Aug 2009 17:08:53 -0700
Benjamin Martinez Velasco  wrote:

> Hola, aur-dev@archlinux.org:
> 
> He creado un perfil en Facebook donde puedo publicar mis fotos, vídeos y 
> eventos, y quiero agregarte a mis amigos para que puedas verlo. Para ello, 
> necesitas registrarte en Facebook y, después, podrás crear también tu propio 
> perfil.
> 
> Gracias,
> Benjamin


Bueno, quien quiere crear un perfil por aur-dev? :P
(Ok, so who wants to create a profile for aur-dev?)


Hobbies and interests: accepting and relaying emails, providing a
medium for discussion, flipping harddisk bits, getting archived (oh
yeah!).

Re: [aur-dev] PKGBUILD parsing on the AUR

[Xyne]

> As I worked on most of the parsing code at the moment: we don't have
> any "if-then-else" statements. AUR can (currently) handle only static
> assignments and bash arrays (or how are they called) and evals.
> 
> This is because we don't evaluate the "bash" part of the script just
> the variable assignments. Would be nice to have a whole bash
> interpreter or similar, but I don't have an idea how to get that done
> securely at the moment Any suggestions?
> E.g. that example on the top would not get through...
> 
> As for the linked feature request: the "depends" should be
> straightforward (I didn't look into it much, yet, but we should have
> all the required data). The "makedepends" is not stored for the web
> interface, so should be added there first. Would be quite valuable, I
> think... Even optdepends could be useful if we would add it one day,
> the only problem is, that it does not seem to have a standard format
> across packages.
> 
> So, on a related note, making such upgrades: patches for the pages
> (php or html...) are straightforward. How does someone send a patch
> which would need a change in the SQL database in the back?
> 
> Cheers,
> Greg

Thanks for the replies, Greg.

I suspected that it was scraping the PKGBUILDs due to discrepancies
reported on the bug tracker (valid PKGBUILDs not being properly parsed
on the AUR, etc).

There is no fully functional solution to extracting data from bash
PKGBUILDs. Sourcing it exposes you to malicious code and implementing a
full interpreter would do the same (as it would have to handle command
outputs, which in turn means running commands). Anything less will not
be able to fully parse all valid PKGBUILDs. That's why I've been
playing around with the idea of a non-bash PKGBUILD [1]. The AUR would
definitely benefit from this as would any application which interacts
with it. It would be nice to accurately resolve dependency trees using
only the JSON-RPC interface, for example.

Where can I view the code for the web pages?


[1] http://xyne.archlinux.ca/ideas/pkgmeta

[aur-dev] PKGBUILD parsing on the AUR

[Xyne]

Hi,

I'm hoping someone could answer these questions:

How does the AUR parse uploaded PKGBUILDs?
How does it handle PKGBUILDs with if-then-else statements? E.g.

depends=('foo' 'baz')
[ "$(uname -m)" == 'x86_64' ] && depends=('lib32-foo' 'lib32-baz')

How could I help implement this?:
http://bugs.archlinux.org/task/15728



Thanks,

Xyne

Re: [aur-dev] Using curl to upload to the AUR

[Xyne]

> I will keep trying to
> find a workaround for curl if this is the case, but could someone
> please check it it's possible to resolve this serverside?

It seems adding '-H "Expect:"' solves the problem but I would describe
this as a kludge.

[aur-dev] Using curl to upload to the AUR

[Xyne]

I am not sure if this is related to the recent move to a new server but
I am no longer able to use curl to upload to the AUR (via aurploader).
I can't see anything different in the package submit form but
attempting to upload a package now returns this page:


http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";>
http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
 
  417 - Expectation Failed
 
 
  417 - Expectation Failed
 



As the aurploader code has not changed in months, this must be due
either to a change in curl or in the AUR. I am still able to login
using aurploader and retrieve the page with the submit form. Googling
suggests that this is a lighttpd server error. I will keep trying to
find a workaround for curl if this is the case, but could someone
please check it it's possible to resolve this serverside?

Thanks.

Xyne