Re: Call for Help: need script for patching ports tree, building with poudriere

[Craig Rodrigues]

On Sun, Dec 6, 2015 at 6:12 AM, Ricky G  wrote:

> Hey,
>  Added this to github.
> https://github.com/Ultima1252/portest
>

Nice work!  Thanks for putting the effort into writing this script.
I'll provide feedback on github.

--
Craig
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

RE: Call for Help: need script for patching ports tree, building with poudriere

[Ricky G]

Hey,
 Added this to github. 
https://github.com/Ultima1252/portest 
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Call for Help: need script for patching ports tree, building with poudriere

[Torsten Zuehlsdorff]

Hello,


I'm sorry Ricky. I tried to contact you already, but hotmail.com blocked
my mailserver. And it is as always surprisingly hard to even find the
removal request form...


Why is it being blocked? What was the error code or other information
related to the blocking? There are numerous articles describing being
blocked by "hotmail" as well as other ISP, etcetera, but no one can really
help you unless you explain what the problem is.


The error code is 550 SC-001. This is quite funny, because my
mail-server is not listed at any spam/blocklists and in the last week
there were only 5 emails send to hotmail/outlook.com by a customer using
the mail-server.
(Yes, i could not write to you either)

I'm getting quite tired to keep up with the various changes and obscure
policies of other mail service provider. Yahoo for example changed their
policy, making usage of mailinglists nearly impossible.

I wasn't seeking for help on this issue - sorry if this was misleading.
I just make a promise to Ricky and want to give feedback, that i tried
to accomplish it, but were slowed down by technical (and schedule) issues.


Try: https://support.live.com/eform.aspx?productKey=edfsmsbl3&ct=eformts

More info is available here:
http://pc-freak.net/blog/howto-remove-delist-your-mail-server-ip-from-hotmail-live-com-and-msn-mail-server-blacklist/


I'm aware of this form and did request the removal already. Some minutes 
ago i get a positive feedback.



It could be as simple as your SPF record being incorrect.


It is not incorrect - it is simply not set. There is much controversy 
around this technique, but at least i never had a problem because of not 
using it. ;)


Greetings,
Torsten
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Call for Help: need script for patching ports tree, building with poudriere

[Carmel NY]

On Tue, 1 Dec 2015 12:26:05 +0100, Torsten Zuehlsdorff stated:

>On 01.12.2015 12:11, Carmel NY wrote:
>
>>> I'm sorry Ricky. I tried to contact you already, but hotmail.com blocked
>>> my mailserver. And it is as always surprisingly hard to even find the
>>> removal request form...  
>>
>> Why is it being blocked? What was the error code or other information
>> related to the blocking? There are numerous articles describing being
>> blocked by "hotmail" as well as other ISP, etcetera, but no one can really
>> help you unless you explain what the problem is.  
>
>The error code is 550 SC-001. This is quite funny, because my 
>mail-server is not listed at any spam/blocklists and in the last week 
>there were only 5 emails send to hotmail/outlook.com by a customer using 
>the mail-server.
>(Yes, i could not write to you either)
>
>I'm getting quite tired to keep up with the various changes and obscure 
>policies of other mail service provider. Yahoo for example changed their 
>policy, making usage of mailinglists nearly impossible.
>
>I wasn't seeking for help on this issue - sorry if this was misleading. 
>I just make a promise to Ricky and want to give feedback, that i tried 
>to accomplish it, but were slowed down by technical (and schedule) issues.

Try: https://support.live.com/eform.aspx?productKey=edfsmsbl3&ct=eformts

More info is available here:
http://pc-freak.net/blog/howto-remove-delist-your-mail-server-ip-from-hotmail-live-com-and-msn-mail-server-blacklist/

It could be as simple as your SPF record being incorrect.

Good Luck

-- 
Carmel
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Call for Help: need script for patching ports tree, building with poudriere

[Torsten Zuehlsdorff]

On 01.12.2015 12:11, Carmel NY wrote:


I'm sorry Ricky. I tried to contact you already, but hotmail.com blocked
my mailserver. And it is as always surprisingly hard to even find the
removal request form...


Why is it being blocked? What was the error code or other information related
to the blocking? There are numerous articles describing being blocked by
"hotmail" as well as other ISP, etcetera, but no one can really help you
unless you explain what the problem is.


The error code is 550 SC-001. This is quite funny, because my 
mail-server is not listed at any spam/blocklists and in the last week 
there were only 5 emails send to hotmail/outlook.com by a customer using 
the mail-server.

(Yes, i could not write to you either)

I'm getting quite tired to keep up with the various changes and obscure 
policies of other mail service provider. Yahoo for example changed their 
policy, making usage of mailinglists nearly impossible.


I wasn't seeking for help on this issue - sorry if this was misleading. 
I just make a promise to Ricky and want to give feedback, that i tried 
to accomplish it, but were slowed down by technical (and schedule) issues.


Greetings,
Torsten
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Call for Help: need script for patching ports tree, building with poudriere

[Carmel NY]

On Tue, 1 Dec 2015 08:30:06 +0100, Torsten Zühlsdorff stated:

>I'm sorry Ricky. I tried to contact you already, but hotmail.com blocked 
>my mailserver. And it is as always surprisingly hard to even find the 
>removal request form...

Why is it being blocked? What was the error code or other information related
to the blocking? There are numerous articles describing being blocked by
"hotmail" as well as other ISP, etcetera, but no one can really help you
unless you explain what the problem is.

-- 
Carmel
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Call for Help: need script for patching ports tree, building with poudriere

[Torsten Zühlsdorff]

Hello,


That is a good attitude :) I am doing this also just to get in
touch with something new and learn. As a professional trainer there
is a trick in learning: whenever a question arises - by yourself or
by another person: take the chance and look for it. So you will
remember the answer much better. :) If you need a
programming-mentor for this project feel free to contact me
off-list.


That would be great! your time will be most appreciated!


I'm sorry Ricky. I tried to contact you already, but hotmail.com blocked 
my mailserver. And it is as always surprisingly hard to even find the 
removal request form...


In short: because of a schedule shift i am very short on time. I have to 
match an deadline for an import project first. After that i will review 
your script and get more input for us from Craig. :)


Do you have any code repository in action? It would be suboptimal if i 
review you last send version just to figure out you already changed the 
mentioned parts. ;)


Hopefully my removal request is processed in the meantime.

Thanks you very much,
Torsten
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

RE: Call for Help: need script for patching ports tree, building with poudriere

[Ricky G]

> That is a good attitude :) I am doing this also just to get in touch 
> with something new and learn.
> As a professional trainer there is a trick in learning: whenever a 
> question arises - by yourself or by another person: take the chance and 
> look for it. So you will remember the answer much better. :)
> If you need a programming-mentor for this project feel free to contact 
> me off-list.


 That would be great! your time will be most appreciated!
> 
> To answer my own question, yes there is
> 
> You wrote:
> 
> ===
> if [ -d "${PORTSDIR}" ]; then
> else
> ===
> 
> Which directly skips the if condition without doing something.
> 
> Rewrite it with:
> 
> ===
> if ! [ -d "${PORTSDIR}" ]; then
> ===
> 
> This will test if the dir is non existent - exactly what you want! :)


 Yes, I remember this problem now and its fixed! I couldn't figure out where 
the explanation mark went. Just missed the space! Thanks so much for clearing 
that up.
> >> Line 162: BUILD_LIST="`(cat ${PATCH_FILE} | grep -o '^Index:[ ].*'
> >> | sed 's/Index: // ; s/Mk.*// ; s/Tools.*// ; s/Templates.*// ;
> >> s/Keywords.*//' | grep -o -e '.*\/.*\/' | sed 's/.*\/.*\/files\/$//
> >> ; s/\/$//' | sort | awk '!a[$0]++')`"
> >>
> >> This line is relative complicated. Is there any reason not to let
> >> svn do the work? Like: BUILD_LIST="`${SVN} status | grep -o -e
> >> '.*\/.*\/' | awk '{ print $2 }'`> Your way is faster (and more
> >> accurate, since i'm currently too short on time), but the other way
> >> would allow applying multiple patches and manual changes without
> >> any problems. But i'm not sure if this is really needed.
>  >
> > Using the diff file instead of svn is of course faster because there
> > has much less data to go through. I was considering adding multiple
> > diff support. Going to take a closer look at this. Do you think its
> > more important to focus on the diff file or the port tree for
> > generating a build list?
> 
> That is a hard question. I asked for svn because i am from Germany and 
> facing often problems with umlauts. My lastname is "Zühlsdorff" but i 
> rewrite it for non-german person and programs to "Zuehlsdorff". Sounds 
> the same, means the same.
> 
> Back to the core of the question: encoding problems.
> 
> Can you guarantee that cat and sed work correctly on files with for 
> example an "ü" or containing something like '合�莸�'? ;)> > Using svn avoids 
> this problem, because in the portstree you could not 
> find filenames with such characters.


 Just added an option that will generate a list via tree over diff. You have a 
very valid point about encoding. Have never considered this and I have no idea 
how to work this problem out. My only (human) language is english. Will message 
you off-list to see if we can fix this issue.

 There are a few things that need to be filtered in order to guarantee a 
properly generated list. Also the change I made recently, it fixes the issue 
with striping possible 15ish ports that end with files, however it will remove 
the cat/port/files instead of just striping files. Still trying to find the 
best solution.


> 
> An additional though: you are assuming that everything is fine with the 
> input. It accepts all tracks of my music folder without any hestination 
> as a patch-file. ;) Okay, patching fails - but i tries hard and the new 
> versions provides much binary trash through the "cat" after failing.
> 
> I don't believe we must make it bullet proof. Here we need some input 
> from Craig about the way and environment the script will be used in.


 Another great point. Did consider this when I started, however haven't yet 
come up with the proper solution for verifying diff file or not. The way I was 
considering was just something along the lines of head -5 | grep -o Index:
as it is the most consistent thing in patch files. If you wondering why I 
choose -5, in cases of comments being on header, 5 lines seems reasonable. This 
could cause more encoding issues tho, I'm not sure.


> > The main reason I didn't want to use the
> > port tree, at least in my case, is because I have so many ports that
> > are not sync'd with the current FreeBSD tree.
> 
> Okay, but therefore you added the PORTSDIR variable?


 Think this may have been misunderstood, or I mislead. I have many ports not 
synced with the current head. Rather not generate a list for each of those as 
well as the patched ones I want to test.


> >> Line 201: svn revert -R would do the same, both of our approaches
> >> have a problem: the ignore for example added files.
>  >
> > Yeah, I agree. This is a big problem with added patchfiles. I added a
> > "Hard Revert" (-R) option that I was considering making make the
> > default that should fix this problem. Can you check and see if it
> > does for you?
> 
> I will check this.
> 
> > I know some people maybe weary of the rm -rf command,
> > It is also why I added so many checks on a properly set port tree. Do
> > you think it would be better

Re: Call for Help: need script for patching ports tree, building with poudriere

[Torsten Zuehlsdorff]

Hello Ricky,


Thank you very much for your script!

I took a short look at it and have some questions/suggestions:

Line 119/120: There you're burning an if. I have more skill in bash
than in sh - but is there no negation possible?


Honestly, the main reason I started this is to just see what I could
do. Feedback is most appreciated as improving my skills is one of my
goals here. Bash and sh are extremely similar, this is most likely
lack of knowledge on my part. Do you mind explaining further?


That is a good attitude :) I am doing this also just to get in touch 
with something new and learn.
As a professional trainer there is a trick in learning: whenever a 
question arises - by yourself or by another person: take the chance and 
look for it. So you will remember the answer much better. :)
If you need a programming-mentor for this project feel free to contact 
me off-list.


To answer my own question, yes there is

You wrote:

===
if [ -d "${PORTSDIR}" ]; then
else
===

Which directly skips the if condition without doing something.

Rewrite it with:

===
if ! [ -d "${PORTSDIR}" ]; then
===

This will test if the dir is non existent - exactly what you want! :)


Line 162: BUILD_LIST="`(cat ${PATCH_FILE} | grep -o '^Index:[ ].*'
| sed 's/Index: // ; s/Mk.*// ; s/Tools.*// ; s/Templates.*// ;
s/Keywords.*//' | grep -o -e '.*\/.*\/' | sed 's/.*\/.*\/files\/$//
; s/\/$//' | sort | awk '!a[$0]++')`"

This line is relative complicated. Is there any reason not to let
svn do the work? Like: BUILD_LIST="`${SVN} status | grep -o -e
'.*\/.*\/' | awk '{ print $2 }'`> Your way is faster (and more
accurate, since i'm currently too short on time), but the other way
would allow applying multiple patches and manual changes without
any problems. But i'm not sure if this is really needed.

>

Using the diff file instead of svn is of course faster because there
has much less data to go through. I was considering adding multiple
diff support. Going to take a closer look at this. Do you think its
more important to focus on the diff file or the port tree for
generating a build list?


That is a hard question. I asked for svn because i am from Germany and 
facing often problems with umlauts. My lastname is "Zühlsdorff" but i 
rewrite it for non-german person and programs to "Zuehlsdorff". Sounds 
the same, means the same.


Back to the core of the question: encoding problems.

Can you guarantee that cat and sed work correctly on files with for 
example an "ü" or containing something like '合気道'? ;)


Using svn avoids this problem, because in the portstree you could not 
find filenames with such characters.


An additional though: you are assuming that everything is fine with the 
input. It accepts all tracks of my music folder without any hestination 
as a patch-file. ;) Okay, patching fails - but i tries hard and the new 
versions provides much binary trash through the "cat" after failing.


I don't believe we must make it bullet proof. Here we need some input 
from Craig about the way and environment the script will be used in.



The main reason I didn't want to use the
port tree, at least in my case, is because I have so many ports that
are not sync'd with the current FreeBSD tree.


Okay, but therefore you added the PORTSDIR variable?


Line 201: svn revert -R would do the same, both of our approaches
have a problem: the ignore for example added files.

>

Yeah, I agree. This is a big problem with added patchfiles. I added a
"Hard Revert" (-R) option that I was considering making make the
default that should fix this problem. Can you check and see if it
does for you?


I will check this.


I know some people maybe weary of the rm -rf command,
It is also why I added so many checks on a properly set port tree. Do
you think it would be better to just do this "Hard Revert" and remove
the other option? Thanks for the feedback!


The revert option should do what they say: reverting. The technical 
details are the second step. As a rule of thumb: do not lie. When you 
claim to revert do this. If this needs an rm than do the rm.



I'v already started
working on 0.0.2! I just finished adding better failed patch output.
Going to also add a option to override the scripts portsdir, and add
support for multiple diff files, and take another look at my
BUILD_LIST to make it simpler. Keep the feed back coming please! =]


That are great news :)

One last question: is there any reason you choose this license? The 
FreeBSD license contains just the first 2 paragraphs which makes 
redistribution much easier. Especially paragraph 3 makes you script for 
example incompatible with the GNU licensed programs.


https://www.freebsd.org/copyright/freebsd-license.html

Greetings,
Torsten
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

RE: Call for Help: need script for patching ports tree, building with poudriere

[Ricky G]

Portest version 0.0.2 
Changelog:
* Fixed Usage with incorrect description on some options* Added multiple 
patchfile/glob support* More failed patch output, Still working on formating. 
Ideas would be great.* Changed BUILT_LIST filter to fix unintended stripping of 
multiple ports* Changed revert (-R) to (-r) and added a new revert (-R) 
method.* Several changed to revert more explained below

 As pointed out to me, svn revert -R seems to somehow miss small filenames. 
This is an svn bug and as a workaround I decided to remove the -r option as it 
was unreliable altogether, move the -R to -r, and added a filtered revert for 
the smaller filenames. This solution works well on all my tests with 15+ large 
diff files, but of course more testing is needed. The new -R option will 
instead of reverting according to the diff, revert based on changed from the 
tree. This method is much slower but in the event -r may fail an -R will fix 
everything regardless of .diff.

portest
Description: Binary data
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

RE: Call for Help: need script for patching ports tree, building with poudriere

[Ricky G]

> Thank you very much for your script!
> 
> I took a short look at it and have some questions/suggestions:
> 
> Line 119/120:
> There you're burning an if. I have more skill in bash than in sh - but 
> is there no negation possible?

 Honestly, the main reason I started this is to just see what I could do. 
Feedback is most appreciated as improving my skills is one of my goals here. 
Bash and sh are extremely similar, this is most likely lack of knowledge on my 
part. Do you mind explaining further?
> 
> Line 162:
> BUILD_LIST="`(cat ${PATCH_FILE} | grep -o '^Index:[ ].*' | sed 's/Index: 
> // ; s/Mk.*// ; s/Tools.*// ; s/Templates.*// ; s/Keywords.*//' | grep 
> -o -e '.*\/.*\/' | sed 's/.*\/.*\/files\/$// ; s/\/$//' | sort | awk 
> '!a[$0]++')`"
> 
> This line is relative complicated. Is there any reason not to let svn do 
> the work? Like:
> BUILD_LIST="`${SVN} status | grep -o -e '.*\/.*\/' | awk '{ print $2 }'`> 
> Your way is faster (and more accurate, since i'm currently too short on 
> time), but the other way would allow applying multiple patches and 
> manual changes without any problems. But i'm not sure if this is really 
> needed.
 Using the diff file instead of svn is of course faster because there has much 
less data to go through. I was considering adding multiple diff support. Going 
to take a closer look at this. Do you think its more important to focus on the 
diff file or the port tree for generating a build list? The main reason I 
didn't want to use the port tree, at least in my case, is because I have so 
many ports that are not sync'd with the current FreeBSD tree.
 The main reason that command is complicated is to strip many possible non-port 
files/folders so that the list wont contain something like Mk/Uses/. The build 
list you suggested would work, but most of the sed command is still needed to 
stripe non-ports. Looking at it again, I see where I could make it a bit 
shorter and easier to read. The sort and awk command at the end is just make it 
look nice and remove duplicates. Not really required.
> Line 201:
> svn revert -R would do the same, both of our approaches have a problem: 
> the ignore for example added files.
 Yeah, I agree. This is a big problem with added patchfiles. I added a "Hard 
Revert" (-R) option that I was considering making make the default that should 
fix this problem. Can you check and see if it does for you? I know some people 
maybe weary of the rm -rf command, It is also why I added so many checks on a 
properly set port tree. Do you think it would be better to just do this "Hard 
Revert" and remove the other option?
Thanks for the feedback! I'v already started working on 0.0.2! I just finished 
adding better failed patch output. Going to also add a option to override the 
scripts portsdir, and add support for multiple diff files, and take another 
look at my BUILD_LIST to make it simpler. Keep the feed back coming please! =]

Ultima
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Call for Help: need script for patching ports tree, building with poudriere

[Torsten Zühlsdorff]

Hello Ricky,


Hello, Decided to not add a config file to keep it simpler, so you'll
need to edit the variables to your needs below the options. The only
packages that it may need is portlint, porttools and/or poudriere.
Let me know what you think or needs improvement.


Thank you very much for your script!

I took a short look at it and have some questions/suggestions:

Line 119/120:
There you're burning an if. I have more skill in bash than in sh - but 
is there no negation possible?


Line 162:
BUILD_LIST="`(cat ${PATCH_FILE} | grep -o '^Index:[ ].*' | sed 's/Index: 
// ; s/Mk.*// ; s/Tools.*// ; s/Templates.*// ; s/Keywords.*//' | grep 
-o -e '.*\/.*\/' | sed 's/.*\/.*\/files\/$// ; s/\/$//' | sort | awk 
'!a[$0]++')`"


This line is relative complicated. Is there any reason not to let svn do 
the work? Like:

BUILD_LIST="`${SVN} status | grep -o -e '.*\/.*\/' | awk '{ print $2 }'`

Your way is faster (and more accurate, since i'm currently too short on 
time), but the other way would allow applying multiple patches and 
manual changes without any problems. But i'm not sure if this is really 
needed.


Line 201:
svn revert -R would do the same, both of our approaches have a problem: 
the ignore for example added files.


Greetings,
Torsten
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

RE: Call for Help: need script for patching ports tree, building with poudriere

[Ricky G]

 Hello, 
 Decided to not add a config file to keep it simpler, so you'll need to edit 
the variables to your needs below the options. The only packages that it may 
need is portlint, porttools and/or poudriere. Let me know what you think or 
needs improvement.  

portest
Description: Binary data
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Call for Help: need script for patching ports tree, building with poudriere

[Bryan Drewery]

On 11/7/2015 12:28 AM, Craig Rodrigues wrote:
> Hi,
> 
> I've been working on some advanced scripting with Eitan Adler.
> We are interested in experimenting with integrating Jenkins + Phabricator
> using: https://wiki.jenkins-ci.org/display/JENKINS/Phabricator+Plugin
> For starters, we want to see what we can accomplish with ports.

I'm too busy to help on this project at the time, but if you come across
any hooks that would be beneficial please let me know and I'll add them
into Poudriere.

https://github.com/freebsd/poudriere/wiki/hooks

-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

RE: Call for Help: need script for patching ports tree, building with poudriere

[Ricky G]

 Almost finished with this, just wanted to give an update. I named this script 
portest, I still need to add a few more things and do some bug testing but so 
far it seems to work pretty well. This is the usage page for the script.
Usage: portest [-f bulkfile] [-apPrRtv] patchfile
Options:-a --  Do everything except revert (-ptf build.ports.txt)-f 
--  Generate a poudiere usable bulk file-p --  Patch and give 
output-P --  Do not exit on failed patch-r --  Revert the files 
listed in patchfile-R --  Paranoid revert, will remove and restore 
anything and   everything the patchfile may have changed or added   
 -t --  Test with portlint-v --  Show version of portest
By default, (no options) portest shows the ports that will be modified

 Decided to add the -a option to do the task you asked for specifically. So far 
every option works on the patch you supplied and several others I had laying 
around. One thing that it will not detect is if slave port will be affected by 
the change, also changes to Uses or any other non CAT/PORT directory will not 
be detected. I'm still trying to figure out how to solve these problem. Are 
they important? or not so much?
 Right now on my todo is to add a config file to change the default settings, 
Add a -T option that will do port test or poudrieretesting depending on what 
was selected in the config file. Also later on choose between svn, git or 
portsnap. Once i'm finished adding the config file ill post it for some testing 
and if you can give me some feedback on what else you need or don't like.
Ultima
Date: Mon, 9 Nov 2015 13:55:38 -0800
Subject: Re: Call for Help: need script for patching ports tree, building with 
poudriere
From: rodr...@freebsd.org
To: ricky1...@hotmail.com
CC: freebsd-test...@freebsd.org; freebsd-ports@freebsd.org

On Sun, Nov 8, 2015 at 6:18 PM, Ricky G  wrote:



Hello, 
 Started working on a simple sh script to do as requested. So far have 1,2,4,5 
complete. There is much error checking to be added still though. Wanted to ask 
a few things first.
2- Does port tree checkout matter? Right now my script assumes svnlite 
generated the patch at the root of the tree5- Does duplicates matter? poudriere 
doesn't care but is this strictly for poudriere?6- How should the output be 
handled?

The script should not worry about the port tree checkout so much,
since that will probably be done outside the script.  For now, if the script
assumes that the port checkout is an SVN tree, and you can run SVN operations 
on it, that should be sufficient.

For duplicates, I don't think it matters so much.  I believe poudriere is smart 
enough so that if you pass it a list of ports with duplicate entries,
it does the right thing.

At this point, the output format is not so important.  For now,
making sure that error status is propagated as a return value from
the script (0 on success, non-zero on failure) is a good start.

The only other suggestion that I have is that when you work on the
script, make sure that you put a proper license in it (BSD license preferred).

Thanks.

--
Craig
  
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Call for Help: need script for patching ports tree, building with poudriere

[Craig Rodrigues]

On Sun, Nov 8, 2015 at 6:18 PM, Ricky G  wrote:

> Hello,
>
>  Started working on a simple sh script to do as requested. So far have
> 1,2,4,5 complete. There is much error checking to be added still though.
> Wanted to ask a few things first.
>
> 2- Does port tree checkout matter? Right now my script assumes svnlite
> generated the patch at the root of the tree
> 5- Does duplicates matter? poudriere doesn't care but is this strictly for
> poudriere?
> 6- How should the output be handled?
>
>
The script should not worry about the port tree checkout so much,
since that will probably be done outside the script.  For now, if the script
assumes that the port checkout is an SVN tree, and you can run SVN
operations on it, that should be sufficient.

For duplicates, I don't think it matters so much.  I believe poudriere is
smart enough so that if you pass it a list of ports with duplicate entries,
it does the right thing.

At this point, the output format is not so important.  For now,
making sure that error status is propagated as a return value from
the script (0 on success, non-zero on failure) is a good start.

The only other suggestion that I have is that when you work on the
script, make sure that you put a proper license in it (BSD license
preferred).

Thanks.

--
Craig
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

RE: Call for Help: need script for patching ports tree, building with poudriere

[Ricky G]

Hello, 
 Started working on a simple sh script to do as requested. So far have 1,2,4,5 
complete. There is much error checking to be added still though. Wanted to ask 
a few things first.
2- Does port tree checkout matter? Right now my script assumes svnlite 
generated the patch at the root of the tree5- Does duplicates matter? poudriere 
doesn't care but is this strictly for poudriere?6- How should the output be 
handled?


> Date: Sat, 7 Nov 2015 00:28:28 -0800
> Subject: Call for Help: need script for patching ports tree, building with 
> poudriere
> From: rodr...@freebsd.org
> To: freebsd-test...@freebsd.org; freebsd-ports@freebsd.org
> 
> Hi,
> 
> I've been working on some advanced scripting with Eitan Adler.
> We are interested in experimenting with integrating Jenkins + Phabricator
> using: https://wiki.jenkins-ci.org/display/JENKINS/Phabricator+Plugin
> For starters, we want to see what we can accomplish with ports.
> 
> Can someone provide a script that does the following:
> 
> (1)  Takes a patch file
> (2)  Applies the patch file to a checked out ports tree
> (3)  If patching fails, quit and report an error
> (4)  If patch succeeds, do a simple guess to figure out which ports were
> affected
> (5)  Create a file build.ports.txt which lists these ports, and can be used
> as input
>  to poudriere, by invoking:
> 
>   poudriere bulk -f build.ports.txt
> 
> (6)  In addition, run portlint, port test, and any other appropriate
> scripts to
>  test that the patched ports following the porting style guidelines.
> 
> For example, if the patch in this review was submitted:
> https://reviews.freebsd.org/D3791
> then a build.ports.txt file with this content would be created:
> 
> 
> databases/py-Pyrseas
> databases/py-pylibmc
> databases/py-redis
> deskutils/py-bugwarrior
> deskutils/py-taskw
> devel/py-Jinja2
> devel/py-TGScheduler
> devel/py-apscheduler
> devel/py-asn1
> devel/py-asyncio
> devel/py-biplist
> devel/py-check-manifest
> devel/py-circuits
> devel/py-dateutil
> devel/py-doit
> devel/py-fabric
> devel/py-filemagic
> devel/py-foolscap
> devel/py-freezegun
> devel/py-future
> devel/py-lazy
> devel/py-logan
> devel/py-mock
> devel/py-nose
> devel/py-phabricator
> devel/py-pip
> devel/py-protobuf
> devel/py-pyflakes
> devel/py-pygit2
> devel/py-pygithub
> devel/py-pytest-cache
> devel/py-pytest
> devel/py-python-bugzilla
> devel/py-sanetime
> devel/py-scripttest
> devel/py-semantic_version
> devel/py-setuptools
> devel/py-simplejson
> devel/py-tox
> devel/py-twiggy
> devel/py-tzlocal
> devel/py-virtualenv
> finance/py-stripe
> ftp/py-pyftpdlib
> math/py-graphillion
> math/py-pycosat
> net/py-beanstalkc
> net/py-eventlet
> net/py-netaddr
> net/py-oauth2
> net/py-pynsq
> net/py-pyzmq
> net/py-twitter-tools
> net/turses
> science/py-h5py
> security/py-cryptography
> security/py-ecdsa
> security/py-libnacl
> security/py-oauthlib
> security/py-openssl
> security/py-pycrypto
> security/py-pysha3
> sysutils/py-nagiosplugin
> sysutils/py-supervisor
> textproc/py-diff-match-patch
> textproc/py-mistune
> textproc/py-transifex-client
> textproc/py-wtforms
> www/geeknote
> www/py-aiohttp
> www/py-bleach
> www/py-cactus
> www/py-django-contrib-comments
> www/py-django-mezzanine
> www/py-feedgenerator
> www/py-flask-wtf
> www/py-flexget
> www/py-gandi.cli
> www/py-grequests
> www/py-gunicorn
> www/py-pelican
> www/py-praw
> www/py-pyjwt
> www/py-rackspace-monitoring
> www/py-requests-oauthlib
> www/py-requests
> www/py-slimit
> www/py-slumber
> 
> and poudriere would be invoked to build those ports.
> 
> Does such a script exist?
> 
> If not, would someone by interested in working on such a script, and
> providing it
> via GitHub at https://github.com/freebsd/freebsd-ci/ ?
> 
> Thanks.
> 
> --
> Craig
> ___
> freebsd-ports@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
  
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Call for Help: need script for patching ports tree, building with poudriere

[Philip M. Gollucci]

Look at pgollucci/freebsd-bugzilla-cli

On Saturday, November 7, 2015, Craig Rodrigues  wrote:

> Hi,
>
> I've been working on some advanced scripting with Eitan Adler.
> We are interested in experimenting with integrating Jenkins + Phabricator
> using: https://wiki.jenkins-ci.org/display/JENKINS/Phabricator+Plugin
> For starters, we want to see what we can accomplish with ports.
>
> Can someone provide a script that does the following:
>
> (1)  Takes a patch file
> (2)  Applies the patch file to a checked out ports tree
> (3)  If patching fails, quit and report an error
> (4)  If patch succeeds, do a simple guess to figure out which ports were
> affected
> (5)  Create a file build.ports.txt which lists these ports, and can be used
> as input
>  to poudriere, by invoking:
>
>   poudriere bulk -f build.ports.txt
>
> (6)  In addition, run portlint, port test, and any other appropriate
> scripts to
>  test that the patched ports following the porting style guidelines.
>
> For example, if the patch in this review was submitted:
> https://reviews.freebsd.org/D3791
> then a build.ports.txt file with this content would be created:
>
>
> databases/py-Pyrseas
> databases/py-pylibmc
> databases/py-redis
> deskutils/py-bugwarrior
> deskutils/py-taskw
> devel/py-Jinja2
> devel/py-TGScheduler
> devel/py-apscheduler
> devel/py-asn1
> devel/py-asyncio
> devel/py-biplist
> devel/py-check-manifest
> devel/py-circuits
> devel/py-dateutil
> devel/py-doit
> devel/py-fabric
> devel/py-filemagic
> devel/py-foolscap
> devel/py-freezegun
> devel/py-future
> devel/py-lazy
> devel/py-logan
> devel/py-mock
> devel/py-nose
> devel/py-phabricator
> devel/py-pip
> devel/py-protobuf
> devel/py-pyflakes
> devel/py-pygit2
> devel/py-pygithub
> devel/py-pytest-cache
> devel/py-pytest
> devel/py-python-bugzilla
> devel/py-sanetime
> devel/py-scripttest
> devel/py-semantic_version
> devel/py-setuptools
> devel/py-simplejson
> devel/py-tox
> devel/py-twiggy
> devel/py-tzlocal
> devel/py-virtualenv
> finance/py-stripe
> ftp/py-pyftpdlib
> math/py-graphillion
> math/py-pycosat
> net/py-beanstalkc
> net/py-eventlet
> net/py-netaddr
> net/py-oauth2
> net/py-pynsq
> net/py-pyzmq
> net/py-twitter-tools
> net/turses
> science/py-h5py
> security/py-cryptography
> security/py-ecdsa
> security/py-libnacl
> security/py-oauthlib
> security/py-openssl
> security/py-pycrypto
> security/py-pysha3
> sysutils/py-nagiosplugin
> sysutils/py-supervisor
> textproc/py-diff-match-patch
> textproc/py-mistune
> textproc/py-transifex-client
> textproc/py-wtforms
> www/geeknote
> www/py-aiohttp
> www/py-bleach
> www/py-cactus
> www/py-django-contrib-comments
> www/py-django-mezzanine
> www/py-feedgenerator
> www/py-flask-wtf
> www/py-flexget
> www/py-gandi.cli
> www/py-grequests
> www/py-gunicorn
> www/py-pelican
> www/py-praw
> www/py-pyjwt
> www/py-rackspace-monitoring
> www/py-requests-oauthlib
> www/py-requests
> www/py-slimit
> www/py-slumber
>
> and poudriere would be invoked to build those ports.
>
> Does such a script exist?
>
> If not, would someone by interested in working on such a script, and
> providing it
> via GitHub at https://github.com/freebsd/freebsd-ci/ ?
>
> Thanks.
>
> --
> Craig
> ___
> freebsd-test...@freebsd.org  mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-testing
> To unsubscribe, send any mail to "freebsd-testing-unsubscr...@freebsd.org
> "
>


-- 
-
Curb: Your ride is here
4096R/D21D2752
 ECDF B597
B54B 7F92 753E  E0EA F699 A450 D21D 2752
Philip M. Gollucci (pgollu...@p6m7g8.com) c: 703.336.9354
Member,   Apache Software Foundation
Committer,FreeBSD Foundation
Consultant,   P6M7G8 Inc.
Sr. Director IT Operations,   Curb

What doesn't kill us can only make us stronger;
Except it almost kills you.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Call for Help: need script for patching ports tree, building with poudriere

[Craig Rodrigues]

On Sat, Nov 7, 2015 at 9:28 AM, Torsten Zühlsdorff <
mailingli...@toco-domains.de> wrote:

>
>
> It:
> - assumes that patches are generated against /usr/ports (which makes
> applying much easier)
> - apply the patches
> - get the ports affected by the diff with svn diff
> - runs portlint
> - runs poudriere for many versions...
> - reset every change made to the portstree
>
>
That flow of steps is very close to what I need.
The scripts in http://github.com/freebsd/freebsd-ci
are mostly shell or Python, but I'm flexible,
as long as the scripts run under FreeBSD.

I think you mentioned one problem about starting something, and then
running out of time to finish working on it.
This problem is very common. :(

Hopefully if we have multiple people working on something and
collaborating, we can get something going that many people can
maintain and enhance in future.

I would say go ahead and provide your script via GitHub pull request,
unless you want to clean it up a little before providing it.

Thanks.
--
Craig
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Call for Help: need script for patching ports tree, building with poudriere

[Torsten Zühlsdorff]

Hello Craig,


I've been working on some advanced scripting with Eitan Adler.
We are interested in experimenting with integrating Jenkins + Phabricator
using: https://wiki.jenkins-ci.org/display/JENKINS/Phabricator+Plugin
For starters, we want to see what we can accomplish with ports.

Can someone provide a script that does the following:

(1)  Takes a patch file
(2)  Applies the patch file to a checked out ports tree
(3)  If patching fails, quit and report an error
(4)  If patch succeeds, do a simple guess to figure out which ports were
affected
(5)  Create a file build.ports.txt which lists these ports, and can be used
as input
  to poudriere, by invoking:

   poudriere bulk -f build.ports.txt

(6)  In addition, run portlint, port test, and any other appropriate
scripts to
  test that the patched ports following the porting style guidelines.

For example, if the patch in this review was submitted:
https://reviews.freebsd.org/D3791
then a build.ports.txt file with this content would be created:


[List of ports]

and poudriere would be invoked to build those ports.

Does such a script exist?


I wrote myself a similar script, but it is very basic (and PHP).

It:
- assumes that patches are generated against /usr/ports (which makes 
applying much easier)

- apply the patches
- get the ports affected by the diff with svn diff
- runs portlint
- runs poudriere for many versions...
- reset every change made to the portstree

There are many glitches and it is not this fast, but it safe much time 
while i wrote port updates. I get a very simple report if everything is 
fine and then i submit my PR. But of course any other workflow is possible.


I aimed to support multiple poudriere server to speed up the process. 
But i have way more ideas than time... ;)



If not, would someone by interested in working on such a script, and
providing it
via GitHub at https://github.com/freebsd/freebsd-ci/ ?


I will help you, but currently (which means till the end of the year) i 
am very short on time.


Are there any requirements to the script in language, performance, etc. ?

Greetings,
Torsten

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"