On 24 January 2018 at 11:32, Samuel Rakitničan <
srakitni...@fedoraproject.org> wrote:
> > 1) easier to test upgrades between Fedora versions
> >
> > As each Fedora major release may have in updates only security and
> critical
> > fixes and ABI (libraries SONAME changes) will be completely locked down
> it
> > will be easier work on a set of test units testing upgrades on top of
> > different sets of installed packages.
>
> Doesn't relate to packages which rarely see an update if at all
>
But *why* asking for troubles and really not keep as REALLY stable what
STABLE in distro releases SHOULD be? (even +updates)
I'm really sure that there are much more interesting things touch ABI up to
the EOS. Isn't it?
> 2) more people (packagers and regular end users) will be focused on
> testing
> > of latest Fedora version
>
> Do you have some results of a testing or you just assume this would be the
> case
>
Yes. Out of my 7 years working and leading PLD more than half I've spend
working on two PLD version (const devel and stable).
All necessary changes ni CVS have been kept on ***branches***. Today with
git it will be way easier than +10 years ago.
Is it enough?
Minimal overhead on maintaining all packages where in only two flat CVS
modules SOURCES and SPECS.
I'm not suggesting to repeat this because current Fedora git model is good
enough with additional ACLs control.
Remember that it was only few of us actively working every day on whole
distribution.
Even with so imaginable small man/hours resources we been able to punch
first 3K (without perl/php packages) source packages barrier when at this
time in RH was still IIRC still less than 1k.
When poldek was in first quite usable version (credits to Paweł Gajda)
quite quickly was possible to reach high level of rpm dependencies
consistency or few types of conflicts checked on top of really wide/broad
set of packages without tying to test everything.
Common and strictly guarded spec files indentation/format was FUNDAMENTAL
to reach very high level of replace ability on maintaining already CLEAN
specs files.
Will divert here (again as I'm quite often doing) to specs
format/indentation ..
When only recently I've spotted that core Fedora packages (those which are
in RHEL) are maintained by (probably) ONLY RH employees (even as secondary
admins), and we are talking about at maybe few hundredths people strong
team I'm 100% sure why FPG parts about "common specs format" are only empty
wish.
Why? Because communication factor grows with factorial number of people.
In so big team always will be someone who "I didn't know that is is now new
rule" or "I don't like this".
Those people are probably quite often are so busy that they are "running
with empty barrows on transporting bricks between place where truck
unloaded bricks and construction place because they always says that we are
so busy that we don't have time to load and unload those bricks" (old
communist time joke).
Because many specs are really complicated because it was no proper peer
review as result some isolated woks on packages.
This at the end created highly over complicated (by what was dine in the
past). This on next level as consequence caused probably as well some
number of bad cases on moving packages between people. And finally decision
"we cannot loose straight control on this core Fedora distribution packages
because only few on this world knows how to maintan each package".
This caused as result limited level of Fedora<->RH trust.
No .. I know that if it is true it is pure bollocks. When I was working PLD
I've been controlling ALL packages. Everyone of us in PLD was able to do
this using only "sense of broken patterns". Each day review new changes
took me only up to 1h. Rest of the free time I've been spending on adding
second the same batch (if not more) my own changes.
Yes, I'm good on finding those "broken patterns" and in PLD I was only one
and this is why (ONLY) PLD started going down when other people in PLD
started thinking on monetize whole project (all without telling me even
single word about this).
Yea I was young, naive and still idealist .. who did't?
What I'm trying to say is that IMO I probably know why up to now it was not
possible to introduce common indentation in RH and Fedora as well.
(bad news)
I know probably as well why now it will be even harder.
.. because (as result)
a) some people may loose some responsibilities
b) it will be no more "my little precious" spec files
c) some people by this may loose some part of they "uniqueness"
.
.
Choose something ..
In other words on this area way bigger some "human/ego" obstacles than all
those technical ones. Am I close?
If not .. try to think why (probably as first but up to now not last) PLD
was able to attract attract some people with quite basic technical skills
when after few year we reached things which even RH with thousands
technical employees is not able after +10 years still is not able to reach.
More .. even with few time more Fedora guys still many things are
unreachable.
So why in PLD was possible to so high levels? Are we talking about some
freaks/geniuses? Of course not ..
We been spending much more time (proportionally) as packagers on improve
our METHODOLOGIES ..
Please tell me that I'm not enough close to REAL core ..
(I'm not afraid to say publicly "I was wrong" and apologize all RH hard
working employees if I'm wrong)
If I'm really wrong just please tell me why it is like this (without
wrapping this into the cotton) .. why it was not possible to standardize
so simple/basic thing in Fedora and RH as well?
However if I'm right no one from RH should feel guilty and only recall old
Roman sentence "errare humanum est, sed in errare perseverare diabolicum".
Nevertheless, I'm not kind of idiot who is trying to fight and I know that
even "Hercules is a wimp when he must fight against heaps of enemies"
(Latin original is way shorter "nec Hercules contra plures")
No, I'm only a guy who most of his life spend looking for right "why?"
questions.
Guy who knows that only knowledge of correct question is causing almost
instantly find right answer.
Yes, I know that I'm exposing myself quite often on lack of understanding
but that is only the cost.
Cost which I'm (always) ready to pay .. to have much higher
yield/speed/whatever.
OK .."Ab ovo" ..
(I promise .. to the end of this email will be only about technicalities)
Second PLD component was "builder" script which was used on downloading
specs and necessary, tagging, retagging, build packages, committing,
deleting and send build request to distro build systems .. etc.
Partially it is analog of the fedpkg but it was self constrained, single
file pure POSIX SH script which had been using only awk, sed, rpm/rpmbuild
and cvs command.
I must say that when I saw first time number of fedpkg dependencies I was
really shocked that what I've (mostly) done in few tenths KB SH script in
Fedora has been pumped to size which is possible to see.
OK .. fedpkg is doing few more things than PLD builder, but I'm sure that
using SH + maybe few new commands still it is possible to write such tool
in maybe 2-3 times bigger SH script.
Look on --help output:
Usage: builder [-D|--debug] [-V|--version] [--short-version] [-a|--as_anon]
[-b|-ba|--build]
[-bb|--build-binary] [-bs|--build-source] [-bc] [-bi] [-bl]
[-u|--try-upgrade]
[{-cf|--cvs-force}] [{-B|--branch} <branch>] [{-d|--cvsroot} <cvsroot>]
[-g|--get] [-h|--help] [--http] [{-l|--logtofile} <logfile>]
[-m|--mr-proper]
[-q|--quiet] [--date <yyyy-mm-dd> [-r <cvstag>] [{-T|--tag <cvstag>]
[-Tvs|--tag-version-stable] [-Ts|--tag-stable] [-Tv|--tag-version]
[{-Tp|--tag-prefix} <prefix>] [{-tt|--test-tag}] [--use-greed-sources]
[-nu|--no-urls] [-v|--verbose] [--opts <rpm opts>] [--short-circuit]
[--show-bconds] [--with/--without <feature>] [--define <macro> <value>]
<package>[.spec][:cvstag]
-5, --update-md5 - update md5 comments in spec, implies -nd -ncs
-a5, --add-md5 - add md5 comments to URL sources, implies -nc -nd -ncs
-n5, --no-md5 - ignore md5 comments in spec
-D, --debug - enable builder script debugging mode,
-debug - produce rpm debug package (same as --opts -debug)
-V, --version - output builder version string
--short-version - output builder short version
-a, --as_anon - get files via pserver as c...@cvs.pld-linux.org,
-b, -ba, --build - get all files from CVS repo or HTTP/FTP and build
package
from <package>.spec,
-bb, --build-binary - get all files from CVS repo or HTTP/FTP and build
binary
only package from <package>.spec,
-bp, --build-prep - execute the %prep phase of <package>.spec,
-bc - execute the %build phase of <package>.spec,
-bi - execute the %install phase of <package>.spec
-bl - execute the %files phase of <package>.spec
-bs, --build-source - get all files from CVS repo or HTTP/FTP and only pack
them into src.rpm,
--short-circuit - short-circuit build
-B, --branch - add branch
-c, --clean - clean all temporarily created files (in BUILD,
SOURCES,
SPECS and $RPM_BUILD_ROOT and CVS/Entries) after
rpmbuild commands.
-m, --mr-proper - clean all temporarily created files (in BUILD,
SOURCES,
SPECS and $RPM_BUILD_ROOT and CVS/Entries). Doesn't run
any rpm building.
-cf, --cvs-force - use -F when tagging (useful when moving branches)
-d <cvsroot>, --cvsroot <cvsroot>
- setup $CVSROOT,
--define <macro> <value>
- define a macro <macro> with value <value>,
--alt_kernel <kernel>
- same as --define 'alt_kernel <kernel>'
--nodeps - rpm won't check any dependences
-g, --get - get <package>.spec and all related files from CVS repo
or HTTP/FTP,
-h, --help - this message,
--http - use http instead of ftp,
-l <logfile>, --logtofile <logfile>
- log all to file,
-nc, --no-cvs - don't download sources from CVS, if source URL is
given,
-ncs, --no-cvs-specs
- don't check specs in CVS
-nd, --no-distfiles - don't download from distfiles
-nm, --no-mirrors - don't download from mirror, if source URL is given,
-nu, --no-urls - don't try to download from FTP/HTTP location,
-ns, --no-srcs - don't download Sources
-ns0, --no-source0 - don't download Source0
-nn, --no-net - don't download anything from the net
-pm, --prefer-mirrors - prefer mirrors (if any) over distfiles for SOURCES
--no-init - don't initialize builder paths (SPECS and SOURCES)
-ske,
--skip-existing-files - skip existing files in get_files
--opts <rpm opts> - additional options for rpm
-q, --quiet - be quiet,
--date yyyy-mm-dd - build package using resources from specified CVS date,
-r <cvstag>, --cvstag <cvstag>
- build package using resources from specified CVS tag,
-A - build package using CVS resources as any sticky
tags/date/kopts being reset.
-R, --fetch-build-requires
- fetch what is BuildRequired,
-RB, --remove-build-requires
- remove all you fetched with -R or
--fetch-build-requires
remember, this option requires confirmation,
-FRB, --force-remove-build-requires
- remove all you fetched with -R or
--fetch-build-requires
remember, this option works without confirmation,
-sd, --source-distfiles - list sources available from distfiles (intended
for offline
operations; does not work when Icon field is present
but icon file is absent),
-sc, --source-cvs - list sources available from CVS
-sdp, --source-distfiles-paths - list sources available from distfiles -
paths relative to distfiles directory (intended for
offline
operations; does not work when Icon field is present
but icon file is absent),
-sf, --source-files - list sources - bare filenames (intended for offline
operations; does not work when Icon field is present
but icon file is absent),
-sp, --source-paths - list sources - filenames with full local paths
(intended for
offline operations; does not work when Icon field is
present
but icon file is absent),
-su, --source-urls - list urls - urls to sources and patches
intended for copying urls with spec with lots of
macros in urls
-T <cvstag> , --tag <cvstag>
- add cvs tag <cvstag> for files,
-Tvs, --tag-version-stable
- add cvs tags STABLE and NAME-VERSION-RELEASE for
files,
-Ts, --tag-stable
- add cvs tag STABLE for files,
-Tv, --tag-version
- add cvs tag NAME-VERSION-RELEASE for files,
-Tp, --tag-prefix <prefix>
- add <prefix> to NAME-VERSION-RELEASE tags,
-tt, --test-tag <prefix>
- fail if tag is already present,
-ir, --integer-release-only
- allow only integer and snapshot releases
-v, --verbose - be verbose,
-u, --try-upgrade - check version, and try to upgrade package
-un, --try-upgrade-with-float-version
- as above, but allow float version
--use-greed-sources
- try download source from tag head if don't find it in
current tag
-U, --update - refetch sources, don't use distfiles, and update md5
comments
-Upi, --update-poldek-indexes
- refresh or make poldek package index files.
-np, --nopatch <patchnumber>
- don't apply <patchnumber>
--show-bconds - show available conditional builds, which can be used
- with --with and/or --without switches.
--show-bcond-args - show active bconds, from ~/.bcondrc. this is used by
./repackage.sh script. in other words, the output is
parseable by scripts.
--with/--without <feature>
- conditional build package depending on
%_with_<feature>/
%_without_<feature> macro switch. You may now use
--with feat1 feat2 feat3 --without feat4 feat5 --with
feat6
constructions. Set GROUP_BCONDS to yes to make use of
it.
--target <platform>, --target=<platform>
- build for platform <platform>.
--init-rpm-dir - initialize ~/rpm directory structure
All -b and -t options are present here because everything started when rpm
command and not like today rpmbuild command was in use.
As you see whoever knew already how to use rpm/build almost instantly will
be able to use "builder".
As you see typical "don't move, improve" approach was used.
This is why I'm using fedpkg ONLY when must because there is no other
possible way :)
This is kind of philosophically different approach.
-----
I want only point on:
-u, --try-upgrade - check version, and try to upgrade package
-un, --try-upgrade-with-float-version
- as above, but allow float version
Very interesting function which have been using Source URLs [1] to detect
is available new version tarballs -> inject into current spec new version
str -> download new tarball -> rebuild package -> commit change ->
(re)label necessary resources in CVS.
[1] we have been spending significant part of the time to keep all those
URLs in up-to-date state.
One detail about some context ..
Few days ago I've been upgrading zabbix (3.4.5->3.4.6). With upgrade
package I've done it in single line (still I have it in shell history).
$ cd ~/rpmbuild/SPECS; sed -i 's/\(Version:.\( *\|\t*\)\)\(.*\)/\13.4.6/'
zabbix.spec; wget -P ../SOURCES $(rpmbuild -bp --define "prep %dump"
zabbix.spec 2>&1 | awk '/SOURCEURL0/ {print $3}'); rpmbuild -ba --quiet
zabbix.spec; sudo rpm -Fvh ../RPMS/x86_64/zabbix*
Because I was 100% sure that it will be OK I've uses ";" between all
commands
So it is nothing more than simple demonstration of some possibilities ..
1) This oneliner is simple shows how to use rpmbuild as specs PARSER. Still
not to many people knows that rpmbuild internal parser can be used as
external tool
By above want I want to say?
in case fedpkg maybe it is good moment to use similar approach to introduce
in Fedora (semi) AUTOMATED UPDATES?
Some packages are so well maintained by source code maintainers that in
Ferdora packages there is no even single patch.
Flow of some perl, php or other scripting language packages every day is
sometimes quite high and all those packages are very small/simple.
Why waste so much time on update those "better" (from exact point of view)
packages <many_big_question_marks>
Definitely in context of such "better" packages would be IMO possible to
introduce (semi) automated updates.
I know that even among those "core/RH packages" there a lot of ..
(do you see next "human type obstacle"?
"You <censored>. This can make some of us .. jobless !^@#$% or at least
unhappy because we will >>nothing to farm<<"
My answer will be "if you want to farm something it is a lot of game
platforms around")
This "semi" automation may mean in many cases that package maintainer will
only need to review build log and say "That looks OK .. please finish this"
-> (click/tap), and after some time packager receives message "causa finita
est".
-----------
Second:
--show-bconds - show available conditional builds, which can be used
- with --with and/or --without switches.
--show-bcond-args - show active bconds, from ~/.bcondrc. this is used by
./repackage.sh script. in other words, the output is
parseable by scripts.
Which should IMO better talk to many people saying/believing that "%bconds
are bad!! and I (mighty packager) do this better using my own %ifology (C)"
My answer always will be "No, you did't!!!"
I don't need to even move to prove this .. proof is in (to) maaany Fedora
specs.
----------------
And last 3rd:
-5, --update-md5 - update md5 comments in spec, implies -nd -ncs
-a5, --add-md5 - add md5 comments to URL sources, implies -nc -nd -ncs
-n5, --no-md5 - ignore md5 comments in spec
Quite early we started storing in commented lines md5 check sums of source
tarballs.
it was used in form of lines:
# Source0-md5: <md5_hash>
Those commented lines and "builder as rpm/rpmbuild" extender allowed use
commented lines to implement new features without touch rpm source code.
Maybe it is now good time to add tokens like
{Patch,Source}<num>-{md5,sha256} to the official spec syntax (not
commented)?
Why to use is probably obvious (?)
If still not ..
it allows easier detect that some public source tarballs file servers have
been compromised.
For example Solaris native packages build infrastructure will not allow you
even START build process if it will find that that stored in Makefile hash
is different than this one which is locally available fs as tar ball when
you will bump package version -> download new tarball -> start build.
Do I need to answer on the question why Solaris guys are using such
restricted "hedgehogs during the sex" approach?
I think that at least few people during reading above one eyebrow moved up
.. just slightly.
> Simple more eyeballs using exact latest Fedora than more likely that after
> > hitting sometimes small issue it will be reported to BTS.
> > As consequence RH people making own snapshot to start working next major
> EL
> > version may expect that they will have fewer issues to resolve after
> making
> > such snapshot.
> >
> > 3) fewer packagers will be spending time on backporting some changes
>
> Doesn't relate to packages which rarely see an update
>
Really? More people looking on each change (because they will be not doing
things which is possible to automate may hurt Fedora quality????
hmm .. maybe .. hmm .. if .. "NO, this cannot be truth!"
After sleeping like a baby last nigh today I think that probably know how
may look step by step whole backporting process.
Lets assume that someone is doing the change on the master branch ..
1) on top of normal set of operations using fedpkg and final successful
build new version/release of the package this tool or even Fedora build
infrastructure detects that "aaaa .. this package has el6 and/or el7 labels
.. got you :-)"
2) using positive result of above -> sends email (or chat msg or creates
ticket) to packager and/or wider audience that it is something to review to
backport.
What may happen here next?
2a) ignore signal/close ticket because change has nothing to do with EPEL
2b) try to backport modification which has been committed to the master
Here using git such change can be commuted. If it is conflict it will be
necessary to resolve it manually.
This could be done as well by apply whole patch against el6, el7 branches
without if resolve of the conflict could be to difficult.
3) build EPEL package
4) clean signal/state that exact package has something to review and
backport
5) create every day stats/graphs/list of {fame,shame} ..
Does it may work?
What do you think (yes you anon reader) ..
IMO yes!!!
***********************************************
(AGAIN: try ONLY to think that keeping absolute cleanness, max readability
and simplicity of all specs IMO still is worth of even small overheads.
Only this will open Fedora/RHEL few new rooms to which now is not possible
to enter).
***********************************************
kloczek
--
Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
