Re: [gentoo-dev] git-r3: initial draft for review [v2]

[Michał Górny]

Dnia 2013-09-01, o godz. 16:49:34
William Hubbs willi...@gentoo.org napisał(a):

 On Sat, Aug 31, 2013 at 10:48:32PM +0200, Michał Górny wrote:
  Dnia 2013-08-31, o godz. 11:26:30
  Ulrich Mueller u...@gentoo.org napisał(a):
  
On Sat, 31 Aug 2013, Michał Górny wrote:
   
And time for a small update. 
   
   In git-r3_checkout:
   
   git --no-pager diff --color --stat \
   ${old_commit_id}..${new_commit_id}
   
   I'd rather omit the --color option, otherwise log files will contain
   escape sequences.
  
  I'd rather leave it. The diff is more for pretty-printing anyway, it
  shouldn't really matter in the logs.
 
 Please don't. I also do not want escape sequences in log files.

Ok, '--color' removed. However, I think we should work something out to
get both parties satisfied. Maybe portage feature or a tool to 'uncruft'
logs from escape sequences since many people actually benefit from them.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] [RFC] git.eclass, git-2.eclass... git-r1.eclass?

[William Hubbs]

On Sun, Sep 01, 2013 at 08:58:07PM -0400, James Cloos wrote:
  TW == Tom Wijsman tom...@gentoo.org writes:
 
 TW Also, please just call it git-3.eclass as it is a major change; any
 TW other form of naming will introduce confusion (eg. -r1  -2), also we
 TW probably shouldn't change git-2.eclass as even when masked it doesn't
 TW seem like a good thing to break its current API and documentation as
 TW well as what actually works in the Portage tree.
 
 +1 on all of that.  git-3 is a better name than using -r1.
 
 And leave git-2 there for at /least/ a year.  There are a LOT of out of
 tree git-2 users.

The last time I checked, out of tree eclass users are not a concern for
how long we keep old eclasses in the tree. We only keep them until we
are sure there are no in tree users.

Thanks,

William


signature.asc
Description: Digital signature

[gentoo-dev] rfc: escape sequences in logs

[William Hubbs]

On Mon, Sep 02, 2013 at 01:33:19PM +0200, Michał Górny wrote:
 Dnia 2013-09-01, o godz. 16:49:34
 William Hubbs willi...@gentoo.org napisał(a):
 
  On Sat, Aug 31, 2013 at 10:48:32PM +0200, Michał Górny wrote:
   Dnia 2013-08-31, o godz. 11:26:30
   Ulrich Mueller u...@gentoo.org napisał(a):
   
 On Sat, 31 Aug 2013, Michał Górny wrote:

 And time for a small update. 

In git-r3_checkout:

git --no-pager diff --color --stat \
${old_commit_id}..${new_commit_id}

I'd rather omit the --color option, otherwise log files will contain
escape sequences.
   
   I'd rather leave it. The diff is more for pretty-printing anyway, it
   shouldn't really matter in the logs.
  
  Please don't. I also do not want escape sequences in log files.
 
 Ok, '--color' removed. However, I think we should work something out to
 get both parties satisfied. Maybe portage feature or a tool to 'uncruft'
 logs from escape sequences since many people actually benefit from them.

All,

I'm starting a new thread on this, because I think it might warrant some
discussion.

I can see why someone might want to use escape codes for color displays,
etc. However, imo, escape codes do not belong in log files.

mgorny says many people benefit from having escape codes in log
files, but I see no benefit from it, and I don't like going through
build.log because of them. If you load a build.log into an editor, the
escape sequences are basically trash characters that get in the way.

Another consideration is if someone puts messages from a build.log
directly in a bug and the messages contain escape codes, this will crash
things like pybugz because bugzilla doesn't filter out the escape
character.

Does anyone else have an opinion on this?

Thanks,

William



signature.asc
Description: Digital signature

Re: [gentoo-dev] rfc: escape sequences in logs

[Michał Górny]

Dnia 2013-09-02, o godz. 14:21:52
William Hubbs willi...@gentoo.org napisał(a):

 On Mon, Sep 02, 2013 at 01:33:19PM +0200, Michał Górny wrote:
  Dnia 2013-09-01, o godz. 16:49:34
  William Hubbs willi...@gentoo.org napisał(a):
  
   On Sat, Aug 31, 2013 at 10:48:32PM +0200, Michał Górny wrote:
Dnia 2013-08-31, o godz. 11:26:30
Ulrich Mueller u...@gentoo.org napisał(a):

  On Sat, 31 Aug 2013, Michał Górny wrote:
 
  And time for a small update. 
 
 In git-r3_checkout:
 
 git --no-pager diff --color --stat \
 ${old_commit_id}..${new_commit_id}
 
 I'd rather omit the --color option, otherwise log files will contain
 escape sequences.

I'd rather leave it. The diff is more for pretty-printing anyway, it
shouldn't really matter in the logs.
   
   Please don't. I also do not want escape sequences in log files.
  
  Ok, '--color' removed. However, I think we should work something out to
  get both parties satisfied. Maybe portage feature or a tool to 'uncruft'
  logs from escape sequences since many people actually benefit from them.
 
 All,
 
 I'm starting a new thread on this, because I think it might warrant some
 discussion.
 
 I can see why someone might want to use escape codes for color displays,
 etc. However, imo, escape codes do not belong in log files.

Well, colors are not the only thing that uses escape sequences. They
are used pretty often for various kinds of progress reporting --
percentages, progress bars. Many tools simply don't implement any other
kind of progress reporting, so sometimes it's either escape sequences
or long waiting with no signs of life.

 mgorny says many people benefit from having escape codes in log
 files, but I see no benefit from it, and I don't like going through
 build.log because of them. If you load a build.log into an editor, the
 escape sequences are basically trash characters that get in the way.

I said people benefit from having them output during the build process.
Logs are a different thing, that's why I suggested having a feature
that would remove those from output when generating logs.

 Another consideration is if someone puts messages from a build.log
 directly in a bug and the messages contain escape codes, this will crash
 things like pybugz because bugzilla doesn't filter out the escape
 character.

That is a bug in pybugz and not an argument, you know.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] rfc: escape sequences in logs

[William Hubbs]

On Mon, Sep 02, 2013 at 09:41:28PM +0200, Michał Górny wrote:
 Dnia 2013-09-02, o godz. 14:21:52
 William Hubbs willi...@gentoo.org napisał(a):
 
  On Mon, Sep 02, 2013 at 01:33:19PM +0200, Michał Górny wrote:
   Dnia 2013-09-01, o godz. 16:49:34
   William Hubbs willi...@gentoo.org napisał(a):
   
On Sat, Aug 31, 2013 at 10:48:32PM +0200, Michał Górny wrote:
 Dnia 2013-08-31, o godz. 11:26:30
 Ulrich Mueller u...@gentoo.org napisał(a):
 
   On Sat, 31 Aug 2013, Michał Górny wrote:
  
   And time for a small update. 
  
  In git-r3_checkout:
  
  git --no-pager diff --color --stat \
  ${old_commit_id}..${new_commit_id}
  
  I'd rather omit the --color option, otherwise log files will contain
  escape sequences.
 
 I'd rather leave it. The diff is more for pretty-printing anyway, it
 shouldn't really matter in the logs.

Please don't. I also do not want escape sequences in log files.
   
   Ok, '--color' removed. However, I think we should work something out to
   get both parties satisfied. Maybe portage feature or a tool to 'uncruft'
   logs from escape sequences since many people actually benefit from them.
  
  All,
  
  I'm starting a new thread on this, because I think it might warrant some
  discussion.
  
  I can see why someone might want to use escape codes for color displays,
  etc. However, imo, escape codes do not belong in log files.
 
 Well, colors are not the only thing that uses escape sequences. They
 are used pretty often for various kinds of progress reporting --
 percentages, progress bars. Many tools simply don't implement any other
 kind of progress reporting, so sometimes it's either escape sequences
 or long waiting with no signs of life.
 
  mgorny says many people benefit from having escape codes in log
  files, but I see no benefit from it, and I don't like going through
  build.log because of them. If you load a build.log into an editor, the
  escape sequences are basically trash characters that get in the way.
 
 I said people benefit from having them output during the build process.
 Logs are a different thing, that's why I suggested having a feature
 that would remove those from output when generating logs.

There is a NOCOLOR variable you can set in make.conf, but that would
require users to set it, and it also affects the display.

How does portage write build.log?

  Another consideration is if someone puts messages from a build.log
  directly in a bug and the messages contain escape codes, this will crash
  things like pybugz because bugzilla doesn't filter out the escape
  character.
 
 That is a bug in pybugz and not an argument, you know.

I said things like pybugz.

Bugzilla allowing control characters in the xml is the issue. The python
xmlrpc library raises an exception for malformed xml because of it, so
there isn't much pybugz, or anything that uses that library, can do
about it.

William



signature.asc
Description: Digital signature

Re: [gentoo-dev] rfc: escape sequences in logs

[Ulrich Mueller]

 On Mon, 2 Sep 2013, William Hubbs wrote:

 I can see why someone might want to use escape codes for color
 displays, etc. However, imo, escape codes do not belong in log
 files.

 mgorny says many people benefit from having escape codes in log
 files, but I see no benefit from it, and I don't like going through
 build.log because of them. If you load a build.log into an editor,
 the escape sequences are basically trash characters that get in the
 way.

 Another consideration is if someone puts messages from a build.log
 directly in a bug and the messages contain escape codes, this will
 crash things like pybugz because bugzilla doesn't filter out the
 escape character.

 Does anyone else have an opinion on this?

Log files should be plain text without any embedded font attributes or
other formatting.

I'd consider any tool as broken if it outputs escape sequences when
the output doesn't go to a terminal. (Unless such output was
explicitly asked for.)

Ulrich

Re: [gentoo-dev] rfc: escape sequences in logs

[Kent Fredric]

On 3 September 2013 09:22, Ulrich Mueller u...@gentoo.org wrote:

 I'd consider any tool as broken if it outputs escape sequences when
 the output doesn't go to a terminal. (Unless such output was
 explicitly asked for.)



However, what about when output is going to a terminal *and* a log file?

It seems smarter to output escape sequences in that case.

Additionally, although many people find escape sequences in log files
useless, I find them somewhat useful, for instance, if somebody uploads a
logfile with escape sequences in it, I can just run the file into less -R
or less -r depending on my mood and see the log file *Exactly* as the
user saw it.

I can see why you might not like it if you're reading it in a non-escape
aware editor, but I don't really use editors for reading log files, that
strikes me as doing something wrong.

And it is quite simple to remove escape sequences from log files if I
desire it.

using: app-text/ansifilter

 zcat
/var/log/portage/build/app-accessibility/at-spi2-core-2.6.3:20130825-143158.log.gz
| ansifilter

using perl and Term::ANSIColor ( standard issue with Perl itself )

zcat
/var/log/portage/build/app-accessibility/at-spi2-core-2.6.3:20130825-143158.log.gz
| perl -MTerm::ANSIColor=colorstrip -ple '$_ = colorstrip($_)'


-- 
Kent

[gentoo-dev] [PATCH] kernel-2.eclass: Add support for experimental genpatches.

[Tom Wijsman]

Hello everyone


Please only respond on the gentoo-dev ML.

Attached you will find a patch for kernel-2.eclass that adds support
for experimental genpatches; a response to the positive feedback on the
previous discussion [1], with subject Proper distribution integration
of kernel *-sources, patches and configuration.. We also came up with
solutions for the negative feedback to get the best from both worlds.

 [1]: http://thread.gmane.org/gmane.linux.gentoo.kernel/740
  which starts in message ID 20130701164149.131490f8@TOMWIJ-GENTOO

Its default functionality adds an USE flag which the users can use to
choose whether they want experimental patches or not; this behavior can
be further adjusted using three variables we provide, allowing the
ebuild writer to selectively decide how the patches are handled.

All possible combinations these variables as well as the USE flag can
be set in have been tested; so, in terms of functionality I think there
is nothing broken. mpagano and I have been using this during our recent
bumps as well; so, we are sure it doesn't break the default behavior.

The user can use both K_EXP_GENPATCHES_LIST and UNIPATCH_EXCLUDE to
respectively opt-in and opt-out control which experimental patches get
applied; this allows users to control them as they see fit.

Please note that we will check the patches to ensure they are guarded
by config checks; so, the sole reason the user has for doing the above
is to deal with possible collisions the user may have with user patches.

Please review the attached patch and let me know what you think...

Thank you in advance.


The rest of this mail are FAQ to attempt to answer common questions:

=== How are the genpatches affected? ===

As for genpatches, patches in the 5000-5099 range will be stored in an
experimental tarball; so, they are distributed separately and are not
part of the base and extra tarballs. If the user or ebuild writer does
not explicitly choose for experimental, it will thus not be fetched.

=== Are users that don't want experimental patches affected? ===

No, we will stay as stable and reliable as before; if users don't
explicitly opt in, they will not be affected by it on at least the
vanilla and gentoo sources. Other ebuild writers can see for themselves
if they want to defer from the defaults, but we encourage them not to.

Remember that with this change we try to deduplicate work; while doing
this, we do not force these patches on any user, there will be guards in
two places: The experimental USE flag and menuconfig. The new variables
in the eclass permit configuring which patches get applied as well.

=== How will the users be aware of this? ===

We plan to write a news article to document this as well as create a
sub project page explaining the experimental patches and everything
that comes along (their status, how to enable, pick them, etc ...).

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D
--- a/kernel-2.eclass
+++ b/kernel-2.eclass
@@ -40,7 +40,13 @@
 # K_DEFCONFIG			- Allow specifying a different defconfig target.
 #		  If length zero, defaults to defconfig.
 # K_WANT_GENPATCHES		- Apply genpatches to kernel source. Provide any
-# 		  combination of base and extras
+# 		  combination of base, extras or experimental.
+# K_EXP_GENPATCHES_PULL	- If set, we pull experimental regardless of the USE FLAG
+#		  but expect the ebuild maintainer to use K_EXP_GENPATCHES_LIST.
+# K_EXP_GENPATCHES_NOUSE	- If set, no USE flag will be provided for experimental;
+# 		  as a result the user cannot choose to apply those patches.
+# K_EXP_GENPATCHES_LIST	- A list of patches to pick from experimental to apply when
+# 		  the USE flag is unset and K_EXP_GENPATCHES_PULL is set.
 # K_GENPATCHES_VER		- The version of the genpatches tarball(s) to apply.
 #		  A value of 5 would apply genpatches-2.6.12-5 to
 #		  my-sources-2.6.12.ebuild
@@ -130,18 +136,32 @@
 	# respectively.  Handle this.
 
 	for i in ${K_WANT_GENPATCHES} ; do
-	if [[ ${KV_MAJOR} -ge 3 ]]; then
-		if [[ ${#OKV_ARRAY[@]} -ge 3 ]]; then
-			tarball=genpatches-${KV_MAJOR}.${KV_MINOR}-${K_GENPATCHES_VER}.${i}.tar.xz
+		if [[ ${KV_MAJOR} -ge 3 ]]; then
+			if [[ ${#OKV_ARRAY[@]} -ge 3 ]]; then
+tarball=genpatches-${KV_MAJOR}.${KV_MINOR}-${K_GENPATCHES_VER}.${i}.tar.xz
+			else
+tarball=genpatches-${KV_MAJOR}.${KV_PATCH}-${K_GENPATCHES_VER}.${i}.tar.xz
+			fi
 		else
-			tarball=genpatches-${KV_MAJOR}.${KV_PATCH}-${K_GENPATCHES_VER}.${i}.tar.xz
+			tarball=genpatches-${OKV}-${K_GENPATCHES_VER}.${i}.tar.xz
 		fi
-	else
-		tarball=genpatches-${OKV}-${K_GENPATCHES_VER}.${i}.tar.xz
-	fi
-	debug-print genpatches tarball: $tarball
-	GENPATCHES_URI=${GENPATCHES_URI} mirror://gentoo/${tarball}
-	UNIPATCH_LIST_GENPATCHES=${UNIPATCH_LIST_GENPATCHES} ${DISTDIR}/${tarball}
+
+		local use_cond_start= 

Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Tom Wijsman]

On Mon, 2 Sep 2013 14:21:52 -0500
William Hubbs willi...@gentoo.org wrote:

 I can see why someone might want to use escape codes for color
 displays, etc. However, imo, escape codes do not belong in log files.

They belong there so future display can remain colorful.

Why do they not belong there? What do people have to do who want them?

 mgorny says many people benefit from having escape codes in log
 files, but I see no benefit from it, and I don't like going through
 build.log because of them. If you load a build.log into an editor, the
 escape sequences are basically trash characters that get in the way.

They make some sections differ and therefore stand out; so, you get a
much faster impression of what you are looking at is what you want.

Why do you open them in an editor and not in a viewer?

 Does anyone else have an opinion on this?

There are definitely enough people that want them; so, why not have
them by default and strip them if you don't want them? The opposite,
adding escape codes where there are none; is a much harder thing to do.

Actually, I would love to see even more codes in build.log such that
they come more machine parseable; for instance, there is no indication
which process outputs a certain message, or whether the message
happened on stdout or stderr.

Such information is quite crucial, yet it is missing; it would be neat
if I could just grep the last stderr lines of the last process that's
not emerge or make, giving me exactly the gcc error I need to see. Once
I found that line, grepping context is easy.

Until then, I'm stuck with having to scroll up to that point; and while
I know -j1 helps with that, it would be crude to ignore non -j1 logs...

If you're trying to more efficiently parse logs; consider adding more
information instead, because dropping information does not really gain.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] rfc: escape sequences in logs

[Zac Medico]

On 09/02/2013 01:29 PM, William Hubbs wrote:
 On Mon, Sep 02, 2013 at 09:41:28PM +0200, Michał Górny wrote:
 Dnia 2013-09-02, o godz. 14:21:52
 William Hubbs willi...@gentoo.org napisał(a):

 On Mon, Sep 02, 2013 at 01:33:19PM +0200, Michał Górny wrote:
 Dnia 2013-09-01, o godz. 16:49:34
 William Hubbs willi...@gentoo.org napisał(a):

 On Sat, Aug 31, 2013 at 10:48:32PM +0200, Michał Górny wrote:
 Dnia 2013-08-31, o godz. 11:26:30
 Ulrich Mueller u...@gentoo.org napisał(a):

 On Sat, 31 Aug 2013, Michał Górny wrote:

 And time for a small update. 

 In git-r3_checkout:

 git --no-pager diff --color --stat \
 ${old_commit_id}..${new_commit_id}

 I'd rather omit the --color option, otherwise log files will contain
 escape sequences.

 I'd rather leave it. The diff is more for pretty-printing anyway, it
 shouldn't really matter in the logs.

 Please don't. I also do not want escape sequences in log files.

 Ok, '--color' removed. However, I think we should work something out to
 get both parties satisfied. Maybe portage feature or a tool to 'uncruft'
 logs from escape sequences since many people actually benefit from them.

 All,

 I'm starting a new thread on this, because I think it might warrant some
 discussion.

 I can see why someone might want to use escape codes for color displays,
 etc. However, imo, escape codes do not belong in log files.

 Well, colors are not the only thing that uses escape sequences. They
 are used pretty often for various kinds of progress reporting --
 percentages, progress bars. Many tools simply don't implement any other
 kind of progress reporting, so sometimes it's either escape sequences
 or long waiting with no signs of life.

 mgorny says many people benefit from having escape codes in log
 files, but I see no benefit from it, and I don't like going through
 build.log because of them. If you load a build.log into an editor, the
 escape sequences are basically trash characters that get in the way.

 I said people benefit from having them output during the build process.
 Logs are a different thing, that's why I suggested having a feature
 that would remove those from output when generating logs.
 
 There is a NOCOLOR variable you can set in make.conf, but that would
 require users to set it, and it also affects the display.
 
 How does portage write build.log?

It uses its PipeLogger class to read the output from a pipe, and write
to the log (and stdout if appropriate):


http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=blob_plain;f=pym/portage/util/_async/PipeLogger.py;hb=HEAD

So yes, it would be possible to do translations on-the-fly. However, the
event loop that handles the logging is currently single threaded, and
might become a bottleneck if it has to do lots of output processing
(especially for larger values of --jobs).
-- 
Thanks,
Zac

[gentoo-dev] Last rites: dev-games/neotools, dev-games/neoengine

[Chris Reffett]

# Chris Reffett creff...@gentoo.org (03 Sep 2013)
# Dead upstream, outstanding security bug #260956.
# Masked for removal in 30 days.
dev-games/neoengine
dev-games/neotools



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: escape sequences in logs

[Richard Yao]

On 09/02/2013 03:21 PM, William Hubbs wrote:
 All,
 
 I'm starting a new thread on this, because I think it might warrant some
 discussion.
 
 I can see why someone might want to use escape codes for color displays,
 etc. However, imo, escape codes do not belong in log files.
 
 mgorny says many people benefit from having escape codes in log
 files, but I see no benefit from it, and I don't like going through
 build.log because of them. If you load a build.log into an editor, the
 escape sequences are basically trash characters that get in the way.

I find looking at logs with escape characters to be more pleasant than
looking at them without escape characters. I admit that it is annoying
to view them in a web browser where the escape characters are not
parsed, but that is easily resolved at the terminal.

Just my two cents.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: escape sequences in logs

[Kent Fredric]

On 3 September 2013 16:17, Richard Yao r...@gentoo.org wrote:

 I admit that it is annoying
 to view them in a web browser where the escape characters are not
 parsed, but that is easily resolved at the terminal



You could plausibly also have a filter in bugzilla that detects escape
codes in the log, and adds a display link that routes to a page rendered
through `ansifilter -H` , similar to how we have a diff formatter for
patches.

I'm not sure the exact layers of glue required to make that viable in
bugzilla, but it doesn't seem impossible.


-- 
Kent

perl -e  print substr( \edrgmaM  SPA NOcomil.ic\\@tfrken\, \$_ * 3, 3 )
for ( 9,8,0,7,1,6,5,4,3,2 );

http://kent-fredric.fox.geek.nz