Re: Current Rawhide problems you might want to know about

[Matthew Miller]

On Wed, Oct 14, 2015 at 08:53:43AM -0400, Josh Boyer wrote:
> >> How do I tell?  Anyway I didn't take any explicit step to enable it.
> >> The machine has just been 'dnf update'-d to Rawhide.
> > As I understand it, the plan was to switch it on in rawhide, but I'm
> No.

Thanks — I definitely remember discussion about that but apparently
missed the outcome.


-- 
Matthew Miller

Fedora Project Leader
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Richard W.M. Jones]

On Tue, Oct 13, 2015 at 05:31:04PM -0400, Matthew Miller wrote:
> On Tue, Oct 13, 2015 at 10:21:09PM +0100, Richard W.M. Jones wrote:
> > > Is it DNS? That's my go-to blame victim for ssh delays.
> > I think it's dbus actually:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1271394#c3
> 
> Is kdbus enabled?

How do I tell?  Anyway I didn't take any explicit step to enable it.
The machine has just been 'dnf update'-d to Rawhide.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Josh Boyer]

On Wed, Oct 14, 2015 at 8:06 PM, Ben Boeckel  wrote:
> On Wed, 14 Oct, 2015 at 12:53:43 GMT, Josh Boyer wrote:
>> The kernel module is included in the rawhide (and only rawhide)
>> kernels
>
> Would it be possible to enable it for the kernels from the
> fedora-rawhide-kernel-nodebug repository?

The module is already included in that, as those are build from rawhide sources.

josh
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Ben Boeckel]

On Wed, 14 Oct, 2015 at 12:53:43 GMT, Josh Boyer wrote:
> The kernel module is included in the rawhide (and only rawhide)
> kernels

Would it be possible to enable it for the kernels from the
fedora-rawhide-kernel-nodebug repository?

Thanks,

--Ben

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Ben Boeckel]

On Wed, Oct 14, 2015 at 20:11:08 -0400, Josh Boyer wrote:
> The module is already included in that, as those are build from
> rawhide sources.

Indeed; I had missed the .xz extension when searching.

Thanks,

--Ben
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Matthew Miller]

On Wed, Oct 14, 2015 at 10:10:54AM +0100, Richard W.M. Jones wrote:
> > Is kdbus enabled?
> How do I tell?  Anyway I didn't take any explicit step to enable it.
> The machine has just been 'dnf update'-d to Rawhide.

As I understand it, the plan was to switch it on in rawhide, but I'm
not sure what happens if you just upgrade like that. You should see
kdbus in /proc/cmdline - and no system dbus process. I think it'd be an
interesting exercise to turn it off if it's on, and on if it it's off.
:)

-- 
Matthew Miller

Fedora Project Leader
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Richard W.M. Jones]

On Wed, Oct 14, 2015 at 08:49:18AM -0400, Matthew Miller wrote:
> On Wed, Oct 14, 2015 at 10:10:54AM +0100, Richard W.M. Jones wrote:
> > > Is kdbus enabled?
> > How do I tell?  Anyway I didn't take any explicit step to enable it.
> > The machine has just been 'dnf update'-d to Rawhide.
> 
> As I understand it, the plan was to switch it on in rawhide, but I'm
> not sure what happens if you just upgrade like that. You should see
> kdbus in /proc/cmdline - and no system dbus process. I think it'd be an
> interesting exercise to turn it off if it's on, and on if it it's off.
> :)

As Josh clarified, it's not enabled.

Really strangely - the bug has gone away.  I did not make any kind of
configuration change or install any package.  So there we go.  I left
the bug open and there are some better 'strace's in the bug, but I
suppose unless anyone else sees this soon we can close it.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Josh Boyer]

On Wed, Oct 14, 2015 at 8:49 AM, Matthew Miller
 wrote:
> On Wed, Oct 14, 2015 at 10:10:54AM +0100, Richard W.M. Jones wrote:
>> > Is kdbus enabled?
>> How do I tell?  Anyway I didn't take any explicit step to enable it.
>> The machine has just been 'dnf update'-d to Rawhide.
>
> As I understand it, the plan was to switch it on in rawhide, but I'm

No.

> not sure what happens if you just upgrade like that. You should see
> kdbus in /proc/cmdline - and no system dbus process. I think it'd be an
> interesting exercise to turn it off if it's on, and on if it it's off.

The kernel module is included in the rawhide (and only rawhide)
kernels, however it is not enabled by default.  A user needs to
explicitly specify kdbus=1 on their kernel cmdline to enable kdbus.
The reason being that we do not want to have rawhide default to
something that is not going to ship in the next release, and we are
not going to include the kdbus code in the kernel on a stable release
until it is merged upstream.

josh
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Richard W.M. Jones]

On Tue, Oct 13, 2015 at 09:11:59PM +0100, Richard W.M. Jones wrote:
> On Tue, Oct 13, 2015 at 08:58:13PM +0100, Richard W.M. Jones wrote:
> > * On x86_64, ssh logins take quite a long time - about 30 seconds.  I
> >   have no idea what component to assign this to, so I didn't file a
> >   bug about it yet.
> 
> Not just logins, 'sudo bash' too.

BZ is here:

https://bugzilla.redhat.com/show_bug.cgi?id=1271394

I have no idea how to debug this, or even what component to assign it
to.  Suggestions?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Richard W.M. Jones]

On Tue, Oct 13, 2015 at 08:58:13PM +0100, Richard W.M. Jones wrote:
> * On x86_64, ssh logins take quite a long time - about 30 seconds.  I
>   have no idea what component to assign this to, so I didn't file a
>   bug about it yet.

Not just logins, 'sudo bash' too.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Current Rawhide problems you might want to know about

[Richard W.M. Jones]

A couple of things:

* systemd in aarch64 crashes on boot:
  https://bugzilla.redhat.com/show_bug.cgi?id=1271387

* On x86_64, ssh logins take quite a long time - about 30 seconds.  I
  have no idea what component to assign this to, so I didn't file a
  bug about it yet.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Chris Murphy]

Maybe strace will show where it gets hung up?


---
Chris Murphy
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Chris Murphy]

On Tue, Oct 13, 2015 at 3:31 PM, Matthew Miller
 wrote:
> On Tue, Oct 13, 2015 at 10:21:09PM +0100, Richard W.M. Jones wrote:
>> > Is it DNS? That's my go-to blame victim for ssh delays.
>> I think it's dbus actually:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1271394#c3
>
> Is kdbus enabled?

Any journal messages between those two time stamps where the gap is?

Maybe add a -v to strace and see if something else shows up. Nothing
else is forking, so...

-- 
Chris Murphy
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Matthew Miller]

On Tue, Oct 13, 2015 at 08:58:13PM +0100, Richard W.M. Jones wrote:
> * On x86_64, ssh logins take quite a long time - about 30 seconds.  I
>   have no idea what component to assign this to, so I didn't file a
>   bug about it yet.

Is it DNS? That's my go-to blame victim for ssh delays.

-- 
Matthew Miller

Fedora Project Leader
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Matthew Miller]

On Tue, Oct 13, 2015 at 10:21:09PM +0100, Richard W.M. Jones wrote:
> > Is it DNS? That's my go-to blame victim for ssh delays.
> I think it's dbus actually:
> https://bugzilla.redhat.com/show_bug.cgi?id=1271394#c3

Is kdbus enabled?

-- 
Matthew Miller

Fedora Project Leader
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Richard W.M. Jones]

On Tue, Oct 13, 2015 at 03:00:27PM -0600, Chris Murphy wrote:
> Maybe strace will show where it gets hung up?

Tried it of course - strace doesn't work across the setuid boundary.

Maybe I can attach a trace tool (which?) to sshd (how?) and find out
where it is spending its time.  This stuff should be easier ...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Richard W.M. Jones]

On Tue, Oct 13, 2015 at 05:16:17PM -0400, Matthew Miller wrote:
> On Tue, Oct 13, 2015 at 08:58:13PM +0100, Richard W.M. Jones wrote:
> > * On x86_64, ssh logins take quite a long time - about 30 seconds.  I
> >   have no idea what component to assign this to, so I didn't file a
> >   bug about it yet.
> 
> Is it DNS? That's my go-to blame victim for ssh delays.

I think it's dbus actually:

https://bugzilla.redhat.com/show_bug.cgi?id=1271394#c3

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Chris Murphy]

19844 22:16:16.857460 ppoll([{fd=8, events=POLLIN}], 1, {24,
999885000}, NULL, 8) = 0 (Timeout)
19844 22:16:41.864999
open("/usr/share/locale/en_GB.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) =
-1 ENOENT (No such file or directory)



That's a huge gap while nothing at all is showing up in the strace.


---
Chris Murphy
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Chris Murphy]

On Tue, Oct 13, 2015 at 3:37 PM, Chris Murphy  wrote:
> On Tue, Oct 13, 2015 at 3:31 PM, Matthew Miller
>  wrote:
>> On Tue, Oct 13, 2015 at 10:21:09PM +0100, Richard W.M. Jones wrote:
>>> > Is it DNS? That's my go-to blame victim for ssh delays.
>>> I think it's dbus actually:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1271394#c3
>>
>> Is kdbus enabled?
>
> Any journal messages between those two time stamps where the gap is?
>
> Maybe add a -v to strace and see if something else shows up. Nothing
> else is forking, so...

OK so this is like throwing spaghetti, but maybe valgrind, something like:


valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes
-v sudo bash


Of course that'll almost certainly lead you down a debug package
installation rabbit hole for a period of time.


-- 
Chris Murphy
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Current Rawhide problems you might want to know about

[Alek Paunov]

On 14.10.2015 00:13, Richard W.M. Jones wrote:

On Tue, Oct 13, 2015 at 03:00:27PM -0600, Chris Murphy wrote:

Maybe strace will show where it gets hung up?


Tried it of course - strace doesn't work across the setuid boundary.

Maybe I can attach a trace tool (which?) to sshd (how?) and find out
where it is spending its time.  This stuff should be easier ...



Sysdig [1] logs all (or some if filter is specified) syscalls via kernel 
module (in binary format like tshark):


sysdig -w /tmp/all.scap
sysdig -r /tmp/all.scap

Kind regards,
Alek

[1] http://www.sysdig.org/install/

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct