Re: [osv-dev] GDB stub

[Matias Vara]

On Wed, 17 Feb 2021 at 06:04, Waldek Kozaczuk  wrote:

> Hi Matias,
>
> Good to hear from you.
>
> Are you saying that Toro has already implemented a gdb stub and we can
> "steal" it (:-) or are you suggesting we work together to implement one? I
> am all for that.
>
>
Hahaha, I am currently implementing one but it is still a PoC. I just
started one month ago (see branch firfox#421) and I have managed to make
work most of the commands like break, continue, step, etc. I just step into
the conversation because I was very surprised that the QEMU gdbstub
built-in did not work well in a KVM guest so I wanted to warn you before
you go that path.


> Can you point to any documentation on how such a stub can be implemented?
> Also, does gdb require communication over tcp and then the guest must have
> the networking working? That would prevent early boot debugging.
>

I found the following document very interesting:
https://www.embecosm.com/appnotes/ean4/embecosm-howto-rsp-server-ean4-issue-2.html#sec_exchange_target_remote.
This explains the remote protocol that the gdbstub has to implement. I also
used the project https://github.com/mborgerson/gdbstub to have an idea. The
way I implemented is by using the serial port and then telling QEMU to
forward the serial port to the tcp:1234 (legacy serial port not
virtio-console). Then, you have just to execute gdb with "remote target
localhost:1234". However, this approach is very slow so I have to find a
way to improve it. The problem with early debugging could be if you want to
want to debug before you enable long mode. I think gdb wont be happy with
that. If this is not the case, early debugging could be done by just using
the legacy serial device.

I hope that helps.

Matias


>
> Regards,
> Waldek
>
> On Saturday, February 13, 2021 at 9:30:46 AM UTC-5 matia...@gmail.com
> wrote:
>
>> Hello everyone,
>>
>> I would like to step into this topic, maybe my experience may help and
>> also you may help me too. A couple of months ago, I figured that I can't
>> debug Toro KVM guests by using the QEMU built-in gdbstub. I tried hwbreak
>> but that did not work either. The execution was not stopping on the
>> breakpoints. The QEMU built-in gdbstub works perfectly fine when KVM is not
>> used. I think this is a limitation of the QEMU gdbstub for x86. With a bit
>> of debugging, I figured out that there were actually " KVM_EXIT_DEBUG" when
>> breakpoints were reached but control did not go back to the gdbstub (see
>> QEMU tracepoints). I investigated nothing further. I wanted to avoid
>> writing a gdbstub for Toro but I did not have another choice. At the same
>> time, this allows debugging the unikernel no matters the hypervisor.
>>
>> Regards, Matias.
>>
>> On Wed, 3 Feb 2021 at 00:18, Waldek Kozaczuk  wrote:
>>
>>> Currently, it is only possible to connect from gdb to a running instance
>>> of OSv on QEMU. But ideally, it would be nice to have a gdb stub in OSv
>>> that would let one connect from gdb to OSv running on any hypervisor.
>>>
>>> Per this fragment of this Wiki -
>>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv
>>>
>>> *"The fact that our gdb support works with the unit of cpus, not OSV
>>> threads, makes it convenient to debug OSV itself, but less convenient to
>>> debug application code running inside OSV. Particularly problematic is
>>> single stepping support which isn’t currently supported well: The problem
>>> is that when you try to look at a single OSV thread (see the next section),
>>> a single-step operation (“s” or “n”) makes one step inside the vcpu, not
>>> the osv thread*, and this step might include switching to a different
>>> thread. One way to work around this problem is to compile OSV with
>>> preemption disabled (not recommended, but can be a convenient trick in some
>>> debugging situations). *A better approach would have been to include a
>>> gdb stub into OSV itself, to allow gdb to see OSV threads directly and
>>> single-step them. We’ve made some progress in implementing such a stub, but
>>> it isn’t ready for prime-time yet.*"
>>>
>>> it looks like somebody may have implemented an initial version of the
>>> gdb stub. Is there a working patch of it somewhere?
>>>
>>> Regards,
>>> Waldek
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "OSv Development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to osv-dev+u...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/osv-dev/a3ca8d68-5994-4b51-94c4-539e2024f12dn%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@g

Re: [osv-dev] GDB stub

[Waldek Kozaczuk]

Hi Matias,

Good to hear from you.

Are you saying that Toro has already implemented a gdb stub and we can 
"steal" it (:-) or are you suggesting we work together to implement one? I 
am all for that.

Can you point to any documentation on how such a stub can be implemented? 
Also, does gdb require communication over tcp and then the guest must have 
the networking working? That would prevent early boot debugging.

Regards,
Waldek 

On Saturday, February 13, 2021 at 9:30:46 AM UTC-5 matia...@gmail.com wrote:

> Hello everyone, 
>
> I would like to step into this topic, maybe my experience may help and 
> also you may help me too. A couple of months ago, I figured that I can't 
> debug Toro KVM guests by using the QEMU built-in gdbstub. I tried hwbreak 
> but that did not work either. The execution was not stopping on the 
> breakpoints. The QEMU built-in gdbstub works perfectly fine when KVM is not 
> used. I think this is a limitation of the QEMU gdbstub for x86. With a bit 
> of debugging, I figured out that there were actually " KVM_EXIT_DEBUG" when 
> breakpoints were reached but control did not go back to the gdbstub (see 
> QEMU tracepoints). I investigated nothing further. I wanted to avoid 
> writing a gdbstub for Toro but I did not have another choice. At the same 
> time, this allows debugging the unikernel no matters the hypervisor. 
>
> Regards, Matias.   
>
> On Wed, 3 Feb 2021 at 00:18, Waldek Kozaczuk  wrote:
>
>> Currently, it is only possible to connect from gdb to a running instance 
>> of OSv on QEMU. But ideally, it would be nice to have a gdb stub in OSv 
>> that would let one connect from gdb to OSv running on any hypervisor.
>>
>> Per this fragment of this Wiki - 
>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv
>>
>> *"The fact that our gdb support works with the unit of cpus, not OSV 
>> threads, makes it convenient to debug OSV itself, but less convenient to 
>> debug application code running inside OSV. Particularly problematic is 
>> single stepping support which isn’t currently supported well: The problem 
>> is that when you try to look at a single OSV thread (see the next section), 
>> a single-step operation (“s” or “n”) makes one step inside the vcpu, not 
>> the osv thread*, and this step might include switching to a different 
>> thread. One way to work around this problem is to compile OSV with 
>> preemption disabled (not recommended, but can be a convenient trick in some 
>> debugging situations). *A better approach would have been to include a 
>> gdb stub into OSV itself, to allow gdb to see OSV threads directly and 
>> single-step them. We’ve made some progress in implementing such a stub, but 
>> it isn’t ready for prime-time yet.*"
>>
>> it looks like somebody may have implemented an initial version of the gdb 
>> stub. Is there a working patch of it somewhere?
>>
>> Regards,
>> Waldek
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "OSv Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to osv-dev+u...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/osv-dev/a3ca8d68-5994-4b51-94c4-539e2024f12dn%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/37acb542-8397-4e0f-8dad-9c7ee9003596n%40googlegroups.com.

Re: [osv-dev] GDB stub

[Matias Vara]

Hello everyone,

I would like to step into this topic, maybe my experience may help and also
you may help me too. A couple of months ago, I figured that I can't debug
Toro KVM guests by using the QEMU built-in gdbstub. I tried hwbreak but
that did not work either. The execution was not stopping on the
breakpoints. The QEMU built-in gdbstub works perfectly fine when KVM is not
used. I think this is a limitation of the QEMU gdbstub for x86. With a bit
of debugging, I figured out that there were actually " KVM_EXIT_DEBUG" when
breakpoints were reached but control did not go back to the gdbstub (see
QEMU tracepoints). I investigated nothing further. I wanted to avoid
writing a gdbstub for Toro but I did not have another choice. At the same
time, this allows debugging the unikernel no matters the hypervisor.

Regards, Matias.

On Wed, 3 Feb 2021 at 00:18, Waldek Kozaczuk  wrote:

> Currently, it is only possible to connect from gdb to a running instance
> of OSv on QEMU. But ideally, it would be nice to have a gdb stub in OSv
> that would let one connect from gdb to OSv running on any hypervisor.
>
> Per this fragment of this Wiki -
> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv
>
> *"The fact that our gdb support works with the unit of cpus, not OSV
> threads, makes it convenient to debug OSV itself, but less convenient to
> debug application code running inside OSV. Particularly problematic is
> single stepping support which isn’t currently supported well: The problem
> is that when you try to look at a single OSV thread (see the next section),
> a single-step operation (“s” or “n”) makes one step inside the vcpu, not
> the osv thread*, and this step might include switching to a different
> thread. One way to work around this problem is to compile OSV with
> preemption disabled (not recommended, but can be a convenient trick in some
> debugging situations). *A better approach would have been to include a
> gdb stub into OSV itself, to allow gdb to see OSV threads directly and
> single-step them. We’ve made some progress in implementing such a stub, but
> it isn’t ready for prime-time yet.*"
>
> it looks like somebody may have implemented an initial version of the gdb
> stub. Is there a working patch of it somewhere?
>
> Regards,
> Waldek
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/a3ca8d68-5994-4b51-94c4-539e2024f12dn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CA%2BdKekDtPQ%2BOfZZdiEt67wF%3DBx%3DD_C3gPJ3qYt%2Bqy06e1tC2Sw%40mail.gmail.com.

Re: [osv-dev] GDB stub

[Glauber Costa]

On Mon., Feb. 8, 2021, 3:00 a.m. Nadav Har'El,  wrote:

> On Wed, Feb 3, 2021 at 1:18 AM Waldek Kozaczuk 
> wrote:
>
>> Currently, it is only possible to connect from gdb to a running instance
>> of OSv on QEMU. But ideally, it would be nice to have a gdb stub in OSv
>> that would let one connect from gdb to OSv running on any hypervisor.
>>
>> Per this fragment of this Wiki -
>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv
>>
>> *"The fact that our gdb support works with the unit of cpus, not OSV
>> threads, makes it convenient to debug OSV itself, but less convenient to
>> debug application code running inside OSV. Particularly problematic is
>> single stepping support which isn’t currently supported well: The problem
>> is that when you try to look at a single OSV thread (see the next section),
>> a single-step operation (“s” or “n”) makes one step inside the vcpu, not
>> the osv thread*, and this step might include switching to a different
>> thread. One way to work around this problem is to compile OSV with
>> preemption disabled (not recommended, but can be a convenient trick in some
>> debugging situations). *A better approach would have been to include a
>> gdb stub into OSV itself, to allow gdb to see OSV threads directly and
>> single-step them. We’ve made some progress in implementing such a stub, but
>> it isn’t ready for prime-time yet.*"
>>
>> it looks like somebody may have implemented an initial version of the gdb
>> stub. Is there a working patch of it somewhere?
>>
>
> I have a vague recollection - it was many years ago - that Glauber (CCed)
> was the one who tried doing it.
> Glauber, if it was you, do you remember what you tried?
> If it wasn't you, sorry for the spam :-)
>

I don't think it was me. I have no memory of ever doing this.

But no need to be sorry, it's good to hear from you!

>
>
>> Regards,
>> Waldek
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "OSv Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to osv-dev+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/osv-dev/a3ca8d68-5994-4b51-94c4-539e2024f12dn%40googlegroups.com
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAA6-i6rAWgYqkCOPnGiQABpEfC%3DKqD1jCmmw82LEmrCNrHWyqA%40mail.gmail.com.

Re: [osv-dev] GDB stub

[Nadav Har'El]

On Wed, Feb 3, 2021 at 1:18 AM Waldek Kozaczuk  wrote:

> Currently, it is only possible to connect from gdb to a running instance
> of OSv on QEMU. But ideally, it would be nice to have a gdb stub in OSv
> that would let one connect from gdb to OSv running on any hypervisor.
>
> Per this fragment of this Wiki -
> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv
>
> *"The fact that our gdb support works with the unit of cpus, not OSV
> threads, makes it convenient to debug OSV itself, but less convenient to
> debug application code running inside OSV. Particularly problematic is
> single stepping support which isn’t currently supported well: The problem
> is that when you try to look at a single OSV thread (see the next section),
> a single-step operation (“s” or “n”) makes one step inside the vcpu, not
> the osv thread*, and this step might include switching to a different
> thread. One way to work around this problem is to compile OSV with
> preemption disabled (not recommended, but can be a convenient trick in some
> debugging situations). *A better approach would have been to include a
> gdb stub into OSV itself, to allow gdb to see OSV threads directly and
> single-step them. We’ve made some progress in implementing such a stub, but
> it isn’t ready for prime-time yet.*"
>
> it looks like somebody may have implemented an initial version of the gdb
> stub. Is there a working patch of it somewhere?
>

I have a vague recollection - it was many years ago - that Glauber (CCed)
was the one who tried doing it.
Glauber, if it was you, do you remember what you tried?
If it wasn't you, sorry for the spam :-)


> Regards,
> Waldek
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/a3ca8d68-5994-4b51-94c4-539e2024f12dn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CANEVyjs6jduwbHOr_MhoJqtq8%3DcctrT95qcWTDTxHHLpPtdqEw%40mail.gmail.com.

[osv-dev] GDB stub

[Waldek Kozaczuk]

Currently, it is only possible to connect from gdb to a running instance of 
OSv on QEMU. But ideally, it would be nice to have a gdb stub in OSv that 
would let one connect from gdb to OSv running on any hypervisor.

Per this fragment of this Wiki 
- https://github.com/cloudius-systems/osv/wiki/Debugging-OSv

*"The fact that our gdb support works with the unit of cpus, not OSV 
threads, makes it convenient to debug OSV itself, but less convenient to 
debug application code running inside OSV. Particularly problematic is 
single stepping support which isn’t currently supported well: The problem 
is that when you try to look at a single OSV thread (see the next section), 
a single-step operation (“s” or “n”) makes one step inside the vcpu, not 
the osv thread*, and this step might include switching to a different 
thread. One way to work around this problem is to compile OSV with 
preemption disabled (not recommended, but can be a convenient trick in some 
debugging situations). *A better approach would have been to include a gdb 
stub into OSV itself, to allow gdb to see OSV threads directly and 
single-step them. We’ve made some progress in implementing such a stub, but 
it isn’t ready for prime-time yet.*"

it looks like somebody may have implemented an initial version of the gdb 
stub. Is there a working patch of it somewhere?

Regards,
Waldek

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/a3ca8d68-5994-4b51-94c4-539e2024f12dn%40googlegroups.com.