Re: nanoBSD boot problem (on USB stick or as a HD)
Thanks to all!
as NGie says, I used the labeling through NANO_LABEL and it works very
well!
Now in the /etc/fstab I have the label and not the disk specific partition!
Maybe is better to set default label in the nanobsd.sh and not the driver
in according to NGie.
Cheers,
Stefano
Il giorno mer 16 set 2015 alle 04:26 NGie Cooper ha
scritto:
> On Tue, Sep 15, 2015 at 7:18 PM, Ryan Stone wrote:
> > On Tue, Sep 15, 2015 at 9:53 PM, Julian Elischer
> wrote:
> >
> >> one possibility is to use gpart label to describe the device.
> >> possibly it woudl hav ehte same result in both cases, but I don't know
> for
> >> sure that
> >> it works for root device.. you'd have to test.
> >>
> >>
> > I would recommend a UFS label instead. gpart labels are kind of fragile
> > and easy to mess up. My previous employer has been shipping systems
> where
> > the root fs is specified in fstab via a UFS label for years and it never
> > gave us a problems.
>
> +100
>
> And FWIW as Stefano discovered later, NANO_LABEL does just that (use
> UFS labels).
>
> The only unfortunate thing is that nanobsd uses a NUL string by default:
>
> grep NANO_LABEL tools/tools/nanobsd/defaults.sh
> tools/tools/nanobsd/defaults.sh:NANO_LABEL=""
> tools/tools/nanobsd/defaults.sh: echo newfs ${NANO_NEWFS}
> ${NANO_LABEL:+-L${NANO_LABEL}${lbl}} ${dev}
> tools/tools/nanobsd/defaults.sh: newfs ${NANO_NEWFS}
> ${NANO_LABEL:+-L${NANO_LABEL}${lbl}} ${dev}
> tools/tools/nanobsd/defaults.sh: if [ ! -z ${NANO_LABEL} ];
> then
> tools/tools/nanobsd/defaults.sh: tunefs -L
> ${NANO_LABEL}"s2a" /dev/${MD}s2a
> tools/tools/nanobsd/defaults.sh: # Override user's NANO_DRIVE if
> they specified a NANO_LABEL
> tools/tools/nanobsd/defaults.sh: [ ! -z "${NANO_LABEL}" ] &&
> NANO_DRIVE="ufs/${NANO_LABEL}"
>
> The default NANO_DRIVE is useless though -- the old ata(4) stack is dead:
>
> 95 # The drive name of the media at runtime
> 96 NANO_DRIVE=ad0
>
> Cheers,
> -NGie
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: nanoBSD boot problem (on USB stick or as a HD)
Hi Oliver, Thank you very much! I solved it setting NANO_LABEL="mylabel". Cheers, Stefano Il giorno mar 15 set 2015 alle 12:05 O. Hartmann < ohart...@zedat.fu-berlin.de> ha scritto: > On Tue, 15 Sep 2015 11:31:36 +0200 > Stefano Garzarella wrote: > > > Hi all, > > I created a nanoBSD image for my gsoc project (ptnetmap on bhyve). > > > > I would like to boot this image on USB stick or in the hypervisor as a > HD. > > I have some problem because if I set NANO_DRIVE="da0" (for USB boot) > > in the nanoBSD configuration file, the boot from USB stick works well, > > but when I try to boot the same image in the hypervisor as a HD, > > I have the following mountroot error: > > > > Trying to mount root from ufs:/dev/da0s1a [ro]... > > mountroot: waiting for device /dev/da0s1a ... > > Mounting from ufs:/dev/da0s1a failed with error 19. > > > > Loader variables: > >vfs.root.mountfrom=ufs:/dev/da0s1a > >vfs.root.mountfrom.options=ro > > > > mountroot> > > > > > > At this point I need to manually specify "ufs:/dev/ad0s1a" to properly > mount > > the root. > > > > Can you help me? > > There is some tricks to avoid this mountroot error? > > > > Thanks, > > Stefano Garzarella > > ___ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to " > freebsd-current-unsubscr...@freebsd.org" > > > Check your /etc/fstab. I guess the device for "/" is still /dev/da0s1a. I > had > to patch the nanoBSD scripts to target that issue and booting from UEFI > systems, > but my solution is not very transparent. > > Using labels would be more convenient and would be much more GPT/UEFI > friendly. > > regards, > > Oliver > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
nanoBSD boot problem (on USB stick or as a HD)
Hi all, I created a nanoBSD image for my gsoc project (ptnetmap on bhyve). I would like to boot this image on USB stick or in the hypervisor as a HD. I have some problem because if I set NANO_DRIVE="da0" (for USB boot) in the nanoBSD configuration file, the boot from USB stick works well, but when I try to boot the same image in the hypervisor as a HD, I have the following mountroot error: Trying to mount root from ufs:/dev/da0s1a [ro]... mountroot: waiting for device /dev/da0s1a ... Mounting from ufs:/dev/da0s1a failed with error 19. Loader variables: vfs.root.mountfrom=ufs:/dev/da0s1a vfs.root.mountfrom.options=ro mountroot> At this point I need to manually specify "ufs:/dev/ad0s1a" to properly mount the root. Can you help me? There is some tricks to avoid this mountroot error? Thanks, Stefano Garzarella ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
bhyve: fix bhyve warning CTASSERT
Hi all, when I compile bhyve, I have the following errors from clang: pci_emul.c:750:2: error: unused typedef '__assert750' [-Werror,-Wunused-local-typedef] CTASSERT(sizeof(struct msicap) == 14); pci_emul.c:776:2: error: unused typedef '__assert776' [-Werror,-Wunused-local-typedef] CTASSERT(sizeof(struct msixcap) == 12); pci_emul.c:928:2: error: unused typedef '__assert928' [-Werror,-Wunused-local-typedef] CTASSERT(sizeof(struct pciecap) == 60); I fixed them in this simple way: diff --git a/bhyverun.h b/bhyverun.h index 87824ef..7ac3aa9 100644 --- a/bhyverun.h +++ b/bhyverun.h @@ -32,7 +32,8 @@ #ifndef CTASSERT /* Allow lint to override */ #defineCTASSERT(x) _CTASSERT(x, __LINE__) #define_CTASSERT(x, y) __CTASSERT(x, y) -#define__CTASSERT(x, y)typedef char __assert ## y[(x) ? 1 : -1] +#define__CTASSERT(x, y)typedef char __assert ## y[(x) ? 1 : -1] \ + __unused #endif Cheers, Stefano ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FreeBSD-head ixgbe build failed
Great! Now all work fine!
Thanks,
Stefano
2015-06-24 19:07 GMT+02:00 Eric Joyner :
> I made the change in r284767. It leaves out the backwards compatibility
> ixgbe build target that the stable/10 commit allowed, but you can now build
> the modules seperately.
>
> On Mon, Jun 22, 2015 at 2:29 PM Eric Joyner wrote:
>
>> I guess I could reverse-MFC r283668, then, to make that work on HEAD.
>>
>> On Mon, Jun 22, 2015, 12:07 PM Stefano Garzarella <
>> stefanogarzare...@gmail.com> wrote:
>>
>>> Hi all,
>>> I tried to compile FreeBSD-head with only "device ix" (without "device
>>> ixv") but the build failed.
>>>
>>> I fixed this behavior using this patch:
>>> diff --git a/sys/conf/files b/sys/conf/files
>>> index 65606b0..279b161 100644
>>> --- a/sys/conf/files
>>> +++ b/sys/conf/files
>>> @@ -1774,25 +1774,25 @@ dev/ixgbe/if_ix.c optional ix inet
>>> \
>>> compile-with "${NORMAL_C} -I$S/dev/ixgbe -DSMP"
>>> dev/ixgbe/if_ixv.c optional ixv inet \
>>> compile-with "${NORMAL_C} -I$S/dev/ixgbe -DSMP"
>>> -dev/ixgbe/ix_txrx.coptional ix ixv inet \
>>> +dev/ixgbe/ix_txrx.coptional ix | ixv inet \
>>> compile-with "${NORMAL_C} -I$S/dev/ixgbe -DSMP"
>>> -dev/ixgbe/ixgbe_phy.c optional ix ixv inet \
>>> +dev/ixgbe/ixgbe_phy.c optional ix | ixv inet \
>>> compile-with "${NORMAL_C} -I$S/dev/ixgbe"
>>> -dev/ixgbe/ixgbe_api.c optional ix ixv inet \
>>> +dev/ixgbe/ixgbe_api.c optional ix | ixv inet \
>>> compile-with "${NORMAL_C} -I$S/dev/ixgbe"
>>> -dev/ixgbe/ixgbe_common.c optional ix ixv inet \
>>> +dev/ixgbe/ixgbe_common.c optional ix | ixv inet \
>>> compile-with "${NORMAL_C} -I$S/dev/ixgbe"
>>> -dev/ixgbe/ixgbe_mbx.c optional ix ixv inet \
>>> +dev/ixgbe/ixgbe_mbx.c optional ix | ixv inet \
>>> compile-with "${NORMAL_C} -I$S/dev/ixgbe"
>>> -dev/ixgbe/ixgbe_vf.c optional ix ixv inet \
>>> +dev/ixgbe/ixgbe_vf.c optional ix | ixv inet \
>>> compile-with "${NORMAL_C} -I$S/dev/ixgbe"
>>> -dev/ixgbe/ixgbe_82598.coptional ix ixv inet \
>>> +dev/ixgbe/ixgbe_82598.coptional ix | ixv inet \
>>> compile-with "${NORMAL_C} -I$S/dev/ixgbe"
>>> -dev/ixgbe/ixgbe_82599.coptional ix ixv inet \
>>> +dev/ixgbe/ixgbe_82599.coptional ix | ixv inet \
>>> compile-with "${NORMAL_C} -I$S/dev/ixgbe"
>>> -dev/ixgbe/ixgbe_x540.c optional ix ixv inet \
>>> +dev/ixgbe/ixgbe_x540.c optional ix | ixv inet \
>>> compile-with "${NORMAL_C} -I$S/dev/ixgbe"
>>> -dev/ixgbe/ixgbe_x550.c optional ix ixv inet \
>>> +dev/ixgbe/ixgbe_x550.c optional ix | ixv inet \
>>> compile-with "${NORMAL_C} -I$S/dev/ixgbe"
>>> dev/ixgbe/ixgbe_dcb.c optional ix ixv inet \
>>> compile-with "${NORMAL_C} -I$S/dev/ixgbe"
>>>
>>> cheers,
>>> Stefano
>>>
>>> --
>>> *Stefano Garzarella*
>>> Software Engineer
>>>
>>> e-mail: stefano.garzare...@gmail.com
>>> github: http://github.com/stefano-garzarella
>>> linkedin: http://it.linkedin.com/pub/stefano-garzarella
>>>
>>
--
*Stefano Garzarella*
Software Engineer
e-mail: stefano.garzare...@gmail.com
github: http://github.com/stefano-garzarella
linkedin: http://it.linkedin.com/pub/stefano-garzarella
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
FreeBSD-head ixgbe build failed
Hi all,
I tried to compile FreeBSD-head with only "device ix" (without "device
ixv") but the build failed.
I fixed this behavior using this patch:
diff --git a/sys/conf/files b/sys/conf/files
index 65606b0..279b161 100644
--- a/sys/conf/files
+++ b/sys/conf/files
@@ -1774,25 +1774,25 @@ dev/ixgbe/if_ix.c optional ix inet \
compile-with "${NORMAL_C} -I$S/dev/ixgbe -DSMP"
dev/ixgbe/if_ixv.c optional ixv inet \
compile-with "${NORMAL_C} -I$S/dev/ixgbe -DSMP"
-dev/ixgbe/ix_txrx.coptional ix ixv inet \
+dev/ixgbe/ix_txrx.coptional ix | ixv inet \
compile-with "${NORMAL_C} -I$S/dev/ixgbe -DSMP"
-dev/ixgbe/ixgbe_phy.c optional ix ixv inet \
+dev/ixgbe/ixgbe_phy.c optional ix | ixv inet \
compile-with "${NORMAL_C} -I$S/dev/ixgbe"
-dev/ixgbe/ixgbe_api.c optional ix ixv inet \
+dev/ixgbe/ixgbe_api.c optional ix | ixv inet \
compile-with "${NORMAL_C} -I$S/dev/ixgbe"
-dev/ixgbe/ixgbe_common.c optional ix ixv inet \
+dev/ixgbe/ixgbe_common.c optional ix | ixv inet \
compile-with "${NORMAL_C} -I$S/dev/ixgbe"
-dev/ixgbe/ixgbe_mbx.c optional ix ixv inet \
+dev/ixgbe/ixgbe_mbx.c optional ix | ixv inet \
compile-with "${NORMAL_C} -I$S/dev/ixgbe"
-dev/ixgbe/ixgbe_vf.c optional ix ixv inet \
+dev/ixgbe/ixgbe_vf.c optional ix | ixv inet \
compile-with "${NORMAL_C} -I$S/dev/ixgbe"
-dev/ixgbe/ixgbe_82598.coptional ix ixv inet \
+dev/ixgbe/ixgbe_82598.coptional ix | ixv inet \
compile-with "${NORMAL_C} -I$S/dev/ixgbe"
-dev/ixgbe/ixgbe_82599.coptional ix ixv inet \
+dev/ixgbe/ixgbe_82599.coptional ix | ixv inet \
compile-with "${NORMAL_C} -I$S/dev/ixgbe"
-dev/ixgbe/ixgbe_x540.c optional ix ixv inet \
+dev/ixgbe/ixgbe_x540.c optional ix | ixv inet \
compile-with "${NORMAL_C} -I$S/dev/ixgbe"
-dev/ixgbe/ixgbe_x550.c optional ix ixv inet \
+dev/ixgbe/ixgbe_x550.c optional ix | ixv inet \
compile-with "${NORMAL_C} -I$S/dev/ixgbe"
dev/ixgbe/ixgbe_dcb.c optional ix ixv inet \
compile-with "${NORMAL_C} -I$S/dev/ixgbe"
cheers,
Stefano
--
*Stefano Garzarella*
Software Engineer
e-mail: stefano.garzare...@gmail.com
github: http://github.com/stefano-garzarella
linkedin: http://it.linkedin.com/pub/stefano-garzarella
freebsd_fix_ix.patch
Description: Binary data
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PicoBSD build error after r281316
2015-06-22 17:03 GMT+02:00 Luigi Rizzo : > > > On Monday, June 22, 2015, Stefano Garzarella > wrote: > >> Hi all, >> I'm using picobsd on FreeBSD-head (r284697) to build a picobsd image with >> gcc, but I have the following errors during the init phase >> (release/picobsd/build/picobsd --init): >> >> In file included from >> /usr/home/stefano/repos/freebsd/lib/liblzma/../../contrib/xz/src/liblzma/lz/lz_encoder.c:23: >> /usr/home/stefano/repos/freebsd/lib/liblzma/../../contrib/xz/src/liblzma/common/memcmplen.h:19:24: >> error: immintrin.h: No such file or directory >> >> >> I found that the r281316 commit introduces these lines that maybe cause >> these errors: >> lib/liblzma/config.h:152 >> /* Define to 1 if you have the header file. */ >> #if defined(__FreeBSD__) && defined(__amd64__) >> #define HAVE_IMMINTRIN_H 1 >> #endif >> >> Before r281316 all work fine. >> >> > This seems a clang vs our-old-GCC issue. > Picobsd is compiled with GCC. > > > Wonder if we should consider the compiler in the conditions to check for > immintrin.h availability. > I think the same. Commenting those lines the picobsd script works fine. Thanks, Stefano > > Cheers > Luigi > > Thanks, >> Stefano Garzarella >> >> -- >> *Stefano Garzarella* >> Software Engineer >> >> e-mail: stefano.garzare...@gmail.com >> github: http://github.com/stefano-garzarella >> linkedin: http://it.linkedin.com/pub/stefano-garzarella >> > > > -- > -+--- > Prof. Luigi RIZZO, ri...@iet.unipi.it . Dip. di Ing. dell'Informazione > http://www.iet.unipi.it/~luigi/. Universita` di Pisa > TEL +39-050-2217533 . via Diotisalvi 2 > Mobile +39-338-6809875 . 56122 PISA (Italy) > -+--- > > -- *Stefano Garzarella* Software Engineer e-mail: stefano.garzare...@gmail.com github: http://github.com/stefano-garzarella linkedin: http://it.linkedin.com/pub/stefano-garzarella ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
PicoBSD build error after r281316
Hi all, I'm using picobsd on FreeBSD-head (r284697) to build a picobsd image with gcc, but I have the following errors during the init phase (release/picobsd/build/picobsd --init): In file included from /usr/home/stefano/repos/freebsd/lib/liblzma/../../contrib/xz/src/liblzma/lz/lz_encoder.c:23: /usr/home/stefano/repos/freebsd/lib/liblzma/../../contrib/xz/src/liblzma/common/memcmplen.h:19:24: error: immintrin.h: No such file or directory In file included from /usr/home/stefano/repos/freebsd/lib/liblzma/../../contrib/xz/src/liblzma/lz/lz_encoder_mf.c:16: /usr/home/stefano/repos/freebsd/lib/liblzma/../../contrib/xz/src/liblzma/common/memcmplen.h:19:24: error: immintrin.h: No such file or directory In file included from /usr/home/stefano/repos/freebsd/lib/liblzma/../../contrib/xz/src/liblzma/lzma/lzma_encoder_optimum_fast.c:13: /usr/home/stefano/repos/freebsd/lib/liblzma/../../contrib/xz/src/liblzma/common/memcmplen.h:19:24: error: immintrin.h: No such file or directory In file included from /usr/home/stefano/repos/freebsd/lib/liblzma/../../contrib/xz/src/liblzma/lzma/lzma_encoder_optimum_normal.c:14: /usr/home/stefano/repos/freebsd/lib/liblzma/../../contrib/xz/src/liblzma/common/memcmplen.h:19:24: error: immintrin.h: No such file or directory I found that the r281316 commit introduces these lines that maybe cause these errors: lib/liblzma/config.h:152 /* Define to 1 if you have the header file. */ #if defined(__FreeBSD__) && defined(__amd64__) #define HAVE_IMMINTRIN_H 1 #endif Before r281316 all work fine. Thanks, Stefano Garzarella -- *Stefano Garzarella* Software Engineer e-mail: stefano.garzare...@gmail.com github: http://github.com/stefano-garzarella linkedin: http://it.linkedin.com/pub/stefano-garzarella ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] Patch to add Software/Generic Segmentation Offload (GSO) support in FreeBSD
Hi Ryan,
in gso_dispatch(), I put the "eh_len" parameter in order to have the
offset of the L3 header. In this way, if someone adds QinQ support,
just call gso_dispatch() with the right length of the MAC header.
During the execution the GSO, the MAC header is simply copied
as it is in each new segment.
Instead, for the vxlan support, we can define a new entries in gso_type,
define a new "gso_functions" to properly handle these types of packets
and mark the packet in the network stack with the correct GSO type.
For now we used only 4 bit to encode the gso_type in m_pkthdr.csum_flags,
but, in the future, we can use more bit or a specific field in the m_pkthdr.
Your suggestions are very good, but I tried to make a software TSO,
modifying as little as possible the network stack.
Thanks,
Stefano
2014-09-18 20:50 GMT+02:00 Ryan Stone :
> On Wed, Sep 17, 2014 at 4:27 AM, Stefano Garzarella
> wrote:
> > Much of the advantage of TSO comes from crossing the network stack only
> > once per (large) segment instead of once per 1500-byte frame.
> > GSO does the same both for segmentation (TCP) and fragmentation (UDP)
> > by doing these operations as late as possible.
>
> My initial impression is that this is a layering violation. Code like
> this gives me pause:
>
> + eh = mtod(m, struct ether_vlan_header *);
> + if (eh->evl_encap_proto == htons(ETHERTYPE_VLAN)) {
> + eh_len = ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN;
> + } else {
> + eh_len = ETHER_HDR_LEN;
> + }
> +
> + return gso_dispatch(ifp, m, eh_len);
>
> If someone adds QinQ support, this code must be updated. When vxlan
> support comes in, we must update this code or else the outer UDP
> packet gets fragmented instead of the inner TCP payload being
> segmented. As more tunneling protocols get added to FreeBSD, the
> dispatch code for GSO gets uglier and uglier.
>
> It seems to me that the real problem that we are trying to solve is a
> lack of batching in the kernel. Currently the network stack operates
> on the mbuf (packet) boundary. It seems to me that we could introduce
> a "packet group" concept that is guaranteed to have the same L3 and L2
> endpoint. In the transmit path, we would initially have a single
> (potentially oversized) packet in the group. When TCP segments the
> packet, it would add each packet to the packet group and pass it down
> the stack. Because we guarantee that the endpoints are the same for
> every packet in the group, the L3 code can do a single routing table
> lookup and the L2 code can do a single l2table lookup for the entire
> group.
>
> The disadvantages of packet groups would be that:
> a) You have touch a lot more code in a lot more places to take
> advantage of the concept.
> b) TSO inherently has the same layering problems. If we're going to
> solve the problem for tunneling protocols then GSO might well be able
> to take advantage of them.
>
--
*Stefano Garzarella*
stefano.garzare...@gmail.com
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] Patch to add Software/Generic Segmentation Offload (GSO) support in FreeBSD
Hi Freddie, this is a preliminary version and, for now, we have not analyzed all aspects. Thanks for your suggestion. We will try to analyze how the GSO affects IPFW as soon as possible. Cheers, Stefano 2014-09-18 17:27 GMT+02:00 Freddie Cash : > On Thu, Sep 18, 2014 at 7:16 AM, Stefano Garzarella < > stefanogarzare...@gmail.com> wrote: > >> I saw the discussion about TSO, but the GSO is a software >> implementation unrelated with the hardware. >> Furthermore, if the TSO is enabled (and supported by the NIC), the GSO is >> not executed, because is useless. >> >> After the execution of the GSO, the packets, that are passed to the device >> driver, are smaller (or equal) than MTU, so the TSO is unnecessary. For >> this reason the GSO doesn't look neither "ifp->if_hw_tsomax" nor hardware >> segment limits. >> >> The GSO is very useful when you can't use the TSO. >> > > How does GSO affect IPFW, specifically the libalias(3)-based, in-kernel > NAT? The ipfw(8) man page mentions that it doesn't play nicely with > hardware-based TSO, and that one should disable TSO when using IPFW NAT. > > Will the software-based GSO play nicely with IPFW NAT? Will it make any > difference to packet throughput through IPFW? > > Or is it still way too early in development to be worrying about such > things? :) > > -- > Freddie Cash > fjwc...@gmail.com > -- *Stefano Garzarella* stefano.garzare...@gmail.com ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] Patch to add Software/Generic Segmentation Offload (GSO) support in FreeBSD
Hi Hans, I saw the discussion about TSO, but the GSO is a software implementation unrelated with the hardware. Furthermore, if the TSO is enabled (and supported by the NIC), the GSO is not executed, because is useless. After the execution of the GSO, the packets, that are passed to the device driver, are smaller (or equal) than MTU, so the TSO is unnecessary. For this reason the GSO doesn't look neither "ifp->if_hw_tsomax" nor hardware segment limits. The GSO is very useful when you can't use the TSO. Cheers, Stefano 2014-09-17 22:27 GMT+02:00 Hans Petter Selasky : > On 09/17/14 20:18, Stefano Garzarella wrote: > >> Hi Adrian, >> the results that I sent, regard just one flow, but I can try with two >> simultaneous flows and I'll send you the results. >> >> Thanks, >> Stefano >> >> > Hi Stefano, > > You might have seen the discussion about TSO. Is it so that the proposed > GSO feature only looks at the "ifp->if_hw_tsomax" field, and ignores > hardware limits regarding maximum segment size and maximum segment count? > > --HPS > -- *Stefano Garzarella* stefano.garzare...@gmail.com ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC] Patch to add Software/Generic Segmentation Offload (GSO) support in FreeBSD
Hi Adrian, the results that I sent, regard just one flow, but I can try with two simultaneous flows and I'll send you the results. Thanks, Stefano 2014-09-17 19:27 GMT+02:00 Adrian Chadd : > Hi! > > Cool! > > How many flows were you testing with? Just one or two? > > It's for outbound, so it's not _as_ big a deal as it is for inbound, > but it'd still be nice to know. > > > -a > > > On 17 September 2014 01:27, Stefano Garzarella > wrote: > > Hi all, > > I have recently worked, during my master’s thesis with the supervision > > of Prof. Luigi Rizzo, on a project to add GSO (Generic Segmentation > > Offload) support in FreeBSD. I will present this project at EuroBSDcon > > 2014, in Sofia (Bulgaria) on September 28, 2014. > > > > Following is a brief description of our project: > > > > The use of large frames makes network communication much less > > demanding for the CPU. Yet, backward compatibility and slow links > > requires the use of 1500 byte or smaller frames. Modern NICs with > > hardware TCP segmentation offloading (TSO) address this problem. > > However, a generic software version (GSO) provided by the OS has > > reason to exist, for use on paths with no suitable hardware, such > > as between virtual machines or with older or buggy NICs. > > > > Much of the advantage of TSO comes from crossing the network stack only > > once per (large) segment instead of once per 1500-byte frame. > > GSO does the same both for segmentation (TCP) and fragmentation (UDP) > > by doing these operations as late as possible. Ideally, this could be > done > > within the device driver, but that would require modifications to all > > drivers. > > A more convenient, similarly effective approach is to segment > > just before the packet is passed to the driver (in ether_output()). > > > > Our preliminary implementation supports TCP and UDP on IPv4/IPv6; > > it only intercepts packets large than the MTU (others are left > unchanged), > > and only when GSO is marked as enabled for the interface. > > > > Segments larger than the MTU are not split in tcp_output(), > > udp_output(), or ip_output(), but marked with a flag (contained in > > m_pkthdr.csum_flags), which is processed by ether_output() just > > before calling the device driver. > > > > ether_output(), through gso_dispatch(), splits the large frame as needed, > > creating headers and possibly doing checksums if not supported by > > the hardware. > > > > In experiments agains an LRO-enabled receiver (otherwise TSO/GSO > > are ineffective) we have seen the following performance, > > taken at different clock speeds (because at top speeds the > > 10G link becomes the bottleneck): > > > > > > Testing enviroment (all with Intel 10Gbit NIC) > > Sender: FreeBSD 11-CURRENT - CPU i7-870 at 2.93 GHz + Turboboost > > Receiver: Linux 3.12.8 - CPU i7-3770K at 3.50GHz + Turboboost > > Benchmark tool: netperf 2.6.0 > > > > --- TCP/IPv4 packets (checksum offloading enabled) --- > > Freq. TSO GSO none Speedup > > [GHz] [Gbps] [Gbps] [Gbps] GSO-none > > 2.93 9347 9298 8308 12 % > > 2.53 9266 9401 6771 39 % > > 2.00 9408 9294 5499 69 % > > 1.46 9408 8087 4075 98 % > > 1.05 9408 5673 2884 97 % > > 0.45 6760 2206 1244 77 % > > > > > > --- TCP/IPv6 packets (checksum offloading enabled) --- > > Freq. TSO GSO none Speedup > > [GHz] [Gbps] [Gbps] [Gbps] GSO-none > > 2.93 7530 6939 4966 40 % > > 2.53 5133 7145 4008 78 % > > 2.00 5965 6331 3152101 % > > 1.46 5565 5180 2348121 % > > 1.05 8501 3607 1732108 % > > 0.45 3665 1505651131 % > > > > > > --- UDP/IPv4 packets (9K) --- > > Freq. GSO none Speedup > > [GHz] [Gbps] [Gbps] GSO-none > > 2.93 9440 8084 17 % > > 2.53 7772 6649 17 % > > 2.00 6336 5338 19 % > > 1.46 4748 4014 18 % > > 1.05 3359 2831 19 % > > 0.45 1312 1120 17 % > > > > > > --- UDP/IPv6 packets (9K) --- > > Freq. GSO none Speedup > > [GHz] [Gbps]
[RFC] Patch to add Software/Generic Segmentation Offload (GSO) support in FreeBSD
Hi all, I have recently worked, during my master’s thesis with the supervision of Prof. Luigi Rizzo, on a project to add GSO (Generic Segmentation Offload) support in FreeBSD. I will present this project at EuroBSDcon 2014, in Sofia (Bulgaria) on September 28, 2014. Following is a brief description of our project: The use of large frames makes network communication much less demanding for the CPU. Yet, backward compatibility and slow links requires the use of 1500 byte or smaller frames. Modern NICs with hardware TCP segmentation offloading (TSO) address this problem. However, a generic software version (GSO) provided by the OS has reason to exist, for use on paths with no suitable hardware, such as between virtual machines or with older or buggy NICs. Much of the advantage of TSO comes from crossing the network stack only once per (large) segment instead of once per 1500-byte frame. GSO does the same both for segmentation (TCP) and fragmentation (UDP) by doing these operations as late as possible. Ideally, this could be done within the device driver, but that would require modifications to all drivers. A more convenient, similarly effective approach is to segment just before the packet is passed to the driver (in ether_output()). Our preliminary implementation supports TCP and UDP on IPv4/IPv6; it only intercepts packets large than the MTU (others are left unchanged), and only when GSO is marked as enabled for the interface. Segments larger than the MTU are not split in tcp_output(), udp_output(), or ip_output(), but marked with a flag (contained in m_pkthdr.csum_flags), which is processed by ether_output() just before calling the device driver. ether_output(), through gso_dispatch(), splits the large frame as needed, creating headers and possibly doing checksums if not supported by the hardware. In experiments agains an LRO-enabled receiver (otherwise TSO/GSO are ineffective) we have seen the following performance, taken at different clock speeds (because at top speeds the 10G link becomes the bottleneck): Testing enviroment (all with Intel 10Gbit NIC) Sender: FreeBSD 11-CURRENT - CPU i7-870 at 2.93 GHz + Turboboost Receiver: Linux 3.12.8 - CPU i7-3770K at 3.50GHz + Turboboost Benchmark tool: netperf 2.6.0 --- TCP/IPv4 packets (checksum offloading enabled) --- Freq. TSO GSO none Speedup [GHz] [Gbps] [Gbps] [Gbps] GSO-none 2.93 9347 9298 8308 12 % 2.53 9266 9401 6771 39 % 2.00 9408 9294 5499 69 % 1.46 9408 8087 4075 98 % 1.05 9408 5673 2884 97 % 0.45 6760 2206 1244 77 % --- TCP/IPv6 packets (checksum offloading enabled) --- Freq. TSO GSO none Speedup [GHz] [Gbps] [Gbps] [Gbps] GSO-none 2.93 7530 6939 4966 40 % 2.53 5133 7145 4008 78 % 2.00 5965 6331 3152101 % 1.46 5565 5180 2348121 % 1.05 8501 3607 1732108 % 0.45 3665 1505651131 % --- UDP/IPv4 packets (9K) --- Freq. GSO none Speedup [GHz] [Gbps] [Gbps] GSO-none 2.93 9440 8084 17 % 2.53 7772 6649 17 % 2.00 6336 5338 19 % 1.46 4748 4014 18 % 1.05 3359 2831 19 % 0.45 1312 1120 17 % --- UDP/IPv6 packets (9K) --- Freq. GSO none Speedup [GHz] [Gbps] [Gbps] GSO-none 2.93 7281 6197 18 % 2.53 5953 5020 19 % 2.00 4804 4048 19 % 1.46 3582 3004 19 % 1.05 2512 2092 20 % 0.45 998826 21 % We tried to change as little as possible the network stack to add GSO support. To avoid changing API/ABI, we temporarily used spare fields in struct tcpcb (TCP Control Block) and struct ifnet to store some information related to GSO (enabled, max burst size, etc.). The code that performs the segmentation/fragmentation is contained in the file gso.[h|c] in sys/net. We used 4 bit in m_pkthdr.csum_flags (CSUM_GSO_MASK) to encode the packet type (TCP/IPv4, TCP/IPv6, etc) to prevent access to the TCP/IP/Ethernet headers of each packet. In ether_output_frame(), if the packet requires the GSO ((m->m_pkthdr.csum_flags & CSUM_GSO_MASK) != 0), it is segmented or fragmented, and then they are sent to the device driver. At https://github.com/stefano-garzarella/freebsd-gso you can find the kernel patches for FreeBSD-current, FreeBSD 10-stable, FreeBSD 9-stable, a simple application (gso-stats.c) that prints the GSO statistics and picobsd images with GSO support. At https://github.com/stefano-garzarella/freebsd-gso-src you can get the FreeBSD source with GSO patch (various branch for FreeBSD
Re: Fix Emulex "oce" driver in CURRENT
2014-07-15 12:00 GMT+02:00 Borja Marcos : > > On Jul 15, 2014, at 11:45 AM, Stefano Garzarella wrote: > > > I just tried to run iperf3 with this patch and STABLE-10 and it seems to > > work. > > Do you have a panic? > > So far, so good. I've ran a couple of iperf3 tests (60 seconds, trying > both directions) and it doesn't crash. > > Without the fixes I obtained a panic quite reliably, in less than 30 > seconds. > Still trying. But the bugs you mentioned (lack of locking and > deallocating, etc) seem to be consistent with the kind of failures I saw > and their apparent randomness. > Well. > > So, asking for spiritual counsel now. Would you use this driver in a > production environment instead of the 747 version downloaded from Emulex? I > think the latter is giving slightly better performance but, anyway, I > disable LRO and TSO because I see a horrible impact on NFS performance. > > I made a diff between the two versions (CURRENT and 747) and I saw that the main difference is in the management of buf_ring through drbr API. In the CURRENT driver they use a new function drbr_peek() instead of drbr_dequeue() and I think this is better. However, even in the 747 version seems to have the problem of the lack of locking. Cheers, Stefano Cheers, > > > > > > Borja. > > -- Stefano Garzarella ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Fix Emulex "oce" driver in CURRENT
2014-07-15 11:46 GMT+02:00 Borja Marcos : > > On Jul 15, 2014, at 11:45 AM, Stefano Garzarella wrote: > > > I just tried to run iperf3 with this patch and STABLE-10 and it seems to > work. > > Do you have a panic? > > Still compiling :) Anyway, you didn't suffer panics before, right? Right, I didn't suffer panics with iperf3, but with netmap in emulation mode I had a lot of panics before this patch. Stefano > > > > Borja. > > -- Stefano Garzarella ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Fix Emulex "oce" driver in CURRENT
I just tried to run iperf3 with this patch and STABLE-10 and it seems to work. Do you have a panic? Cheers, Stefano 2014-07-15 11:19 GMT+02:00 Stefano Garzarella : > I think there is some problem with the email formatting. > I send you a file with both patches. > > Cheers, > Stefano > > > 2014-07-15 11:12 GMT+02:00 Borja Marcos : > > >> On Jul 15, 2014, at 10:43 AM, Stefano Garzarella wrote: >> >> > I used the "oce" driver in CURRENT. >> > I think that this patch in combination with the previous one should >> work in >> > 10-STABLE. >> > >> > I have only tested if it works with CURRENT, but now I try if it works >> with >> > 10-STABLE and I'll send you some feedback. >> >> Hmmm. The patch seems to be broken. I have tried to apply it renaming the >> a/usr/src... to oce_if.c.old and oce_if.c, etc, and patch complains: >> >> Patching file oce_if.c using Plan A... >> patch: malformed patch at line 6: int wq_index); >> >> >> Was it broken by the email client formatting? Or am I being especially >> clumsy today? ;) >> >> >> >> >> Borja. >> >> > > > -- > Stefano Garzarella > -- Stefano Garzarella ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Fix Emulex "oce" driver in CURRENT
I think there is some problem with the email formatting. I send you a file with both patches. Cheers, Stefano 2014-07-15 11:12 GMT+02:00 Borja Marcos : > > On Jul 15, 2014, at 10:43 AM, Stefano Garzarella wrote: > > > I used the "oce" driver in CURRENT. > > I think that this patch in combination with the previous one should work > in > > 10-STABLE. > > > > I have only tested if it works with CURRENT, but now I try if it works > with > > 10-STABLE and I'll send you some feedback. > > Hmmm. The patch seems to be broken. I have tried to apply it renaming the > a/usr/src... to oce_if.c.old and oce_if.c, etc, and patch complains: > > Patching file oce_if.c using Plan A... > patch: malformed patch at line 6: int wq_index); > > > Was it broken by the email client formatting? Or am I being especially > clumsy today? ;) > > > > > Borja. > > -- Stefano Garzarella oce_fix_STABLE10.patch Description: Binary data ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Fix Emulex "oce" driver in CURRENT
I used the "oce" driver in CURRENT. I think that this patch in combination with the previous one should work in 10-STABLE. I have only tested if it works with CURRENT, but now I try if it works with 10-STABLE and I'll send you some feedback. Cheers, Stefano 2014-07-15 10:28 GMT+02:00 Borja Marcos : > > On Jul 15, 2014, at 10:22 AM, Stefano Garzarella wrote: > > > Hi, > > I found other problems in the "oce" driver during some experiments with > > netmap in emulation mode. > > What about driver version 10.0.747.0? At least in my configuration it > works perfectly, no crashes despite keeping it running for several days at > full bandwidth. > > I have a server about to go into production. Should this patch work on > 10-STABLE? > > > > > > > Borja. > > > -- Stefano Garzarella ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Fix Emulex "oce" driver in CURRENT
Hi,
I found other problems in the "oce" driver during some experiments with
netmap in emulation mode.
In details:
- missing locking:
- in some functions there are write accesses on the wq struct (tx queue
descriptor)
without acquire LOCK on the queue, particularly in oce_wq_handler() that is
invoked
in the interrupt routine. For this reason there may be race conditions.
- tx cleanup:
- in oce_if_deactivate() the wq queues are drained but some still pending
mbufs are not freed.
For this reason, I added the oce_tx_clean() that releases any pending mbufs.
I also tried experimenting with iperf3 using the same Borja environment and
I don't have panic.
Can you try this patch? Do you still have the panic?
Cheers,
Stefano Garzarella
diff --git a/sys/dev/oce/oce_if.c b/sys/dev/oce/oce_if.c
index af57491..33b35b4 100644
--- a/sys/dev/oce/oce_if.c
+++ b/sys/dev/oce/oce_if.c
@@ -142,6 +142,7 @@ static int oce_tx(POCE_SOFTC sc, struct mbuf **mpp,
int wq_index);
static void oce_tx_restart(POCE_SOFTC sc, struct oce_wq *wq);
static void oce_tx_complete(struct oce_wq *wq, uint32_t wqe_idx,
uint32_t status);
+static void oce_tx_clean(POCE_SOFTC sc);
static int oce_multiq_transmit(struct ifnet *ifp, struct mbuf *m,
struct oce_wq *wq);
@@ -585,8 +586,10 @@ oce_multiq_flush(struct ifnet *ifp)
int i = 0;
for (i = 0; i < sc->nwqs; i++) {
+ LOCK(&sc->wq[i]->tx_lock);
while ((m = buf_ring_dequeue_sc(sc->wq[i]->br)) != NULL)
m_freem(m);
+ UNLOCK(&sc->wq[i]->tx_lock);
}
if_qflush(ifp);
}
@@ -1052,6 +1055,19 @@ oce_tx_complete(struct oce_wq *wq, uint32_t wqe_idx,
uint32_t status)
}
}
+static void
+oce_tx_clean(POCE_SOFTC sc) {
+ int i = 0;
+ struct oce_wq *wq;
+
+ for_all_wq_queues(sc, wq, i) {
+ LOCK(&wq->tx_lock);
+ while (wq->pkt_desc_tail != wq->pkt_desc_head) {
+ oce_tx_complete(wq, 0, 0);
+ }
+ UNLOCK(&wq->tx_lock);
+ }
+}
static void
oce_tx_restart(POCE_SOFTC sc, struct oce_wq *wq)
@@ -1213,6 +1229,8 @@ oce_wq_handler(void *arg)
struct oce_nic_tx_cqe *cqe;
int num_cqes = 0;
+ LOCK(&wq->tx_lock);
+
bus_dmamap_sync(cq->ring->dma.tag,
cq->ring->dma.map, BUS_DMASYNC_POSTWRITE);
cqe = RING_GET_CONSUMER_ITEM_VA(cq->ring, struct oce_nic_tx_cqe);
@@ -1237,6 +1255,8 @@ oce_wq_handler(void *arg)
if (num_cqes)
oce_arm_cq(sc, cq->cq_id, num_cqes, FALSE);
+ UNLOCK(&wq->tx_lock);
+
return 0;
}
@@ -2087,6 +2107,9 @@ oce_if_deactivate(POCE_SOFTC sc)
/* Delete RX queue in card with flush param */
oce_stop_rx(sc);
+ /* Flush the mbufs that are still in TX queues */
+ oce_tx_clean(sc);
+
/* Invalidate any pending cq and eq entries*/
for_all_evnt_queues(sc, eq, i)
oce_drain_eq(eq);
diff --git a/sys/dev/oce/oce_queue.c b/sys/dev/oce/oce_queue.c
index 308c16d..161011b 100644
--- a/sys/dev/oce/oce_queue.c
+++ b/sys/dev/oce/oce_queue.c
@@ -969,7 +969,9 @@ oce_start_rq(struct oce_rq *rq)
int
oce_start_wq(struct oce_wq *wq)
{
+ LOCK(&wq->tx_lock); /* XXX: maybe not necessary */
oce_arm_cq(wq->parent, wq->cq->cq_id, 0, TRUE);
+ UNLOCK(&wq->tx_lock);
return 0;
}
@@ -1076,6 +1078,8 @@ oce_drain_wq_cq(struct oce_wq *wq)
struct oce_nic_tx_cqe *cqe;
int num_cqes = 0;
+ LOCK(&wq->tx_lock); /* XXX: maybe not necessary */
+
bus_dmamap_sync(cq->ring->dma.tag, cq->ring->dma.map,
BUS_DMASYNC_POSTWRITE);
@@ -1093,6 +1097,7 @@ oce_drain_wq_cq(struct oce_wq *wq)
oce_arm_cq(sc, cq->cq_id, num_cqes, FALSE);
+ UNLOCK(&wq->tx_lock);
}
2014-07-07 13:57 GMT+02:00 Borja Marcos :
>
> On Jul 7, 2014, at 1:23 PM, Luigi Rizzo wrote:
>
> > On Mon, Jul 7, 2014 at 1:03 PM, Borja Marcos wrote:
> > we'll try to investigate, can you tell us more about the environment you
> use ?
> > (FreeBSD version, card model (PCI id perhaps), iperf3 invocation line,
> > interface configuration etc.)
> >
> > The main differences between 10.0.747.0 and the code in head (after
> > our fix) is the use
> > of drbr_enqueue/dequeue versus the peek/putback in the transmit routine.
> >
> >
> > Both drivers still have issues when the link flaps because the
> > transmit queue is not cleaned
> > up properly (unlike what happens in the linux driver and all FreeBSD
> > drivers for different
> > hardware), so it might well be that you are seeing some side effect of
> > that or other
> > problem which manifests itself differently depending on the environment.
> >
> > 'instant panic' by itself does not tell us anything about what could
> > be the problem you experience (and we do not see it with either driver).
>
> The environment details are here:
>
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=183391
>
> The way I produce an instant panic is:
>
> 1) Connect to another machine (
Fix Emulex "oce" driver in CURRENT
Hello, I had problems during some experiments with Emulex and "oce" driver in CURRENT. I found several bugs in the "oce" driver and this patch fixes them. - oce_multiq_start(): if the link is down returns ENXIO without consuming the mbuf. A trivial fix is to remove the initial error check, since oce_multiq_transmit() which is called next handles the link down situation correctly. - oce_multiq_transmit(): there is an extra call to drbr_enqueue() causing the mbuf to be enqueued twice when the NIC's queue is full. - oce_multiq_transmit(): same problem fixed recently in ixgbe (r267187) and other drivers: if the mbuf is enqueued, the proper return value is 0 This patch has been reviewed by luigi (in cc). If someone could have a look on this and give me some feedback it would be great. Regards, Stefano Garzarella diff --git a/sys/dev/oce/oce_if.c b/sys/dev/oce/oce_if.c index 70d6393..af57491 100644 --- a/sys/dev/oce/oce_if.c +++ b/sys/dev/oce/oce_if.c @@ -563,9 +563,6 @@ oce_multiq_start(struct ifnet *ifp, struct mbuf *m) int queue_index = 0; int status = 0; - if (!sc->link_status) - return ENXIO; - if ((m->m_flags & M_FLOWID) != 0) queue_index = m->m_pkthdr.flowid % sc->nwqs; @@ -1274,7 +1271,6 @@ oce_multiq_transmit(struct ifnet *ifp, struct mbuf *m, struct oce_wq *wq) drbr_putback(ifp, br, next); wq->tx_stats.tx_stops ++; ifp->if_drv_flags |= IFF_DRV_OACTIVE; - status = drbr_enqueue(ifp, br, next); } break; } @@ -1285,7 +1281,7 @@ oce_multiq_transmit(struct ifnet *ifp, struct mbuf *m, struct oce_wq *wq) ETHER_BPF_MTAP(ifp, next); } - return status; + return 0; } ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
