Re: detaching xnf(4) not recognized

[Mike Belopuhov]

Sebastian Reitenbach writes:

> Hi,
>
> I'm toying with OpenBSD 6.3 image on AWS, trying to add/remove Elastic 
> Network Interfaces (ENI).
> OpenBSD 6.3 (GENERIC.MP) #107: Sat Mar 24 14:21:59 MDT 2018
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> So when I attach an ENI to a running instance, then the system recognizes it:
> xnf1 at xen0 backend 0 channel 7: address 02:2f:d6:3e:88:50
>
> and I can use the interface, i.e. dhcp just works.
>
> When I later detach the ENI, there is nothing in dmesg, that it recognized 
> it, no detach line, or the like.
> However, even with the detached interface, I get ifconfig output:
> xnf1: flags=8843 mtu 1500
> lladdr 02:2f:d6:3e:88:50
> index 5 priority 0 llprio 3
> media: Ethernet manual
> status: active
> inet 10.2.2.105 netmask 0xff00 broadcast 10.2.2.255
>
> For the OS, the Interface is still available and active, but i.e. dhclient 
> obviously
> doesn't get a lease.
>
> Later on, when I reattach the ENI to the instance, similarily to the detach, 
> no new line
> in dmesg, but I can use the Interface again. It's as if it hasn't been away 
> at all.
>
> Is this all intended, just wondering if I miss something?
>
> cheers,
> Sebastian

AWS XenStore events are a bit finicky.  Do you see anything in your
dmesg?

Please try this patch below and let us know what you see.

diff --git sys/dev/pv/xen.c sys/dev/pv/xen.c
index a2a63537378..4b7f0325911 100644
--- sys/dev/pv/xen.c
+++ sys/dev/pv/xen.c
@@ -1467,10 +1467,12 @@ xen_hotplug(void *arg)
 
memset(&xst, 0, sizeof(xst));
xst.xst_id = 0;
xst.xst_cookie = sc->sc_xs;
 
+   printf("Xen hotplug event for %s\n", xdl->dl_node);
+
snprintf(path, sizeof(path), "device/%s", xdl->dl_node);
if ((error = xs_cmd(&xst, XS_LIST, path, &iovp, &iov_cnt)) != 0)
return;
 
seen = malloc(iov_cnt, M_TEMP, M_ZERO | M_WAITOK);
@@ -1484,11 +1486,11 @@ xen_hotplug(void *arg)
keep++;
break;
}
}
if (!keep) {
-   DPRINTF("%s: removing \"%s/%s\"\n", sc->sc_dev.dv_xname,
+   printf("%s: removing \"%s/%s\"\n", sc->sc_dev.dv_xname,
xdl->dl_node, xdv->dv_unit);
LIST_REMOVE(xdv, dv_entry);
config_detach(xdv->dv_dev, 0);
free(xdv, M_DEVBUF, sizeof(struct xen_device));
}
diff --git sys/dev/pv/xenstore.c sys/dev/pv/xenstore.c
index eb77d384b26..412aecc9e2a 100644
--- sys/dev/pv/xenstore.c
+++ sys/dev/pv/xenstore.c
@@ -699,10 +699,12 @@ xs_event(struct xs_softc *xs, struct xs_msg *xsm)
printf("%s: event on \"%s\" without token\n",
xs->xs_sc->sc_dev.dv_xname, xsm->xsm_data);
return (-1);
}
 
+   printf("XenStore event for \"%s\"\n", token);
+
mtx_enter(&xs->xs_watchlck);
TAILQ_FOREACH(xsw, &xs->xs_watches, xsw_entry) {
if (strcmp(xsw->xsw_token, token))
continue;
mtx_leave(&xs->xs_watchlck);

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

[Mike Belopuhov]

On Thu, Oct 26, 2017 at 19:05 +0100, Tom Smyth wrote:
> Lads,
> 
> Im pleased to say that my testing of OpenBSD 6.1  and OpenBSD 6.2
> Release
> amd64 ,
> appear to work  a little better  in Proxmox PVE5.1 as released this week,
> 
> I used iso version 5.1-722cc488-1 from Proxmox
> Updated on 24 October 2017
> 
> The Console no longer freezes but after a few hours
> the console (vga console accessed via Proxmox webinterface seems
> to lag a little
> the interval between pings for instance takes up to 13 seconds, which
> is a bit strange...  ie it takes 13 seconds for each line of Ping result
> which is u
> Ill report more feedback later, but at least OpenBSD is not freezing
> as bad in this
> version of Proxmox PVE 5.1
> 

Hi,

Can you please show us the output of "sysctl kern.timecounter".
If you're currently using an acpihpet0, can you please try
switching to the acpitimer0 (and if that doesn't help, i8254) via

 sysctl kern.timecounter.hardware=acpitimer0

and attempt to reproduce the 13 secod delay.

Regards,
Mike

Re: Hyper-V Disk Performance

[Mike Belopuhov]

On Mon, Oct 23, 2017 at 16:41 -0500, Daniel Boyd wrote:
> Is there a recommended configuration for virtual disks in Hyper-V?  I
> have a virtual machine that I set up recently running 6.2 that has
> *very* slow disk performance.  It took well over an hour to untar
> ports.tar.gz.  The host server is a few years old, but it's running 3
> RAID-5 7200rpm drives, quad-core Xeon and 32 GB RAM... so not exactly a
> slow machine.  And this is the only Hyper-V VM it's hosting.
> 
> I've got the virtual disk configured as IDE / VHDX / Expanding (the
> Hyper-V defaults).  The controller can be IDE or SCSI.  The disk format
> can be VHD or VHDX.  And the disk can be configured as fixed or
> expanding.  I'm going to try converting the disk to fixed and
> defragging my NTFS.
> 
> Any thoughts on IDE vs SCSI and VHD vs VHDX?
> 

Hi,

Can you please tell us which OpenBSD version you're trying to use?
Could you please show us the dmesg?

You should be using 6.2 as it comes with a driver for the paravirtualized
disk interface, hvs(4).

Regards,
Mike

Re: CoDel Flows

[Mike Belopuhov]

On Mon, Oct 16, 2017 at 03:24 +, Glenn Faustino wrote:
> Thanks Mike!
> 
> When I was using newqueue/hfsc I used to assign queues to certain traffic
> like below:
> 
> match inet proto tcp from any to any port ssh set queue (ssh_bulkq,
> ssh_prioq)
> match inet proto {tcp,udp} from any to any port {domain,ntp} set queue
> (dnsq, ackq)
> match inet proto {tcp,udp} from any to any port {www,https} set queue
> (webq, ackq)
> match inet proto tcp from any to any port ftp set queue (webq, ackq)
> 
> And I was looking how to do that when using flow queues but it seems that
> it is not needed anymore (if I'm not mistaken) ,
> all you need to do is define a flow queue and that's it.
>

I see.  Indeed, I'd start with a single flow queue, but that doesn't
necessarily mean that this is going to work in 100% of cases.  Please
take a look at the approach I've outlined here:

https://www.reddit.com/r/openbsd/comments/75ps6h/fqcodel_and_pf/doemlgi/

I don't have enough feedback and experience with this exact setup so
I'm treading carefully here until there's a clear understanding what
works and what doesn't.

And please let me remind you again, that first of all you need to
identify whether or not the problem actually exists.  Once you figured
that your setup is prone to exposing the bufferbloat on the uplink,
setup the flow queue as I've suggested here:

https://www.reddit.com/r/openbsd/comments/75ps6h/fqcodel_and_pf/doemlgi/

Test and then re-evaluate the situation and only if you see the problem
with downloads, attempt to fix it.

Cheers,
Mike

> 
> Regards,
> Glenn
> 
> 
> 
> On Mon, Oct 16, 2017 at 2:06 AM, Mike Belopuhov  wrote:
> 
> > On Sat, Oct 14, 2017 at 02:51 +, Glenn Faustino wrote:
> > > Hi Mike,
> > >
> > > I'm using HFSC with two root queues (1 for uplink and 1 for downlink),
> > can
> > > you please share your config for FQ-CoDel with HFSC with two queues if
> > you
> > > don't mind?
> > >
> > > Regards,
> > > Glenn
> > >
> >
> > I'm not certain which config you're talking about but there's nothing
> > particularly different about any of those that I've used. I'd like to
> > know instead what might be causing trouble for you to figure out what
> > piece of documentation can be improved.
> >
> > Having said that I tend to set quantum a bit lower when CPU resources
> > are abundant and latency is not increased as a result. For example,
> > OpenWRT sets it to 300 by default which might mean that they optimize
> > for an average packet size on the internet (or it might not, I don't
> > know for sure why do they do it). My limited testing hasn't shown any
> > quantifiable gain when doing that.
> >
> > Regards,
> > Mike
> >

Re: CoDel Flows

[Mike Belopuhov]

On Sat, Oct 14, 2017 at 02:51 +, Glenn Faustino wrote:
> Hi Mike,
> 
> I'm using HFSC with two root queues (1 for uplink and 1 for downlink), can
> you please share your config for FQ-CoDel with HFSC with two queues if you
> don't mind?
> 
> Regards,
> Glenn
> 

I'm not certain which config you're talking about but there's nothing
particularly different about any of those that I've used. I'd like to
know instead what might be causing trouble for you to figure out what
piece of documentation can be improved.

Having said that I tend to set quantum a bit lower when CPU resources
are abundant and latency is not increased as a result. For example,
OpenWRT sets it to 300 by default which might mean that they optimize
for an average packet size on the internet (or it might not, I don't
know for sure why do they do it). My limited testing hasn't shown any
quantifiable gain when doing that.

Regards,
Mike

Re: CoDel Flows

[Mike Belopuhov]

On Fri, Oct 13, 2017 at 05:33 +, Daniel Melameth wrote:
> In playing around with the new CoDel/fair traffic sharing, it's not
> clear to me the best way to work with this when also using the
> previous queuing.

It's not clear to me either at least not in the generic case :-)
I guess it depends on what you're trying to achieve.

> pfctl balks when I specify a flow on the root queue
> with child queues present, so I only specify flows on child queues.

Only leaf queues can be used to assign traffic, so if your root queue
isn't a leaf queue, then it's actual packet queue (sorry for overusing
the term) is not being used and therefore it doesn't make sense for
codel to be specified there at all.

> Will CoDel still work as expected if all my child queues have flows,
> but my root queue is using "fifo" (revealed with systat queues)?

Depends on what you expect CoDel to do.  Normally the idea here is
to set an upper bound on latency that all outgoing packets experience.
For example if you have 10 connections and 2 are uploading data and
other 8 are a mix of ACKs and SSH keystrokes, with FIFO you'd normally
see bulk connections saturating the link and not leaving other 8
connections a chance to send a packet.

So you go and create those HFSC queues and try to reserve the bandwidth
for your ACKs, SSH and whatnot.  The approach that FQ-CoDel takes is
different.  You no longer need to reserve bandwidth as FQ-CoDel attempts
to make the bandwidth "available" when needed -- this is what is fair
sharing essentially.  Which in practice means that those 8 connections
are able to send their small packets "practically" whenever they want
without disrupting your uploads.

This means that if all you want is to be able for your outgoing
connections to fair share the bandwidth you don't need to reserve the
bandwidth at all.

> Assuming it does, if one of my child queues is just for TCP ACKs, does
> it make sense to have a small quantum for this queue, but a larger
> quantum for a child queue that focuses on bulk file transfers?

Quantum of service just tilts the balance at the expense of extra CPU
cycles and potentially extra overall latency.  I think you need to
figure out the big picture first and then fine tune.

> Or is
> CoDel orthogonal of child queues and it only really works well with a
> single root flow queue (and requires me to give up bandwidth control
> with child queues)?
>

"Works well this way or that way" would imply that we have enough data
to make such a judgement.  At the moment we don't.  Last week we had it
running with 8192 flows feeding into an LTE connection with a rather
flaky 50Mbit/s downlink (150Mbit/s up) for about a hundred of users.
With a few HFSC tweaks we had almost no observable SSH latency with
ping times to 8.8.8.8 of about 25ms with fairly low variation.  This
setup used two root queues: one on the uplink, one on the downlink.

> Also, the pf.conf man page says the default qlimit is 1024, but, if I
> don't specify a qlimit, pfctl –vsq shows a qlength of 50 when I was
> expecting it to be 1024.  What am I missing?
>

I've updated the man page today to address some of the concerns since
the same question was also brought up on reddit yesterday:

https://www.reddit.com/r/openbsd/comments/75ps6h/fqcodel_and_pf/

The gist of it is that 1024 is not the HFSC default.  When you're
specifying both "flows" and "bandwidth" thus requesting an FQ-CoDel
queue manager for your HFSC queue, the HFSC default qlimit (50) is
still applied.  It's a bit counter-intuitive I guess, so I've removed
mention of this from the man page.

> Thank you!
> 

Re: late 6.1 question ( arp , table , routing )

[Mike Belopuhov]

On Mon, Sep 11, 2017 at 18:41 -0400, sven falempin wrote:
> Feels like it s impossible to use virtual  routing table without a rdomain
> on interface with 6.1
>

I think you were relying on an arp(1) bug that influenced your
understanding of routing tables and routing domains. To recap:

1. There can be only one ARP table per routing domain.

> # arp -V 122 -s 172.16.1.1 ac:64:dd:b0:00:03 [permanent]
> arp: writing to routing socket: No such process
> arp: 172.16.1.1: No such process
>

What you're trying to do here is to add an ARP entry on the
routing domain 122 (not routing table 122).

2. A single routing domain can have multiple routing tables.

> Even if the routing can be modify with
> 
> /sbin/route -T122 add -inet 172.16.1.1 -llinfo -link -static -iface em5 &&
> route -T122 add default 172.16.1.1
>

This adds a default route into the routing table number 122
no matter which routing domain it belongs to (0 in your case).

> -
> 
> # route -T122 show
> Routing tables
> 
> Internet:
> DestinationGatewayFlags   Refs  Use   Mtu  Prio
> Iface
> default172.16.1.1 UGS00 - 8 em5
> 172.16.1.1 link#6 UHLS   01 - 8 em5
> 
> -
> 
> arp table remain empty and i cannot ping -V 122 172.16.1.1
> 
> This was possible 6.0
> 
> Will it be possible in 6.2 ?? YES :-D :o
>

arp(1) had a bug that prevented all of it's execution modes
from working on multiple rdomains, so it appears that you
were relying on "arp -V 122" working on rdomain 0 while in
fact the intention was for it work work on rdomain 122 and
this should be the case now.

Hope this clarifies the situation.

> # uname -a
> OpenBSD lulz.com 6.2 GENERIC.MP#63 amd64
> 
> # ifconfig  em0
> em0: flags=8843 mtu 1500
> lladdr 00:03:2d:36:95:5e
> index 1 priority 0 llprio 3
> groups: egress
> media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
> status: active
> inet 172.16.1.45 netmask 0xff00 broadcast 172.16.1.255
> 
> # route -T 120 add default 172.16.1.1
> add net default: gateway 172.16.1.1
> # ping -V 172.16.1.1
> ping: rtable value is invalid: 172.16.1.1
> # ping -V 120 172.16.1.1
> PING 172.16.1.1 (172.16.1.1): 56 data bytes
> 64 bytes from 172.16.1.1: icmp_seq=0 ttl=255 time=0.444 ms
> 64 bytes from 172.16.1.1: icmp_seq=1 ttl=255 time=0.351 ms
> 
> 
> --
> 
> I m asking a lot but
> Is there a chance to fix this in 6.1 without to much work ??
> Any workaround ( better than mine ) ?
> 
> [0]-[]-[/]
> # ifconfig em5
> em5: flags=8843 mtu 1500
> lladdr fe:e1:ba:28:22:44
> description: citywan
> index 6 priority 0 llprio 3
> groups: egress
> media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause)
> status: active
> inet 172.16.1.33 netmask 0xff00 broadcast 172.16.1.255
> [0]-[]-[/]
> # route -T 111 add default 172.16.1.1
> add net default: gateway 172.16.1.1: Network is unreachable
>

You should send this bug report separately.
Do I understand it correctly that this command doesn't work
util you issue the one below?

 /sbin/route -T122 add -inet 172.16.1.1 -llinfo -link -static -iface em5

> i am not used to these king of behavior change with openbsd.
>
> Best.
> 
> -- 
> --
> -
> Knowing is not enough; we must apply. Willing is not enough; we must do

Re: Calculate the frequency of the tsc timecounter

[Mike Belopuhov]

On Mon, Jul 31, 2017 at 09:48 +0800, Adam Steen wrote:
> Ted Unangst  wrote:
> > we don't currently export this info, but we could add some sysctls. there's
> > some cpufeatures stuff there, but generally stuff isn't exported until
> > somebody finds a use for it... it shouldn't be too hard to add something to
> > amd64/machdep.c sysctl if you're interested.
> 
> I am interested, as i need the info, i will look into it and hopefully
> come back with a patch.

This is a bad idea because TSC as the time source is only usable
by OpenBSD on Skylake and Kaby Lake CPUs since they encode the TSC
frequency in the CPUID. All older CPUs have their TSCs measured
against the PIT. Currently the measurement done by the kernel isn't
very precise and if TSC is selected as a timecounter, the machine
would be gaining time on a pace that cannot be corrected by our NTP
daemon. (IIRC, about an hour a day on my Haswell running with NTP).

To be able to use TSC as a timecounter source on OpenBSD or Solo5
you'd have to improve the in-kernel measurement of the TSC frequency
first. I've tried to perform 10 measurements and take an average and
it does improve accuracy, however I believe we need to poach another
bit from Linux and re-calibrate TSC via HPET:

 
http://elixir.free-electrons.com/linux/v4.12.4/source/arch/x86/kernel/tsc.c#L409

I think this is the most sane thing we can do. Here's a complete
procedure that Linux kernel undertakes:

 
http://elixir.free-electrons.com/linux/v4.12.4/source/arch/x86/kernel/tsc.c#L751

Regards,
Mike

Re: Calculate the frequency of the tsc timecounter

[Mike Belopuhov]

On Wed, Jul 26, 2017 at 19:24 +0800, Adam Steen wrote:
> Hi
> 
> Is there an easy/accurate way to calculate the tsc timecounter frequency?
> like Time Stamp Counters  on Linux. (on a
> Sandy Bridge cpu)
> 
> Another reference Converting Sandy Bridge TSC to wall clock time
> .
> 
> The code below works but i don't really know how accurate it is, at best
> 10,000 Hz.
> 
> Cheers
> Adam
>

Hi,

First of all it's not clear why do you want to calculate TSC
frequency in the userland program?  The kernel does it and
prints the result to the system message buffer (viewed with
the dmesg command).

The second thing worth pointing out is that gettimeofday is a
syscall that queries the timestamp from the timecounter code
(updated every 10ms) with a current delta read directly from
the hardware so that you get an accurate reading, but then
it's adjusted according to the system time adjustment rules
imposed by things like NTP and settimeofday, so essentially
it's not monotonic (unless you can ensure there is no actor
present that is adjusting the time while you're performing
your measurement).  There's also a way for userland to query
a precise monotonically increasing timestamp: clock_gettime
with CLOCK_MONOTONIC as the clock_id.  In this case the
returned timestamp is relative to the moment the system was
brought up but this doesn't matter if all you need is
difference.

The third thing to know is where does this hardware reading
comes from and what's its precision.  Running "sysctl -n
kern.timecounter.hardware" will tell you what is currently
selected as the source and then you can locate that device
your dmesg (with exception of i8254 -- that 1.19Mhz PIT).
For instance on my laptop it's ACPI HPET that is proving a
running counter with the frequency of 14 MHz. This is what's
going to limit the precision of your measurement.

To get a better reading you may try to take a series of say
10 measurements and calculate the average.

The difference between RDTSC and RDTSCP is that the latter
tells you on which CPU the instruction was executed.  This
poses a valid question: is TSC frequency the same on a multi-
socket system.  And I don't have an answer for that one.
AFAIU, this boils down to the motherboard design and if the
manufacturer has selected to use different quartz crystals
for different sockets, then as we know for a fact that no two
quartz crystals are created the same and thus frequency
sourced from them and multiplied by clock generator PLLs to
produce bus and then core clock signals will be slightly
different between sockets.  I believe there's a way to
compensate for that but OpenBSD doesn't do this currently.

However, your code doesn't check on which CPU the RDTSC has
been executed so you can just use RDTSC and hope that TSC
frequencies are the same and all counters on all cores have
been started at the same time by the firmware (which is
another question whether or not this is actually true).

The CPUID call that you can see used there is there to
provide serialization.  I believe there's no need to do it in
this case.  In fact we haven't observed adverse effects w/o
an additional serialization instruction on Skylake where TSC
is used as the default timecounter (e.g. instead of an HPET).

The other issue with your program is that you don't account
for how long does it take to perform a syscall operation.  You
could time it before running the loop and then subtract the
reading.

And finally, a userland program will not run a 1 second loop
uninterrupted since the scheduler will always attempt to
select a different process every 10ms.  Which means that
a potential context switch and a 10ms timeslice of another
process might make its way into your measurement.

This all begs the same question I asked in the beginning: why
do you want to calculate the TSC frequency in the userland
program?


> #include 
> #include 
> #include 
> 
> uint64_t rdtscp()
> {
> uint32_t lo, hi;
>  __asm__ __volatile__ ("RDTSCP\n\t"
>"mov %%edx, %0\n\t"
>"mov %%eax, %1\n\t"
>"CPUID\n\t": "=r" (hi), "=r" (lo):: "%rax",
> "%rbx", "%rcx", "%rdx");
> return (uint64_t)hi << 32 | lo;
> }
> 
> uint64_t rdtsc()
> {
> uint32_t lo, hi;
>  __asm__ __volatile__ ("CPUID\n\t"
>"RDTSC\n\t"
>"mov %%edx, %0\n\t"
>"mov %%eax, %1\n\t": "=r" (hi), "=r" (lo)::
>"%rax", "%rbx", "%rcx", "%rdx");;
> return (uint64_t)hi << 32 | lo;
> }
> 
> uint64_t get_tsc_freq_hz()
> {
> uint64_t start_timestamp, end_timestamp;
> struct timeval tv_start, tv_end;
> 
> gettimeofday(&tv_start, NULL);
> start_timestamp = rdtsc();
> while (1) {
> gettimeofday(&tv_end, NULL);
> if (tv_end.tv_sec > tv_start.tv_sec + 1)
>  

Re: Disk I/O performance of OpenBSD 5.9 on Xen

[Mike Belopuhov]

On Fri, Jul 21, 2017 at 09:15 -0400, Maxim Khitrov wrote:
> On Sat, Jul 16, 2016 at 6:37 AM, Mike Belopuhov  wrote:
> > On 14 July 2016 at 14:54, Maxim Khitrov  wrote:
> >> On Wed, Jul 13, 2016 at 11:47 PM, Tinker  wrote:
> >>> On 2016-07-14 07:27, Maxim Khitrov wrote:
> >>> [...]
> >>>>
> >>>> No, the tests are run sequentially. Write performance is measured
> >>>> first (20 MB/s), then rewrite (12 MB/s), then read (37 MB/s), then
> >>>> seeks (95 IOPS).
> >>>
> >>>
> >>> Okay, you are on a totally weird platform. Or, on an OK platform with a
> >>> totally weird configuration.
> >>>
> >>> Or on an OK platform and configuration with a totally weird underlying
> >>> storage device.
> >>>
> >>> Are you on a magnet disk, are you using a virtual block device or virtual
> >>> SATA connection, or some legacy interface like IDE?
> >>>
> >>> I get some feeling that your hardware + platform + configuration 
> >>> crappiness
> >>> factor is fairly much through the ceiling.
> >>
> >> Dell R720 and R620 servers, 10 gigabit Ethernet SAN, Dell MD3660i
> >> storage array, 1.2 TB 10K RPM SAS disks in RAID6. I don't think there
> >> is anything crappy or weird about the configuration. Test results for
> >> CentOS on the same system: 170 MB/s write, 112 MB/s rewrite, 341 MB/s
> >> read, 746 IOPS.
> >>
> >> I'm assuming that there are others running OpenBSD on Xen, so I was
> >> hoping that someone else could share either bonnie++ or even just dd
> >> performance numbers. That would help us figure out if there really is
> >> an anomaly in our setup.
> >>
> >
> > Hi,
> >
> > Since you have already discovered that we don't provide a driver
> > for the paravirtualized disk interface (blkfront), I'd say that most likely
> > your setup is just fine, but emulated pciide performance is subpar.
> >
> > I plan to implement it, but right now the focus is on making networking
> > and specifically interrupt delivery reliable and efficient.
> >
> > Regards,
> > Mike
> 
> Hi Mike,
> 
> Revisiting this issue with OpenBSD 6.1-RELEASE and the new xbf driver
> on XenServer 7.0. The write performance is much better at 74 MB/s
> (still slower than other OSs, but good enough). IOPS also improved
> from 95 to 167. However, the read performance actually got worse and
> is now at 16 MB/s. Here are the full bonnie++ results:
> 
> Version  1.97   --Sequential Output-- --Sequential Input- 
> --Random-
> Concurrency   1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
> --Seeks--
> MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec 
> %CP
> web4.dhcp.bhsai. 8G   76191  43 10052  17   16044  25 167.3  
> 43
> Latency 168ms 118ms   416ms 488ms
> 
> Here are two dd runs for writing and reading:
> 
> $ dd if=/dev/zero of=test bs=1M count=2048
> 2147483648 bytes transferred in 25.944 secs (82771861 bytes/sec)
> 
> $ dd if=test of=/dev/null bs=1M
> 2147483648 bytes transferred in 123.505 secs (17387767 bytes/sec)
> 
> Here's the dmesg output:
> 
> pvbus0 at mainbus0: Xen 4.6
> xen0 at pvbus0: features 0x2705, 32 grant table frames, event channel 3
> xbf0 at xen0 backend 0 channel 8: disk
> scsibus1 at xbf0: 2 targets
> sd0 at scsibus1 targ 0 lun 0:  SCSI3 0/direct fixed
> sd0: 73728MB, 512 bytes/sector, 150994944 sectors
> xbf1 at xen0 backend 0 channel 9: cdrom
> xbf1: timed out waiting for backend to connect
> 
> Any ideas on why the read performance is so poor?
> 

Yes, 6.1 has a bug that was fixed recently.  Please use -current.
Given how serious were recent fixes, I cannot possibly recommend
using anything but -current on Xen at this point.

Re: re0 and re1 watchdog timeouts, and system freeze

[Mike Belopuhov]

On Mon, Jun 12, 2017 at 15:11 +1000, David Gwynne wrote:
> On Fri, Jun 09, 2017 at 07:19:34PM +0200, Bj??rn Ketelaars wrote:
> > On Fri 09/06/2017 12:07, Martin Pieuchot wrote:
> > > On 08/06/17(Thu) 20:38, Bj??rn Ketelaars wrote:
> > > > On Thu 08/06/2017 16:55, Martin Pieuchot wrote:
> > > > > On 07/06/17(Wed) 09:43, Bj??rn Ketelaars wrote:
> > > > > > On Sat 03/06/2017 08:44, Bj??rn Ketelaars wrote:
> > > > > > > 
> > > > > > > Reverting back to the previous kernel fixed the issue above. 
> > > > > > > Question: can
> > > > > > > someone give a hint on how to track this issue?
> > > > > > 
> > > > > > After a bit of experimenting I'm able to reproduce the problem. 
> > > > > > Summary is
> > > > > > that queueing in pf and use of a current (after May 30), multi 
> > > > > > processor
> > > > > > kernel (bsd.mp from snapshots) causes these specific watchdog 
> > > > > > timeouts
> > > > > > followed by a system freeze.
> > > > > > 
> > > > > > Issue is 'gone' when:
> > > > > > 1.) using an older kernel (before May 30);
> > > > > > 2.) removal of queueing statements from pf.conf. Included below the 
> > > > > > specific
> > > > > > snippet;
> > > > > > 3.) switch from MP kernel to SP kernel.
> > > > > > 
> > > > > > New observation is that while queueing, using a MP kernel, the 
> > > > > > download
> > > > > > bandwidth is only a fraction of what is expected. Exchanging the MP 
> > > > > > kernel
> > > > > > with a SP kernel restores the download bandwidth to expected level.
> > > > > > 
> > > > > > I'm guessing that this issue is related to recent work on PF?
> > > > > 
> > > > > It's certainly a problem in, or exposed by, re(4) with the recent MP 
> > > > > work
> > > > > in the network stack.
> > > > > 
> > > > > It would help if you could build a kernel with MP_LOCKDEBUG defined 
> > > > > and
> > > > > see if the resulting kernel enters ddb(4) instead of freezing.
> > > > > 
> > > > > Thanks,
> > > > > Martin
> > > > 
> > > > Thanks for the hint! It helped in entering ddb. I collected a bit of 
> > > > output,
> > > > which you can find below. If I read the trace correctly the crash is 
> > > > related
> > > > to line 1750 of sys/dev/ic/re.c:
> > > > 
> > > > d->rl_cmdstat |= htole32(RL_TDESC_CMD_EOF);
> > > 
> > > Could you test the diff below, always with a MP_LOCKDEBUG kernel and
> > > tell us if you can reproduce the freeze or if the kernel enters ddb(4)?
> > > 
> > > Another question, how often do you see "watchdog timeout" messages?
> > > 
> > > Index: re.c
> > > ===
> > > RCS file: /cvs/src/sys/dev/ic/re.c,v
> > > retrieving revision 1.201
> > > diff -u -p -r1.201 re.c
> > > --- re.c  24 Jan 2017 03:57:34 -  1.201
> > > +++ re.c  9 Jun 2017 10:04:43 -
> > > @@ -2074,9 +2074,6 @@ re_watchdog(struct ifnet *ifp)
> > >   s = splnet();
> > >   printf("%s: watchdog timeout\n", sc->sc_dev.dv_xname);
> > >  
> > > - re_txeof(sc);
> > > - re_rxeof(sc);
> > > -
> > >   re_init(ifp);
> > >  
> > >   splx(s);
> > 
> > The diff (with a MP_LOCKDEBUG kernel) resulted in similar traces as before.
> > ddb Output is included below.
> > 
> > With your diff the number of timeout messages decreased from 9 to 2 before
> > entering ddb.
> 
> can you try the diff below please?
>

Oh my, unserialized start.  Someone didn't finish the conversion (-;
OK mikeb

> Index: hfsc.c
> ===
> RCS file: /cvs/src/sys/net/hfsc.c,v
> retrieving revision 1.39
> diff -u -p -r1.39 hfsc.c
> --- hfsc.c8 May 2017 11:30:53 -   1.39
> +++ hfsc.c12 Jun 2017 05:08:01 -
> @@ -817,7 +817,7 @@ hfsc_deferred(void *arg)
>   KASSERT(HFSC_ENABLED(ifq));
>  
>   if (!ifq_empty(ifq))
> - (*ifp->if_qstart)(ifq);
> + ifq_start(ifq);
>  
>   hif = ifq->ifq_q;
>  
> 

Re: inet6 nat-to (group) address selection algorithm question

[Mike Belopuhov]

On 18 May 2017 at 19:51, Adam Thompson  wrote:
>
> So I’ve discovered that, when trying to do NAT66 (for a ULA network), a
line like:
>  "match out on egress inet6 from !(egress:network) to any nat-to
(egress:0)"
> doesn’t work.  (Yes, the network in this case is ridiculously simple.)
>
> I believe it doesn’t work because :0 indicates that aliases on the if
should be ignored, and the first IPv6 address on an if is typically going
to be the LLA… which of course doesn’t work very well for the NAT use case!
>
> However, to my surprise, simply using "nat-to (egress)" does work,
instead.
>
> What I’d like to know is, what magic in pf(4) allows it to auto-select a
"better" (in my case, at least) address when performing the translation?
There’s nothing in the man page that I can see talking about this.  The man
page talks about what happens when I *do* specify :0, but not when I
*don’t*.  (And neither does the FAQ, fyi.)
>
> My concern is that the algorithm (or blind luck, depending) that allows
"nat-to (egress)" to work for me will stop working in some slightly
different scenario or at some random point in the future when something
seemingly-unrelated changes.
>
> Thoughts/hints/explanations?  (Other than "read the source", please -
there's a reason I'm not a kernel programmer!)
>
> Thanks,
> -Adam
>

Hi,

Can you please provide a bit of context, i.e. "ifconfig egress"
output.  It's hard to say why "nat-to (egress:0)" doesn't work
w/o looking at what addresses are available.

The rule of a thumb however, is that pf address pools (things that
follow nat-to, rdr-to and some others) of a dynamic interface type
(that's when you use parenthesis to indicate the dynamic nature
of addresses, i.e. they can come and go) will skip some addresses
not-suitable for purposes of nat-to/rdr-to/etc.  Right now this
means link local addresses.

To understand why this is done, you have to answer the question:
if I have a globally routable IPv6 address(es) alongside default
link local address and I want to round-robin NAT my clients so
that they can access internet, do I want some of them to be natted
to a link local address and some not?

If you think that this needs to be documented, please submit a
pf.conf correction.

Cheers,
Mike

Re: pf queue definition: bandwidth resolution problem

[Mike Belopuhov]

Ah, I see what you mean.  Indeed, we have to make sure the remainder
is 0 when we're displaying the bandwidth.  I think the diff below is
what we want.  Works fine here, any OKs?

On Sat, May 13, 2017 at 18:34 +, Carl Mascott wrote:
> You missed the point. I didn't do any testing. I just looked at the output of 
> "pfctl -squeue" (correction: In the original post I wrote "pfctl -srules") 
> and noticed that the assigned queue bandwidths reported by pfctl were in some 
> cases much different than the specified queue bandwidth parameters in 
> pf.conf. To put it another way, the readback of defined bandwidth does not 
> match the definition.
> 
> From observation it looks like specified bandwidths in pf.conf with 4 or more 
> digits followed by 'K' have the 3 LSD's dropped and 'K' changed to 'M' in the 
> output of "pfctl -squeue." Example: 1800K in pf.conf becomes 1M in the output 
> of "pfctl -squeue" -- a very significant difference.
> 
> It remains to be determined whether the fault is in the setting of bandwidth 
> by pf or in the reporting of bandwidth by pfctl. Actual tests with a simple 
> queue, as you suggested, could determine whether pf is enforcing the correct 
> bandwidth values. To make it easiest to see an error the optimum leaf queue 
> max bandwidth to use is 1999K. Then see whether the measured bandwidth is ~2M 
> or ~1M.
> 
> When I have time I'll do a simple test.
> 
> 
> 
> 
> On Sat, 5/13/17, Mike Belopuhov  wrote:
> 
>  Subject: Re: pf queue definition: bandwidth resolution problem
>  To: "Carl Mascott" 
>  Cc: misc@openbsd.org
>  Date: Saturday, May 13, 2017, 12:02 PM
>  
>  On Tue, May 09, 2017 at 19:47 +, Carl
>  Mascott wrote:
>  > Intel Atom D2500
>  1.66GHz
>  > OpenBSD i386 v6.1-stable
>  > 
>  > I can't get pf
>  to give me the queue bandwidths that I specify in
>  pf.conf.
>  > 
>  >
>  pf.conf:
>  > 
>  > queue
>  rootq on $ext_if bandwidth 9M max 9M qlimit 100
>  >         queue qdef parent rootq
>  bandwidth 3650K default
>  >        
>  queue qrtp parent rootq bandwidth 350K min 350K burst 700K
>  for 200ms
>  >         queue qweb parent
>  rootq bandwidth 4M
>  >         queue
>  qpri parent rootq bandwidth 900K min 50K burst 1800K for
>  200ms
>  >         queue qdns parent
>  rootq bandwidth 100K min 10K burst 200K for 1000ms
>  > 
>  > output of pfctl
>  -srules:
>  > 
>  > queue
>  rootq on bge0 bandwidth 9M, max 9M qlimit 100
>  > queue qdef parent rootq bandwidth 3M
>  default qlimit 50
>  > queue qrtp parent
>  rootq bandwidth 350K, min 350K burst 700K for 200ms qlimit
>  50
>  > queue qweb parent rootq bandwidth 4M
>  qlimit 50
>  > queue qpri parent rootq
>  bandwidth 900K, min 50K burst 1M for 200ms qlimit 50
>  > queue qdns parent rootq bandwidth 100K,
>  min 10K burst 200K for 1000ms qlimit 50
>  >
>  
>  > Discrepancies in the above:
>  > 
>  >             
>     defined         actual
>  >     
>             --         -
>  > qdef BW   3650K          3M
>  > qpri burst  1800K          1M
>  > 
>  > It looks like for
>  anything specified as abcdK the result is aM, i.e., for any
>  bandwidth >= 1000K the resulting bandwidth is
>  truncated (not rounded) to M, where
>   = most significant digit. Any bandwidth
>  < 1000K works correctly.
>  > 
>  > Is this a bug, a misfeature, or a
>  feature?
>  > Thanks!
>  >
>  

Index: sbin/pfctl/pfctl_parser.c
===
RCS file: /home/cvs/src/sbin/pfctl/pfctl_parser.c,v
retrieving revision 1.309
diff -u -p -r1.309 pfctl_parser.c
--- sbin/pfctl/pfctl_parser.c   26 Oct 2016 14:15:59 -  1.309
+++ sbin/pfctl/pfctl_parser.c   13 May 2017 19:18:19 -
@@ -1177,7 +1177,7 @@ print_bwspec(const char *prefix, struct 
printf("%s%u%%", prefix, bw->percent);
else if (bw->absolute) {
rate = bw->absolute;
-   for (i = 0; rate >= 1000 && i <= 3; i++)
+   for (i = 0; rate >= 1000 && i <= 3 && (rate % 1000 == 0); i++)
rate /= 1000;
printf("%s%u%c", prefix, rate, unit[i]);
}

Re: Ipsec - Problem configuring host-to-host

[Mike Belopuhov]

On Wed, May 10, 2017 at 18:36 +, jphe...@yenn.ulegend.net wrote:
> Hello.
> 
> I am trying to establish an ipsec connection in transport mode between two
> hosts located in the same LAN, using PSK for authentication and ikev1 for
> automatic keying. So far, my attempts have resulted in failure.
> 
> Host A ( 192.168.1.11 ) runs OpenBSD 6.1 and uses the following
> configurations:
> 
> /etc/ipsec.conf
> ike passive esp transport from 192.168.1.11 to 192.168.1.12 \
> main auth hmac-sha1 enc aes group modp1024 \
> quick auth hmac-sha1 enc aes group modp1024 \
> psk "test"
> 
> /etc/rc.conf.local
> isakmpd_flags=-K -T
> ipsec=YES
> 
> Host B ( 192.168.1.12 ) runs Knoppix 7.7.1, Strongswan 5.5.0, and uses the
> following configurations:
> 
> /etc/ipsec.conf
> [...]
> conn test
> left=192.168.1.12
> right=192.168.1.11
> authby=secret
> auto=start
> type=transport
> ike=aes-sha1-modp1024
> esp=aes-sha1-modp1024
> compress=no
> 
> /etc/ipsec.secrets
> 192.168.1.12 192.168.1.11 : PSK "test"
> 
> This shows up in /var/log/messages on Host A when establishing ipsec between
> both hosts is attempted:
> May 10 16:57:39 server isakmpd[37746]: isakmpd: starting
> May 10 16:57:58 server isakmpd[4052]: attribute_unacceptable:
> ENCRYPTION_ALGORIT HM: got AES_CBC, expected 3DES_CBC
> May 10 16:57:58 server isakmpd[4052]: message_negotiate_sa: no compatible
> propos al found
> May 10 16:57:58 server isakmpd[4052]: dropped message from 192.168.1.12 port
> 500  due to notification type NO_PROPOSAL_CHOSEN
> 
> Notice that isakmpd is expecting 3DES, when I configured the connection to
> use aes at both ends. More worrysome: When I configure the Host B to use
> 3DES for phase 1, isakmpd complains because it was offered PSK, but RSA_SIG
> was expected! This leads me to believe that isakmpd is ignoring the
> configuration parameters.
> 
> Any help is appreciated. 
> 

This is most likely the result of wrong file permissions on
/etc/ipsec.conf.  Try running "ipsecctl -vnf /etc/ipsec.conf"
as root and you'll notice that file permissions need to be
tightened.  What happens is that your configuration doesn't
get loaded and isakmpd attempts to use a default one.

Re: pf queue definition: bandwidth resolution problem

[Mike Belopuhov]

On Tue, May 09, 2017 at 19:47 +, Carl Mascott wrote:
> Intel Atom D2500 1.66GHz
> OpenBSD i386 v6.1-stable
> 
> I can't get pf to give me the queue bandwidths that I specify in pf.conf.
> 
> pf.conf:
> 
> queue rootq on $ext_if bandwidth 9M max 9M qlimit 100
> queue qdef parent rootq bandwidth 3650K default
> queue qrtp parent rootq bandwidth 350K min 350K burst 700K for 200ms
> queue qweb parent rootq bandwidth 4M
> queue qpri parent rootq bandwidth 900K min 50K burst 1800K for 200ms
> queue qdns parent rootq bandwidth 100K min 10K burst 200K for 1000ms
> 
> output of pfctl -srules:
> 
> queue rootq on bge0 bandwidth 9M, max 9M qlimit 100
> queue qdef parent rootq bandwidth 3M default qlimit 50
> queue qrtp parent rootq bandwidth 350K, min 350K burst 700K for 200ms qlimit 
> 50
> queue qweb parent rootq bandwidth 4M qlimit 50
> queue qpri parent rootq bandwidth 900K, min 50K burst 1M for 200ms qlimit 50
> queue qdns parent rootq bandwidth 100K, min 10K burst 200K for 1000ms qlimit 
> 50
> 
> Discrepancies in the above:
> 
> defined actual
> -- -
> qdef BW   3650K  3M
> qpri burst  1800K  1M
> 
> It looks like for anything specified as abcdK the result is aM, i.e., for any 
> bandwidth >= 1000K the resulting bandwidth is truncated (not rounded) to 
> M, where  = most significant digit. Any bandwidth < 
> 1000K works correctly.
> 
> Is this a bug, a misfeature, or a feature?
> Thanks!
> 

Borrowing is enabled by default, so it's hard to say what affects
your test since you didn't specify how are you exactly testing this.
Every queue in your setup can borrow up to 9M of bandwidth, preventing
others from doing the same, especially since you're attempting to
measure a 200ms burst.  How are you exactly doing this?

I suggest you create a simple isolated test where you have only the
specified traffic, e.g. generated with tcpbench.  Then you disable
borrowing by specifying "max" bandwidth:

queue rootq on $ext_if bandwidth 9M max 9M
queue qdef parent rootq bandwidth 3650K max 3650K default

W/o other queues.  Then you proceed to add others, with or without
borrowing enabled undestanding what you're doing and what an how
you're testing.

Inside pfctl parser all bandwitdh specifications are converted to
bits per second (where K means multiplied by 1000 and M by 1.000.000).

Re: Looking for Xen blkfront driver xbf(4) tests

[Mike Belopuhov]

On Sun, Dec 11, 2016 at 05:09 -0500, mabi wrote:
> Hi,
> 
> Thanks for your efforts and making OpenBSD work even better on
> Xen. I use Xen for all types of virtualization and started only
> recently using OpenBSD 6.0 as domU. My current test setup is a 2
> node redundant cluster with Xen 4.4.1 and Debian 8 with DRBD for
> sync-replication and ZFS (RAIDZ-1) as storage with 3 Seagate
> enterprise 7.2k SATA (ST5000NM0024) disks on each nodes.
> 
> So far so good I managed to re-configure the current kernel and
> re-compiled it with xbf enabled and at reboot it immediately used
> the xbf driver and switched using sd instead of wd. You will find
> the output of my dmesg below.
> 
> For now the only thing a tried out is a quick "dd" as I was
> wondering how much more write throughput I could get on my guests
> disk using xbf. As you can see below I get around 81 MB/s and I
> remember before using xbf I would get around 25 MB/s. The read
> throughput didn't change much, if I remember correctly I had in both
> cases with and without xbf around 60 MB/s.
> 
> $ dd if=/dev/zero of=file2.xbf bs=1M count=1000
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes transferred in 12.277 secs (85405965 bytes/sec)
> 
> Now is there anything else you would like to know/test or benchmarks
> you would like me to run? Keep in mind I am no dev but I am happy to
> help if it can make things progress with running OpenBSD even better
> on Xen.
> 
> Cheers,
> Mabi
> 

Hi,

Thanks for taking your time to test and report.  There's nothing
special to test, just using the disk in a normal way is enough.
After a few reports from Nathanael Rensen several bugs have been
fixed.

I've looked through a bunch of Xen disk subsystem documents and
noted that one of the ways to improve performance is to use
persistent grants.  However it would be nice to establish a
baseline, i.e. what kind of performance do NetBSD, FreeBSD and
Linux guests get out of Blkfront in the VM with the same
configuration on the same host compared to OpenBSD.

It's worth noting that MAXPHYS value limiting the size of an
individual I/O transfer is different on other systems.
Furthermore currently xbf(4) driver limits it further to 11
page segments (44k) since we don't support indirect requests
that potentially can get us extra 20k (MAXPHYS is 64k on
OpenBSD) but would add additional tax on grant table entries.

The other point of interest is the number of outstanding
requests configured by the driver.  If not limited by the
host system, xbf(4) attempts to use 256 requests, but smaller
EC2 instances limit that to just 32 requests which can result
is large performance difference.  To learn the amount of
configured outstanding requests a num-ring-pages property
must be queried:

  # for xbf in $(hostctl device/vbd); do
 dev=$(dmesg | grep $xbf | cut -f 1 -d ' ')
 npages=$(hostctl device/vbd/$xbf/num-ring-pages)
 if [ $? -eq 0 ]; then
   echo $dev $((npages * 32))
 else
   echo $dev 32
 fi
   done

Output would look like so:

  sd0 256
  sd1 256
  cd0 256

Cheers,
Mike

Re: Looking for Xen blkfront driver xbf(4) tests

[Mike Belopuhov]

On Wed, Dec 07, 2016 at 19:30 +0100, Mike Belopuhov wrote:
> Hi,
> 
> I've committed today a driver for the Xen paravirtualized disk
> interface also known as Blkfront.  Despite being pretty stable
> for me so far, it's not enabled by default at the moment.
> Therefore I'm looking for additional tests on different Xen
> versions and EC2 instances to ensure robustness and performance
> of the software.
> 
> To enable the driver, uncomment the xbf line in the kernel
> config file (/sys/arch/amd64/conf/GENERIC) and re-configure and
> re-build the kernel.  The system will automatically switch all
> available wd* disks to sd* but, unless you have opted out of
> using disklabel UIDs in the /etc/fstab, there's no configuration
> tweaking required.
> 
> Please report successes and failures.  In case of a reproducible
> issue, please enable the XEN_DEBUG define in /sys/dev/pv/xenvar.h,
> rebuild you kernel and send me relevant lines from the log (copied
> from the console or /var/log/messages).
> 
> Cheers,
> Mike

Reyk has endured some EC2 breakage and helped a great deal with
debugging.  As a result there has been some critical changes after
the initial check in so please make sure that xbf.c is at 1.5.

Cheers,
Mike

Looking for Xen blkfront driver xbf(4) tests

[Mike Belopuhov]

Hi,

I've committed today a driver for the Xen paravirtualized disk
interface also known as Blkfront.  Despite being pretty stable
for me so far, it's not enabled by default at the moment.
Therefore I'm looking for additional tests on different Xen
versions and EC2 instances to ensure robustness and performance
of the software.

To enable the driver, uncomment the xbf line in the kernel
config file (/sys/arch/amd64/conf/GENERIC) and re-configure and
re-build the kernel.  The system will automatically switch all
available wd* disks to sd* but, unless you have opted out of
using disklabel UIDs in the /etc/fstab, there's no configuration
tweaking required.

Please report successes and failures.  In case of a reproducible
issue, please enable the XEN_DEBUG define in /sys/dev/pv/xenvar.h,
rebuild you kernel and send me relevant lines from the log (copied
from the console or /var/log/messages).

Cheers,
Mike

Re: iked.conf - ikelifetime?

[Mike Belopuhov]

On 26 November 2016 at 03:29,   wrote:
> Hello:
>
> What are the units for "ikelifetime" in iked.conf?
>
> Per the man page, for "lifetime," it states:
>  lifetime time [bytes bytes] ... Several unit specifiers are recognized
> (ignoring case): ‘m’ and ‘h’ for minutes and hours, and ‘K’,
> ‘M’ and ‘G’ for kilo-, mega- and gigabytes accordingly.
>
> But, for "ikelifetime" it only states:
>
> ikelifetime time
> The optional ikelifetime parameter defines the IKE SA expiration
timeout
> by the time SA was created. A zero value disables active IKE SA rekeying.
This
> is the default.
>
> So, what units does the "time" variable for "ikelifetime" expect?
> Minutes?  Seconds?  Something else?
>

It's the same one as for the 'lifetime' parameter, i.e. you can say 120m or
2h.
I've amended the manual page.

> Thanks
>
> ted

Re: Attach EBS volume on aws OpenBSD instance

[Mike Belopuhov]

On 1 November 2016 at 16:46, zack  wrote:
> Hi,
>
> I'm running OpenBsd 6.0 instance on AWS using the public AMI. Attached
> EBS volume as /dev/xvda as i found in previous discussion, but it don't
> seems to get detected on the instance, nothing shows up on dmesg nor i
> can't find the new device. How does this attached volume appear on the
> instance?
>
> Thanks
> Zack
>

Hi Zack,

If the volume that you're attaching is not the one you're booting
from, then it won't appear on the instance at all because all
non-boot volumes are attached as paravirtualized disk interfaces
for which we lack driver support.  Right now we're limited to a
single volume but you may add additional BSD disk labels and/or
growfs existing filesystems.  I plan to add support for
paravirtualized disk interfaces, but don't have time to do it
right now.

Cheers,
Mike

Re: Disk I/O performance of OpenBSD 5.9 on Xen

[Mike Belopuhov]

On 14 July 2016 at 14:54, Maxim Khitrov  wrote:
> On Wed, Jul 13, 2016 at 11:47 PM, Tinker  wrote:
>> On 2016-07-14 07:27, Maxim Khitrov wrote:
>> [...]
>>>
>>> No, the tests are run sequentially. Write performance is measured
>>> first (20 MB/s), then rewrite (12 MB/s), then read (37 MB/s), then
>>> seeks (95 IOPS).
>>
>>
>> Okay, you are on a totally weird platform. Or, on an OK platform with a
>> totally weird configuration.
>>
>> Or on an OK platform and configuration with a totally weird underlying
>> storage device.
>>
>> Are you on a magnet disk, are you using a virtual block device or virtual
>> SATA connection, or some legacy interface like IDE?
>>
>> I get some feeling that your hardware + platform + configuration crappiness
>> factor is fairly much through the ceiling.
>
> Dell R720 and R620 servers, 10 gigabit Ethernet SAN, Dell MD3660i
> storage array, 1.2 TB 10K RPM SAS disks in RAID6. I don't think there
> is anything crappy or weird about the configuration. Test results for
> CentOS on the same system: 170 MB/s write, 112 MB/s rewrite, 341 MB/s
> read, 746 IOPS.
>
> I'm assuming that there are others running OpenBSD on Xen, so I was
> hoping that someone else could share either bonnie++ or even just dd
> performance numbers. That would help us figure out if there really is
> an anomaly in our setup.
>

Hi,

Since you have already discovered that we don't provide a driver
for the paravirtualized disk interface (blkfront), I'd say that most likely
your setup is just fine, but emulated pciide performance is subpar.

I plan to implement it, but right now the focus is on making networking
and specifically interrupt delivery reliable and efficient.

Regards,
Mike

Re: How to turn off disk elevator

[Mike Belopuhov]

On Wed, Jul 13, 2016 at 12:48 +0200, Peter N. M. Hansteen wrote:
> On Wed, Jul 13, 2016 at 12:39:14PM +0200, Christian Rner wrote:
> > > Hello, you should use virtio drivers for the disk in KVM.
> > 
> > I already use virtio ;-) But there is no need for the BSD kernel to do 
> > further
> > scheduling.
> 
> I'm not at all there is a knob to twiddle here, but if there is, will you
> realistically see any difference in performance (assuming this is about
> shaving cycles off)?
>

Christian has a point, there's no need to involve the default block
number oriented elevator (nscan) for devices that don't care about
this.

Unfortunately, as it stands the NOOP elevator (called FIFO in OpenBSD)
will only be used by disks properly advertising SCSI Thin Provisioning
support like this SSD for example:

sd0 at scsibus1 targ 0 lun 0:  SCSI3 0/direct fixed 
naa.50025388a08facb7
sd0: 244198MB, 512 bytes/sector, 500118192 sectors, thin

The controller in question (vioblk) must fake those SCSI pages to let
sd(4) driver query them and select FIFO scheduler.  It's not doing it
at the moment.

It's possible to change the default disk elevator to FIFO by changing
BUFQ_DEFAULT define in /sys/sys/buf.h to BUFQ_FIFO and recompile the
kernel.  There's no way to change it in the runtime.

I would argue that instead of faking SCSI pages, it must be possible
for the controller to hint sd(4) drivers that FIFO scheduling is
preferred.

Re: issue with xnf running 15th jan snapshot on Xen v. 4.5.1

[Mike Belopuhov]

Hi,

Thanks for taking your time to test and write a report.  I've fixed
a few issues since then and was going to ask you to test a -current
kernel.  Could you also please send bug reports like this to bugs@
rather than misc@.

Thank you!

Cheers,
Mike

On Sat, Jan 16, 2016 at 20:56 +0200, Imre Oolberg wrote:
> Hi!
> 
> I read a news today about Xen being enabled and tried it out with the Jan
> 15th snapshot
> 
> SHA256 (install59.iso) =
> 8d16aeb686a1dcc3ce6e8c5192f8708d3878f7690429c843176c5e755386e4f9
> 
> on Xen v. 4.5.1 compiled from Xen sources on Debian v. 8 Jessie running on
> an ordinary amd64 Intel platform. (Actually i am using this very Xen
> platform to run my workstation as another virtual guest, so platform itselt
> is running solid.)
> 
> I booted it with
> 
> # cat openbsd.cfg
> builder = "hvm"
> name = "openbsd"
> memory = "512"
> vcpus = 1
> vif = [ 'mac=00:16:3e:79:22:22,bridge=xenbr0' ]
> disk = [ 'phy:/dev/vg_data_wd_1tb/openbsd,hda,w',
> 'file:/tmp/install59.iso-20160115-ss.iso,hdd:cdrom,r']
> 
> boot="dc"
> vnc=1
> 
> then connected with VNC and installed OpenBSD S.S.
> 
> It boots afterwards ok and when configuring xnf0 network device with
> ifconfig xnf0 192.168.10.128 (or with dhclient) i can ping it from outside
> and it seems to work forever. Also i can see other lan packets passing by
> xnf0 with tcpdump. But when i connect from outside say with ssh client then
> it hangs (i.e. cant enter anything from vnc window, created simple cronjob
> within OpenBSD and i see it stops updateing log file etc).  While connecting
> i see this traffic on dom0
> 
> root@moraal-dom0:/etc/xen# tcpdump -ni vif43.0 host 192.168.10.128
> 
> 19:55:05.985258 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [S], seq
> 3673277459, win 29200, options [mss 1460,sackOK,TS val 40346540 ecr
> 0,nop,wscale 7], length 0
> 19:55:05.985611 IP 192.168.10.128.22 > 192.168.10.10.33616: Flags [S.], seq
> 3141065115, ack 3673277460, win 16384, options [mss
> 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,TS val 345100460 ecr 40346540],
> length 0
> 19:55:05.986017 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [.], ack
> 1, win 229, options [nop,nop,TS val 40346540 ecr 345100460], length 0
> 19:55:05.987116 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [P.], seq
> 1:33, ack 1, win 229, options [nop,nop,TS val 40346541 ecr 345100460],
> length 32
> 19:55:06.012874 IP 192.168.10.128.22 > 192.168.10.10.33616: Flags [P.], seq
> 1:22, ack 33, win 2168, options [nop,nop,TS val 345100460 ecr 40346541],
> length 21
> 19:55:06.013151 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [.], ack
> 22, win 229, options [nop,nop,TS val 40346547 ecr 345100460], length 0
> 19:55:06.022554 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [.], seq
> 33:1481, ack 22, win 229, options [nop,nop,TS val 40346549 ecr 345100460],
> length 1448
> 19:55:06.022562 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [P.], seq
> 1481:2001, ack 22, win 229, options [nop,nop,TS val 40346549 ecr 345100460],
> length 520
> 19:55:06.023068 IP 192.168.10.128.22 > 192.168.10.10.33616: Flags [.], ack
> 2001, win 1926, options [nop,nop,TS val 345100460 ecr 40346549], length 0
> 19:55:06.023416 IP 192.168.10.128.22 > 192.168.10.10.33616: Flags [P.], seq
> 22:998, ack 2001, win 1926, options [nop,nop,TS val 345100460 ecr 40346549],
> length 976
> 19:55:06.026537 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [P.], seq
> 2001:2049, ack 998, win 244, options [nop,nop,TS val 40346550 ecr
> 345100460], length 48
> 19:55:06.226645 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [P.], seq
> 2001:2049, ack 998, win 244, options [nop,nop,TS val 40346601 ecr
> 345100460], length 48
> 19:55:06.430626 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [P.], seq
> 2001:2049, ack 998, win 244, options [nop,nop,TS val 40346652 ecr
> 345100460], length 48
> 19:55:06.838658 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [P.], seq
> 2001:2049, ack 998, win 244, options [nop,nop,TS val 40346754 ecr
> 345100460], length 48
> 19:55:07.654663 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [P.], seq
> 2001:2049, ack 998, win 244, options [nop,nop,TS val 40346958 ecr
> 345100460], length 48
> 19:55:09.290635 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [P.], seq
> 2001:2049, ack 998, win 244, options [nop,nop,TS val 40347367 ecr
> 345100460], length 48
> 19:55:12.558753 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [P.], seq
> 2001:2049, ack 998, win 244, options [nop,nop,TS val 40348184 ecr
> 345100460], length 48
> 19:55:19.102779 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [P.], seq
> 2001:2049, ack 998, win 244, options [nop,nop,TS val 40349820 ecr
> 345100460], length 48
> 19:55:32.174732 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [P.], seq
> 2001:2049, ack 998, win 244, options [nop,nop,TS val 40353088 ecr
> 345100460], length 48
> 19:55:58.350805 IP 192.168.10.10.33616 > 192.168.10.128.22: Flags [P.], seq
> 2001:2049, ack 998, win 244, option

Re: Strange network issue during startup

[Mike Belopuhov]

On 9 October 2015 at 19:15, Alessandro DE LAURENZIS
 wrote:
> Hi Mike,
>
> On Fri 09/10/2015 19:07, Mike Belopuhov wrote:
>> Hi,
>>
>> Please drop the diff I've sent you and try current.
>> The fix should be in the if_trunk.c revision 1.121.
>>
>
> Just tried 9 Oct snapshot: problem solved!
>
> Thank you very much for your prompt reaction.
>
> Just out of my curiosity: was this the same problem in [0]?
>
> Cheers
>
> [0] http://marc.info/?l=openbsd-misc&m=144422539521881&w=2
>

Yeah, this is the diff that I've committed.

> --
> Alessandro DE LAURENZIS
> [mailto:just22@gmail.com]
> LinkedIn: http://it.linkedin.com/in/delaurenzis

Re: Strange network issue during startup

[Mike Belopuhov]

On 7 October 2015 at 07:04, Alessandro DE LAURENZIS
 wrote:
> I'll try your patch asap.
>

Hi,

Please drop the diff I've sent you and try current.
The fix should be in the if_trunk.c revision 1.121.

Regards,
Mike

> All the best
>
> --
> Alessandro DE LAURENZIS
> [mailto:just22@gmail.com]
> LinkedIn: http://it.linkedin.com/in/delaurenzis

Re: unbreak trunk(4)

[Mike Belopuhov]

On Wed, Oct 07, 2015 at 15:41 +0200, Mike Belopuhov wrote:
> Hi,
> 
> If you have noticed recent problems with trunk(4) please try the
> diff below as it fixes a subtle issue (not introduced by my changes!)
> with setting lladdr on non primary trunk ports: trunk_port_ioctl
> needs to be able to lookup the trunk port, but we didn't put it on
> the list yet, doh!
> 
> OK's are welcome as well.
> 

boo hoo, it clashes with some uncommitted changes, here's a rebased version.


diff --git sys/net/if_trunk.c sys/net/if_trunk.c
index 8678fe4..2aaa339 100644
--- sys/net/if_trunk.c
+++ sys/net/if_trunk.c
@@ -358,18 +358,18 @@ trunk_port_create(struct trunk_softc *tr, struct ifnet 
*ifp)
tr->tr_primary = tp;
tp->tp_flags |= TRUNK_PORT_MASTER;
trunk_lladdr(&tr->tr_ac, tp->tp_lladdr);
}
 
-   /* Update link layer address for this port */
-   trunk_port_lladdr(tp,
-   ((struct arpcom *)(tr->tr_primary->tp_if))->ac_enaddr);
-
/* Insert into the list of ports */
SLIST_INSERT_HEAD(&tr->tr_ports, tp, tp_entries);
tr->tr_count++;
 
+   /* Update link layer address for this port */
+   trunk_port_lladdr(tp,
+   ((struct arpcom *)(tr->tr_primary->tp_if))->ac_enaddr);
+
/* Update trunk capabilities */
tr->tr_capabilities = trunk_capabilities(tr);
 
/* Add multicast addresses to this port */
trunk_ether_cmdmulti(tp, SIOCADDMULTI);

unbreak trunk(4)

[Mike Belopuhov]

Hi,

If you have noticed recent problems with trunk(4) please try the
diff below as it fixes a subtle issue (not introduced by my changes!)
with setting lladdr on non primary trunk ports: trunk_port_ioctl
needs to be able to lookup the trunk port, but we didn't put it on
the list yet, doh!

OK's are welcome as well.

diff --git sys/net/if_trunk.c sys/net/if_trunk.c
index a97741a..9a3dbb5 100644
--- sys/net/if_trunk.c
+++ sys/net/if_trunk.c
@@ -356,18 +356,18 @@ trunk_port_create(struct trunk_softc *tr, struct ifnet 
*ifp)
tr->tr_primary = tp;
tp->tp_flags |= TRUNK_PORT_MASTER;
trunk_lladdr((struct ifnet *)&tr->tr_ac, tp->tp_lladdr);
}
 
-   /* Update link layer address for this port */
-   trunk_port_lladdr(tp,
-   ((struct arpcom *)(tr->tr_primary->tp_if))->ac_enaddr);
-
/* Insert into the list of ports */
SLIST_INSERT_HEAD(&tr->tr_ports, tp, tp_entries);
tr->tr_count++;
 
+   /* Update link layer address for this port */
+   trunk_port_lladdr(tp,
+   ((struct arpcom *)(tr->tr_primary->tp_if))->ac_enaddr);
+
/* Update trunk capabilities */
tr->tr_capabilities = trunk_capabilities(tr);
 
/* Add multicast addresses to this port */
trunk_ether_cmdmulti(tp, SIOCADDMULTI);

Re: Strange network issue during startup

[Mike Belopuhov]

On Tue, Oct 06, 2015 at 08:01 +0200, Alessandro DE LAURENZIS wrote:
> Hello Mike,
> 
> Thanks for your feedback,
> 
> On Mon 05/10/2015 16:43, Mike Belopuhov wrote:
> > 
> > Can you please add an "ifconfig -A" invocation to your hostname.trunk0:
> > 
> > trunkproto failover
> > trunkport  em0
> > trunkport  iwn0
> > !/sbin/ifconfig -A >/root/ifconfig.out 2>&1
> > dhcp
> > 
> > And send me the output.
> 
> [snip]
> lo0: flags=8049 mtu 32768
> priority: 0
> groups: lo
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
> inet 127.0.0.1 netmask 0xff00
> em0: flags=8b43 mtu 
> 1500
> lladdr 00:21:86:94:34:8e
> priority: 0
> trunk: trunkdev trunk0
> media: Ethernet autoselect (none)
> status: no carrier
> iwn0: flags=8943 mtu 1500
> lladdr 00:21:86:94:34:8e
> priority: 4
> trunk: trunkdev trunk0
> groups: wlan
> media: IEEE802.11 autoselect (DS1)
> status: no network
> ieee80211: nwid  wpakey  wpaprotos wpa1,wpa2 
> wpaakms psk wpaciphers tkip,ccmp wpagroupcipher tkip
> enc0: flags=0<>
> priority: 0
> groups: enc
> status: active
> trunk0: flags=8843 mtu 1500
> lladdr 00:21:86:94:34:8e
> priority: 0
> trunk: trunkproto failover
> trunkport iwn0 active
> trunkport em0 master
> groups: trunk egress
> media: Ethernet autoselect
> status: active
> inet 192.168.1.13 netmask 0xff00 broadcast 192.168.1.255
> pflog0: flags=141 mtu 33144
> priority: 0
> groups: pflog
> [snip]
> 
> To me, it seems that trunk0 is correctly bound to the wifi and has an IP
> address assigned...
>

Is this from the /root/ifconfig.out?  Trunk interface should not
have an IP address at this point.  How does your /etc/hostname.trunk0
look like right now?

> > 
> > Could you also try to capture the initial output from the dhclient
> > or make a photo of the DHCP progress (and failure).
> > 
> 
> Nothing "special":
> 
> [snip]
> DHCPREQUEST on trunk0 to 255.255.255.255
> DHCPREQUEST on trunk0 to 255.255.255.255
> DHCPREQUEST on trunk0 to 255.255.255.255
> DHCPREQUEST on trunk0 to 255.255.255.255
> DHCPREQUEST on trunk0 to 255.255.255.255
> DHCPDISCOVER on trunk0 - interval 1
> DHCPDISCOVER on trunk0 - interval 2
> DHCPDISCOVER on trunk0 - interval 3
> DHCPDISCOVER on trunk0 - interval 8
> DHCPDISCOVER on trunk0 - interval 11
> DHCPDISCOVER on trunk0 - interval 11
> DHCPDISCOVER on trunk0 - interval 14
> DHCPDISCOVER on trunk0 - interval 9
> No acceptable DHCPOFFERS received
> Trying recorded lease 192.168.1.13
> bound to 192.168.1.13 - renewal in 21600 seconds.
> [snip]
>

[snip]

> bear with me, I need a couple of days to try this; I'll keep you
> updated.
>

Please take your time.

> Cheers
> 
> -- 
> Alessandro DE LAURENZIS
> [mailto:just22@gmail.com]
> LinkedIn: http://it.linkedin.com/in/delaurenzis

Support for ActivCard, CRYPTOCard and SNK-004 authentication tokens

[Mike Belopuhov]

Hello,

We're currently evaluating if we should keep providing support
for ActivCard, CRYPTOCard and SNK-004 authentication tokens via
login_token(8).  If you're a user of "activ", "crypto", "snk" or
"token" authentication methods (check your /etc/login.conf),
please speak up so that we could estimate a number of active
users.  If noone speaks up the support for these authentication
methods will be removed and won't ship with OpenBSD 5.9.

Regards,
Mike

Re: "Logjam Attack": is OpenIKED and OpenSMTPD vulnerable?

[Mike Belopuhov]

On 25 May 2015 at 14:33, Pablo Méndez Hernández  wrote:
> Hi,
>
> Any statement for iked?
>

iked implements IKEv2 which doesn't use SSL/TLS.  So this
attack doesn't directly apply to IKEv2.  However we would
accept MODP 1024 and better by default.  Perhaps we
should bump it to 2048 minimum.

Re: IPv6 help needed

[Mike Belopuhov]

Hi,

Are you running 5.7-current or a -release?
Did you try running iked?  It has a -6 option (see the
description in the man page) that disables this behavior.
This was done in the light of F. Gont's RFC 7359.

Cheers.

On 22 May 2015 at 01:16, Chris Moody  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> YOU ARE THE MAN
>
> That did it.  I never would've thought to look at that.
>
> =
> root@spike ~ # ping6 2001:470::255::254
> PING6(56=40+8+8 bytes) 2001:470::255::254 --> 2001:470::255::254
> 16 bytes from 2001:470::255::254, icmp_seq=0 hlim=64 time=0.150 ms
> 16 bytes from 2001:470::255::254, icmp_seq=1 hlim=64 time=0.276 ms
> ^C
> - --- 2001:470::255::254 ping6 statistics ---
> 2 packets transmitted, 2 packets received, 0.0% packet loss
> round-trip min/avg/max/std-dev = 0.150/0.213/0.276/0.063 ms
>
> root@spike ~ # ping6 2001:470::255::11
> PING6(56=40+8+8 bytes) 2001:470::255::254 --> 2001:470::255::11
> 16 bytes from 2001:470::255::11, icmp_seq=0 hlim=64 time=0.389 ms
> ^C
> - --- 2001:470::255::11 ping6 statistics ---
> 1 packets transmitted, 1 packets received, 0.0% packet loss
> round-trip min/avg/max/std-dev = 0.389/0.389/0.389/0.000 ms
> =
>
> Is this something that persists across reboots? or does it require
> like an /etc/rc.conf.local setting to bypass?
>
> Thank you incredibly!!
>
> - -Chris
>
> On 5/21/15 3:23 PM, Mike Belopuhov wrote:
>> ipsecctl -F
> -BEGIN PGP SIGNATURE-
> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
> Comment: GPGTools - https://gpgtools.org
>
> iQEcBAEBCgAGBQJVXmdDAAoJEOO37M6Idh8NCkEIAJ2TA+/+gdi+Rx2L23PUdNjs
> ZjoTU0fedSowaJic8m2wl3cvyfdHNmc/PXqWVza+2FLmEdG3D08S5H1YuoMEGSJs
> GKfKxEnOOtna7am2Sfe4hHxRmhAdX40VEpsqs2g/jKi/5aMpsGxLpgwnSKZit0m7
> 0nnbKG0r14nS3plUmE6n+6P/+mhQ2daBY2LfCbePz/VlFSQHea6WWj3VlVcGwPGx
> 52siP6F++vc3Jm5HhlPaw/Ow5s/Cbi+ykB4znYWXupHfZmWDYG2lQ2a0+7I68OrA
> gtaVMQcYtm/g8fMrGnfyxW3TuFlz6fdMntckGq5RrzlN3z6EgxPEZhckEYoyIzU=
> =nuOm
> -END PGP SIGNATURE-

Re: IPv6 help needed

[Mike Belopuhov]

hey, do you happen to see anything in the "ipsecctl -sf" output?
if there's a deny flow, try flushing it via "ipsecctl -F".

On 21 May 2015 at 23:22, Chris Moody  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> Hello - new list-subscriber here.
>
> Need a little help debugging IPv6 on OpenBSD 5.7.   I've got a VMware VM
> running as a router.  IPv4 is working beautifully.  V6 is not working
> for some reason.  I'm unable to even ping6 the system's local interfaces.
>
> I've cleared all v6 interface settings and am trying to bring them up
> one interface at a time while debugging but still no luck.
>
> link-local and global addresses are not responding.  PF is disabled
> while troubleshooting.
>
> I've included some relevant v6 settings and outputs in case it sheds
> some light or gives any ideas.
>
> ==[ pf disabled
> root@spike ~ # pfctl -d
> pfctl: pf not enabled
>
> ==[ first interface I'm bringing up
> root@spike ~ # ifconfig em5
> em5: flags=8843 mtu 1500
> lladdr 00:0c:29:cc:5c:76
> priority: 0
> groups: egress
> media: Ethernet autoselect (1000baseT full-duplex,master)
> status: active
> inet 192.168.255.254 netmask 0xff00 broadcast 192.168.255.255
> inet6 fe80::20c:29ff:fecc:5c76%em5 prefixlen 64 scopeid 0x6
> inet6 2001:470::255::254 prefixlen 64
>
> ==[ unable to ping6 local interface's link-local address
> root@spike ~ # ping6 fe80::20c:29ff:fecc:5c76
> PING6(56=40+8+8 bytes) ::1 --> fe80::20c:29ff:fecc:5c76
> ping6: sendmsg: No route to host
> ping6: wrote fe80::20c:29ff:fecc:5c76 16 chars, ret=-1
> ^C
> - --- fe80::20c:29ff:fecc:5c76 ping6 statistics ---
> 1 packets transmitted, 0 packets received, 100.0% packet loss
>
> ==[ unable to ping6 local interface's global address
> root@spike ~ # ping6 2001:470::255::254
> PING6(56=40+8+8 bytes) 2001:470::255::254 --> 2001:470::255::254
> ping6: sendmsg: No route to host
> ping6: wrote 2001:470::255::254 16 chars, ret=-1
> ^C
> - --- 2001:470::255::254 ping6 statistics ---
> 1 packets transmitted, 0 packets received, 100.0% packet loss
>
> ==[ lo0 only has it's link-local default address
> root@spike ~ # ifconfig lo0
> lo0: flags=8049 mtu 32768
> priority: 0
> groups: lo
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb
> inet6 ::1 prefixlen 128
> inet 127.0.0.1 netmask 0xff00
>
> ==[ IPv6 forwarding is enabled
> root@spike ~ # sysctl net.inet6.ip6.forwarding
> net.inet6.ip6.forwarding=1
>
> ==[ netstat -rn output
> Internet6:
> DestinationGateway
> Flags   Refs  Use   Mtu  Prio Iface
> ::/104 ::1
> UGRS   00 32768 8 lo0
> ::/96  ::1
> UGRS   00 32768 8 lo0
> ::127.0.0.0/104::1
> UGRS   00 32768 8 lo0
> ::224.0.0.0/100::1
> UGRS   00 32768 8 lo0
> ::255.0.0.0/104::1
> UGRS   00 32768 8 lo0
> :::0.0.0.0/96  ::1
> UGRS   00 32768 8 lo0
> 2001:470::255::/64 link#6
> UC 10 - 4 em5
> 2001:470::255::11  00:0c:29:80:a8:bd
> UHLc   00 - 4 em5
> 2001:470::255::254 00:0c:29:cc:5c:76
> UHLl   00 - 1 _*lo0   ??? <- don't understand why
> lo0 shows up here*_
> 2002::/24  ::1
> UGRS   00 32768 8 lo0
> 2002:7f00::/24 ::1
> UGRS   00 32768 8 lo0
> 2002:e000::/20 ::1
> UGRS   00 32768 8 lo0
> 2002:ff00::/24 ::1
> UGRS   00 32768 8 lo0
> fe80::/10  ::1
> UGRS   00 32768 8 lo0
> fe80::%em5/64  link#6
> UC 00 - 4 em5
> fe80::20c:29ff:fecc:5c76%em5   00:0c:29:cc:5c:76
> UHLl   00 - 1 lo0
> fec0::/10  ::1
> UGRS   00 32768 8 lo0
> ff01::/16  ::1
> UGRS   00 32768 8 lo0
> ff01::%em5/32  link#6
> UC 00 - 4 em5
> ff02::/16  ::1
> UGRS   00 32768 8 lo0
> ff02::%em5/32  link#6
> UC 00 - 4 em5
> root@spike ~ #
> ==
>
> Does anyone have any ideas.  I'm betting it's a face-palm level user
> error that I'm just not seeing.
>
> Any help is appreciated.
>
> Thanks,
> - -Chris
> Comment: GPGTools - https://gpgtools.org
>
> iQEcBAEBCgAGBQJVXkyYAAoJEOO37M6Idh8NH1oH/j4KOTXNmGk3S4jBM55guzEb
> WHDZGS+9tbmKcy428mJKTp3awf/8f9wyBBUh5phIpoLUYpj7dbvBI9LcrP8FCI8i
> nd23DD5szQNlJ5XLaAejkdnUvC0Y9B6WTnvPlvUbemMHjOpawE8zmUJFOZ+rE83W
> 27XDP8qF5PvmNzkI8wrszuHIHAMCYMXdEZlJb0

Re: iked without psk

[Mike Belopuhov]

On 6 November 2014 10:19, Peter J. Philipp  wrote:
> Hi,
>
> Since my upgrade on saturday to 5.6 my iked stopped working with psk.
> I've disabled it by now but the config was something of the order of:
>
> ikev2 active esp from 192.168.179.1 to 192.168.179.10 psk "icutwithanulu!"
> ikev2 active esp from 192.168.179.10 to 192.168.179.1 psk "icutwithanulu!"
>
> And this had worked before 5.6.  It even worked when I upgraded the
> first firewall and the other firewall was still 5.5.  But two firewalls
> with 5.6 it stopped working.
>
> I'm looking for pointers on how to make rsa keys work.  I followed the
> manpage of ikectl but the IPSEC doesn't establish itself and I get:
>
> Nov  6 10:17:36 venus iked[15811]: ca_getreq: no valid local certificate
> found
>
> Any hints would be appreciated.
>
> -peter
>

hi,

psk is now fixed in current.

there are two other ways to authenticate hosts:  rsa pubkeys (a recent
addition - works the same way as in isakmpd) and x.509 certificates.
both these options do not require any special config options (it's "rsa"
actually, but that's the default) and will be hooked up on startup.

the procedure to setup x.509 certificates is described in ikectl(8) and
i would strongly suggest using this tool.

regarding rsa keys: i have just committed a man page update taken from
isakmpd(8) but essentially it's just an

 hostA# scp /etc/iked/local.pub root@hostB:/etc/iked/pubkeys/ipv4/host.A.IP.Addr
 hostB# scp /etc/iked/local.pub root@hostA:/etc/iked/pubkeys/ipv4/host.B.IP.Addr

and off you go.

the important part is to keep your srcids and dstids sane, for instance
if you're installing pubkeys under /ipv4/ you should use IPv4 IDs in
the iked.conf.

hope this helps and please try with -current iked again.

Re: pf anchor references

[Mike Belopuhov]

On Mon, Jun 02, 2014 at 17:51 +0200, Mike Belopuhov wrote:
> Hi,
> 
> I've been chasing some bugs in the pfctl anchor code for a couple
> of weeks and I'm not astonished at how loose the handling is in
> general.  Lot's of rules and checks are being violated by some
> code paths while honoured by others.  The problem I have is that
> it's truly a rabbit's hole: almost every bug I hit is a feature
> in disguise.
> 
> One thing that I particularly hate is called an anchor reference.
> A good example of this is a ruleset like this:
> 
> anchor "foo" {
> block all
> }
> anchor "foo"
> 
> If you believe it should be a syntax error let me disappoint you.
> The second 'anchor "foo"' is just a reference to the anchor named
> "foo" defined before:
> 
> # echo -e 'anchor "foo" {\n block all\n}\nanchor "foo"' | pfctl -f -
> # pfctl -a '*' -sr
> anchor "foo" all {
>   block drop all
> }
> anchor "foo" all {
>   block drop all
> }
> # echo -e 'pass all' | pfctl -a 'foo' -f -
> # pfctl -a '*' -sr
> anchor "foo" all {
>   pass all flags S/SA
> }
> anchor "foo" all {
>   pass all flags S/SA
> }
> 
> And to demonstrate a reference specified by an absolute path we can
> add anchor "/foo" inside "foo":
> 
> # echo -e 'anchor "/foo"' | pfctl -a 'foo' -f -
> 
> This obviously gets us an endless loop if we decide to print out
> contents recursively (pfctl -a '*' -sr).  Thankfully this doesn't
> affect ruleset evaluation in the kernel.
> 
> The basic question I have is why do we need this dangerous and (at
> least in my eyes) pretty useless mechanism?
> 
> Any opinions on this?  Does anyone make any sensible use of it?
> Should we try to simplify this by removing ambiguous functionality?
> 
> Cheers,
> Mike

While I've got a few responses on tech supporting me, I should
probably ask misc@ as well for additional scrutiny.  And besides,
I've come up with an example that might simplify ruleset creation
for say multiple customer vlans or rdomains.  Does anyone use it
like that?  This looks like an anchor template that we want to
specify but not use directly in the main ruleset, but it plays
somewhat nicely with "quick" anchors.

(The ruleset below was tested and works correctly)

Cheers,
Mike


block all

anchor "customer1" quick on rdomain 1 {
anchor "/allow-egress"
anchor "/allow-ssh"
anchor "/allow-icmp-echo"
}

anchor "customer2" quick on rdomain 2 {
anchor "/allow-egress"
anchor "/allow-ssh"
}

pass in quick on rdomain 0 proto tcp to (self) port ssh
pass out quick on rdomain 0

# end of ruleset evaluation
block quick

anchor "allow-ssh" {
pass in proto tcp to (self) port ssh
}

anchor "allow-icmp-echo" {
pass in inet proto icmp to (self) icmp-type echoreq code 0
}

anchor "allow-egress" {
pass out
}

Re: nfs server causing lockups lately

[Mike Belopuhov]

On Sun, Apr 13, 2014 at 01:27 -0400, Luke Tidd wrote:
> march 7th snapshot and before, no problem
> 
> next upgrade was april 9th snapshot, nfsd usage for more than a minute
> or so will bring down the system reproducibly.
> 
> tried again with a bit ago (april 12th) snapshot, still crashing
> 
> OpenBSD/amd64 (largemarge.arikui.org) (tty00)
> 
> login: panic: nfs getstream
> Stopped at  Debugger+0x5:   leave
> RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
> IF RUNNING SMP, USE 'mach ddbcpu <#>' AND 'trace' ON OTHER PROCESSORS, TOO.
> DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
> ddb{0}> trace
> Debugger() at Debugger+0x5
> panic() at panic+0xfe
> nfsrv_getstream() at nfsrv_getstream+0x29c
> nfsrv_rcv() at nfsrv_rcv+0x1b3
> tcp_input() at tcp_input+0x1db7
> ip_ours() at ip_ours+0x1e9
> ipv4_input() at ipv4_input+0x362
> ipintr() at ipintr+0x7f
> netintr() at netintr+0x85
> softintr_dispatch() at softintr_dispatch+0x5d
> Xsoftnet() at Xsoftnet+0x2d
> --- interrupt ---
> end trace frame: 0x0, count: -11
> 0x8:

hi,

sounds like an spl got lost somewhere.  can you please run with this
diff and 'sysctl kern.splassert=2' (tweaking in /etc/sysctl.conf is
easier though).

cheers

diff --git sys/nfs/nfs_socket.c sys/nfs/nfs_socket.c
index 6946a28..918ee21 100644
--- sys/nfs/nfs_socket.c
+++ sys/nfs/nfs_socket.c
@@ -1678,10 +1678,12 @@ nfsrv_getstream(struct nfssvc_sock *slp, int waitflag)
char *cp1, *cp2;
int len;
struct mbuf *om, *m2, *recm;
u_int32_t recmark;
 
+   splsoftassert(IPL_SOFTNET);
+
if (slp->ns_flag & SLP_GETSTREAM)
panic("nfs getstream");
slp->ns_flag |= SLP_GETSTREAM;
for (;;) {
if (slp->ns_reclen == 0) {

Re: Packet Filter nat-to issue

[Mike Belopuhov]

On 28 February 2014 10:15, Loïc Blot  wrote:
> Hello,
> i encounter a strange problem today on PF. I don't know if this i normal
> but the result is illogic.
>
> I have this rule:
>
> pass out quick proto tcp from  to port { smtp smtps 587
> imap imaps pop3 pop3s } nat-to $natto_iface
>
> Tables contain IPv4 addresses only.
>
> After applying this rule (i added IPv6 support yesterday), those
> protocols weren't NAT-ed by PF.
>
> By investigating, i found this:
>
> pfctl -sr | grep nat-to
>
> pass out quick inet6 proto tcp from  to any port = 465
> flags S/SA nat-to <__automatic_d309aaac_0> round-robin
>
> Then i look at __automatic_d309aaac_0, because inet6 was strange !
>
> pfctl -t __automatic_d309aaac_1 -T show
>2001:660:3bbb:::2
>fe80::92b1:1cad:fe18:ea18
>
> To resolve this problem i added inet keyword to my rule.
>
> Is this normal ?

yes, you've got what you've asked for.  you should say "pass out quick inet"
if you don't want inet6.

> Maybe a fix was required on pf parser?
>
> Have a nice day
>
>
> --
> Best regards,
>
> Loïc BLOT, Engineering
> UNIX Systems, Security and Network Engineer
> http://www.unix-experience.fr

Re: iked's ikev2 segfaults during connection initiation from strongswan

[Mike Belopuhov]

On 25 September 2013 14:41, LEVAI Daniel  wrote:
> Hi!
>
> I'm trying to setup StrongSwan (oh, the pain...) to iked(8) IPsec.  When
> trying to bring up the connection from the Linux end (ipsec up
> ), the iked(8) at the OpenBSD (5.3-stable) endpoint
> segfaults. I'm trying to use certs and public keys for authentication
> for this host-to-host ESP tunnel connection.
> For the life of me I can not get a coredump from the ikev2 program, but
> attaching gdb to its PID won't give me a bt either because it can't seem
> to load the symbol table. I've recompiled iked from sources with
> CFLAGS=-g and without stripping, but still, no luck.
>

use "CFLAGS=-g -DDEBUG" to disable chroot and generate a core dump.

> The network looks like this:
> [ Linux StrongSwan ] <--> [ NAT gw  ]O--Internetz--O[  
> ]
> | |
> `== IPsec IKEv2 =='
>
> Here is the output of iked -dvv from the start until the sig11.
> I'm sorry about the "anonimization", if it confuses the reader I'll
> gladly elaborate.
>

you can also try "iked -dvvT" and see if that works.

Re: IPSec endpoints won't talk to each other

[Mike Belopuhov]

On 24 September 2013 16:35, Hugo Osvaldo Barrera
 wrote:
> On 2013-09-24 09:44, James Griffin wrote:
>> * Hugo Osvaldo Barrera  [2013-09-24 03:53:46
> -0300]:
>>
>> > Hi,
>> >
>> > I've been experimenting a bit with IPSec and creating a VPN using it.
> I've
>> > been successful, but have encountered an odd issue.
>> >
>> > I've two hosts, linking two networks:
>> >
>> > Host A's /etc/iked.conf:
>> > ikev2 active esp from 172.16.0.0/16 to 172.17.0.0/16 \
>> >   peer 174.136.104.18 psk "a-test-key"
>> >
>> > Host B's /etc/iked.conf:
>> > ikev2 esp from 172.17.0.0/16 to 172.16.0.0/16 \
>> >   peer 190.210.108.249 psk "a-test-key"
>> >
>> > (Of course those are not the real keys).
>> >
>> > I can ssh 172.17.0.1 from the 172.16.0.0 network fine and viceversa.
>> >
>> > So far so good.
>> >
>> > BUT I can't establish any TCP connection from Host A to Host B's public
>> > IP address and viceversa.
>>
>> So you can connect using internal addresses but not using public address.
> Just a thought, but have you opened the necessary ports on your router? What
> is your setup like?
>>
>> [ ... ]
>>
>
> They're both connected directly to the internet with no router in front
> of them. With the tunnel disabled, everything works fine between both.
>
> --
> Hugo Osvaldo Barrera
>
> [demime 1.01d removed an attachment of type application/pgp-signature]
>

could you please tcpdump on enc and real interfaces on host A and see
if your traffic (ssh, ping, traceroute) is sent encrypted or not and
on which interface.  and what does host B receive.

Re: Interface input errors incrementing

[Mike Belopuhov]

On 20 September 2013 08:36, Darren Spruell  wrote:
> Running 5.3 (release) i386 on a soekris net4521 with 2 connected
> sis(4). The device is a router/firewall on a home network with a cable
> Internet connection. One of these interfaces has in the last few weeks
> started to build higher rates of input errors as time increases
> between reboots. This seems to result in intermittent packet loss
> (5-10%) or increased latency (3-4x RTT) for routed traffic. When the
> situation is at it's worst we can reboot the host and then symptoms
> are gone for a short time before starting to occur again. This
> interface connects to an Ethernet tap that connects to a 10/100 LAN
> switch.
>
> Any idea what the issue could be?
>

lots of hardware treats rx queue full event as an rx error.  this
condition is also triggered by the MCLGETI very often.  we have
seen this recently in the bge (if_bge.c rev1.334) and it's very
well possible that it's the same "problem" here.

please try to remove SIS_RXSTAT_OVERRUN from the SIS_RXSTAT_ERROR
in /sys/dev/pci/if_sisreg.h and retry your test.  i'm not 100%
sure it's the case and it's what i think it is since NatSemi
datasheets are rather vague in this regard.  but it might shed
some light on the problem.

Re: Strange packets lost

[Mike Belopuhov]

On 25 September 2013 11:03, Loïc BLOT  wrote:
> Hello all,
> i have searched many options but i haven't any new idea.
>
> I have 4 openbsd routers (2 on each site). Each router create a GRE
> tunnel with it's pair.
>
> Here is the configuration:
>
> | S1R1 --- gre + ospf --- S2R1 |
> LAN S1 (OSPF & RIP) |  | LAN S2 (OSPF & RIP)
> | S1R2 --- gre + ospf --- S2R2 |
>
> The routing rules are correct, ssh, http(s), smtp, ntp, ldap and many
> other protocols works as expected between the two sites.
>
> But i have a problem with my Avaya phones on S2 which need to contact
> the S1 gatekeeper. Some packets are lost, and (by sniffing every
> interface) i don't found where the packets goes.
>
> If i capture LAN S1 link, i have this capture:
>
> 10:06:24.003479 192.168.238.121.56641 > 192.168.106.38.411: S
> 2621611805:2621611805(0) win 5840  0,nop,wscale 4> (DF)
> 10:06:24.003607 192.168.106.38.411 > 192.168.238.121.56641: S
> 3090220105:3090220105(0) ack 2621611806 win 5840 
> (DF)
> 10:06:24.018842 192.168.238.121.56641 > 192.168.106.38.411: . ack 1 win
> 365 (DF)
> 10:06:24.023582 192.168.238.121.56641 > 192.168.106.38.411: P 1:74(73)
> ack 1 win 365 (DF)
> 10:06:24.023710 192.168.106.38.411 > 192.168.238.121.56641: . ack 74 win
> 46 (DF)
> 10:06:24.024086 192.168.106.38.411 > 192.168.238.121.56641: .
> 1:1461(1460) ack 74 win 46 (DF)
> 10:06:24.024329 192.168.106.38.411 > 192.168.238.121.56641: .
> 1461:2921(1460) ack 74 win 46 (DF)
> 10:06:27.017704 192.168.106.38.411 > 192.168.238.121.56641: .
> 1:1461(1460) ack 74 win 46 (DF)
> 10:06:33.017772 192.168.106.38.411 > 192.168.238.121.56641: .
> 1:1461(1460) ack 74 win 46 (DF)
> 10:06:45.017907 192.168.106.38.411 > 192.168.238.121.56641: .
> 1:1461(1460) ack 74 win 46 (DF)
> 10:07:09.018198 192.168.106.38.411 > 192.168.238.121.56641: .
> 1:1461(1460) ack 74 win 46 (DF)
> 10:07:57.018732 192.168.106.38.411 > 192.168.238.121.56641: .
> 1:1461(1460) ack 74 win 46 (DF)
> 10:08:24.019074 192.168.106.38.411 > 192.168.238.121.56641: FP
> 2921:4273(1352) ack 74 win 46 (DF)
> 10:08:24.034803 192.168.238.121.56641 > 192.168.106.38.411: . ack 1 win
> 365 (DF)
>
> If i capture the GRE tunnel i have this capture:
>
> 10:06:23.987975 192.168.238.121.56641 > 192.168.106.38.411: S
> 2621611805:2621611805(0) win 5840  0,nop,wscale 4> (DF)
> 10:06:24.003614 192.168.106.38.411 > 192.168.238.121.56641: S
> 3090220105:3090220105(0) ack 2621611806 win 5840 
> (DF)
> 10:06:24.018833 192.168.238.121.56641 > 192.168.106.38.411: . ack 1 win
> 365 (DF)
> 10:06:24.023573 192.168.238.121.56641 > 192.168.106.38.411: P 1:74(73)
> ack 1 win 365 (DF)
> 10:06:24.023716 192.168.106.38.411 > 192.168.238.121.56641: . ack 74 win
> 46 (DF)
> 10:08:24.019083 192.168.106.38.411 > 192.168.238.121.56641: FP
> 2921:4273(1352) ack 74 win 46 (DF)
> 10:08:24.034793 192.168.238.121.56641 > 192.168.106.38.411: . ack 1 win
> 365 (DF)
>
> A part of the TCP transaction disappear and i don't know why.
> Have you got ideas ???
>

this looks like a classical mtu problem.  gre tunnel lowers the mtu
and your tcp traffic uses mss of 1460 bytes and sets DF.  therefore
it gets dropped once the router figures out it can't send that much
data over the gre link.

possible solutions are using path mtu discovery on clients or making
sure their mtu is less than 1500 or doing forced fragmentation and
defragmentation on the router or configuring the application to use
smaller mss value (setsockopt TCP_MAXSEG).

Re: 10GBit OpenBSD Firewall

[Mike Belopuhov]

On 2 September 2013 12:26, Andy  wrote:
> Hi everyone,
>
> I know this gets discussed a lot and I don't want to re-tread old
> ground, but I am in the process of specifying firewall hardware for
> 10Gbit and would like to check the rules-of-thumb I am using..
>
> We would like to reach 1 Million pps, and would like to forward at least
> up-to the current ALTQ limit (~4Gbit) (we have to use queuing).
>
> We should be looking for ?;
> Single fastest CPU speed possible
> Hyper-threading disabled
> Turbo mode enabled (2 cores only, one for the kernel and one for user-land)
> Big L3 cache
> Unbuffered memory (buffering slower)
> Sandy Bridge or Sand Bridge EP (2011 socket has quad channel memory access)
> 1600MHz memory
> Intel X520-SR2 NIC
>
> I am currently looking at the Supermicro X10SLM-F mainboard (
> http://shop.transtec.co.uk/GB/E/products/server/infrastructure_server.html?mod=prod&name=SI1180A304R).
>
> This is only Sandy Bridge (1155 socket, so dual channel memory access),
> but has fully unbuffered memory access (larger boards support 520GB RAM
> and so are buffered etc).
> Getting an E3-1280 v3 CPU (cannot afford the 1290 v3).
> And 1600MHz unbuffered ECC RAM
>
> Am I missing anything important, or is there some hardware combination
> that anyone has had some great success with for a reasonable price
> (~2,000 GBP)?
>
> Thanks for your time and I'm sorry for bringing this question up again,
> but hardware changes regularly and I greatly value the opinions of
> others on this list.
> Regards, Andy.
>

Hi Andy,

While your thinking is right with the current state of affairs
you won't be able to reach 1Mpps even w/o pf due to the inability
to utilize more than one CPU for network processing.  With Pf and
ALTQ performance is plummeting and you can only hope for half of
the forwarding rate.

We are trying to address problems with MP networking right now,
but due to the lack of manpower the progress is slow.

With best regards,
Mike

Re: BCM5719C/BCM5720 partially working

[Mike Belopuhov]

On Fri, May 24, 2013 at 15:48 +0200, David Imhoff wrote:
> Mike Belopuhov wrote on 2013-05-23 21:55:
> >On Thu, May 23, 2013 at 11:04 +0200, David Imhoff wrote:
> >>The problem seems to be in the Auto polling of the mac link state.
> >>ifconfig shows the correct link state, but the BGE_STS_LINK bit in
> >>the driver is never set. Debugging the interrupt routine I found the
> >>driver never receives a link state changed attention, instead the
> >>last interrupt indicates a auto-poll error and bit 0 of register
> >>BGE_AUTOPOLL_STS is set.
> >>Looking at the FreeBSD and Linux drivers I found that they don't use
> >>auto-polling. Also the programmers guide of the BCM5718 family(page
> >>204) doesn't describe the auto-polling method anymore, as was done
> >>in the PG of other BCM57xx.
> >I agree that this might be a better solution than what I had:
> >setting the BGE_STS_LINK_EVT flag back in bge_link_upd to force
> >the bge_intr into running it every time untill it gets the link;
> >and it seems to work.
> 
> I also tried fixing auto-polling mode, but it kept giving me a lot
> of bogus interrupts(few hundred) after bringing the link up. Using
> the LNKRDY signal just gives 2 interrupts.
> 
> >>So I disabled Auto-polling for the BCM5718 family. This made all
> >>ports work on my Dell R320(BCM5720) and BCM5719. My BCM5719 still
> >>doesn't fully work because now i run into bge_watchdog reset's when
> >>sending big packet, the same issue reported by Robert Young on 5
> >>April.
> >>
> >Right, it seems like the TX completion interrupts never arrive
> >for large packets.  Forcing txeof doesn't help.  In fact I can
> >see some funny things going on here: for small ping packets 1
> >out of 1 tx buffer descriptors contain an mbuf to scratch, for
> >170 byte packets 1 out of 2 tx bd's contain an mbuf, for 250
> >byte packets 1 out of 3 tx bd's contain an mbuf, for 500 byte
> >packets TX completion interrupt never arrives.
> 
> In my setup the problem seems RX related. Sending a 1450
> byte ping to a non-existing host using a forced arp entry works.
> But when i do the same on the remote host and run tcpdump on
> the BCM5719 interface, the controller crashes and the
> watchdog will trigger when trying to send a (small) packet.
> 
> My steps to reproduce:
>  - connect bge host to a remote host using a hub or direct link
>  - on bge host:
># ifconfig bge0 192.168.1.1
># tcpdump -n -i bge0
>  - on remote host:
># ifconfig ??? 192.168.1.2
># arp -s 192.168.1.4 02:11:11:11:11:14
># ping -c 1 192.168.1.4
># ping -s 1450 -c 5 192.168.1.4
>  - on bge host, tcpdump shows only 1 packet received
>  - on bge host: ping -c 1 192.168.1.2
>  - wait a bit, and notice the watchdog reset
> 

Hmm, rx is screwed up for sure. take a look at "-s 200": it won't
lock up, but messages are received incorrectly, part of the payload
gets duplicated.  

So it looks like that chip gets hung when it's receiving, but since
we don't transmit anything and don't reset the timer watchdog fires.


> >I had a suspicion that this is because ASF mode is turned on
> >and we fail to share a port with IPMI firmware, but right now
> >I'm more inclined to believe that this is just another TX bug.
> 
> Both of my BCM5719 cards are PCI/E expansion cards, so no port
> sharing on them.
> 

It confirms my suspicion that it's not IPMI related.

> >>I also noticed if_bge.c line 2293:
> >>BGE_SETBIT(sc, BGE_MI_MODE, BGE_MIMODE_AUTOPOLL|10<<16);
> >>Does any one have a clue why the (10<<16) is there? it changes the
> >>MI clock. But why? and why OR 0xA with the default 0xC, instead of
> >>0x2? FreeBSD and Linux use the default of 0xC for the MI Clock.
> >>
> >I think it's a remnant of the original code and it should be
> >BGE_MIMODE_BASE (as in "the other frequency").
> 
> But since BGE_MIMODE_BASE is the default value after reset and
> OR'ing the value wouldn't be correct, I suggest to just remove
> the '|(10<<16)'.
> 

I'm not OR'ing it in, I'm overwriting the value.

> >Please inspect my diff below; I believe it's more complete.
> >diff --git sys/dev/pci/if_bge.c sys/dev/pci/if_bge.c
> >index 1d37192..d357443 100644
> >--- sys/dev/pci/if_bge.c
> >+++ sys/dev/pci/if_bge.c
> >@@ -1055,10 +1055,22 @@ bge_miibus_statchg(struct device *dev)
> > (mii->mii_media_active & IFM_ETH_FMASK) != sc->bge_flowflags) {
> > sc-&

Re: BCM5719C/BCM5720 partially working

[Mike Belopuhov]

On Thu, May 23, 2013 at 11:04 +0200, David Imhoff wrote:
> Hi,
> 
> Having finally found the BCM5718 Programmers guide from
> http://www.broadcom.com/support/ethernet_nic/open_source.php I
> managed to get a bit further with this problem.
> 
> The problem seems to be in the Auto polling of the mac link state.
> ifconfig shows the correct link state, but the BGE_STS_LINK bit in
> the driver is never set. Debugging the interrupt routine I found the
> driver never receives a link state changed attention, instead the
> last interrupt indicates a auto-poll error and bit 0 of register
> BGE_AUTOPOLL_STS is set.
> 
> Looking at the FreeBSD and Linux drivers I found that they don't use
> auto-polling. Also the programmers guide of the BCM5718 family(page
> 204) doesn't describe the auto-polling method anymore, as was done
> in the PG of other BCM57xx.
> 

I agree that this might be a better solution than what I had:
setting the BGE_STS_LINK_EVT flag back in bge_link_upd to force
the bge_intr into running it every time untill it gets the link;
and it seems to work.

> So I disabled Auto-polling for the BCM5718 family. This made all
> ports work on my Dell R320(BCM5720) and BCM5719. My BCM5719 still
> doesn't fully work because now i run into bge_watchdog reset's when
> sending big packet, the same issue reported by Robert Young on 5
> April.
> 

Right, it seems like the TX completion interrupts never arrive
for large packets.  Forcing txeof doesn't help.  In fact I can
see some funny things going on here: for small ping packets 1
out of 1 tx buffer descriptors contain an mbuf to scratch, for
170 byte packets 1 out of 2 tx bd's contain an mbuf, for 250
byte packets 1 out of 3 tx bd's contain an mbuf, for 500 byte
packets TX completion interrupt never arrives.

I had a suspicion that this is because ASF mode is turned on
and we fail to share a port with IPMI firmware, but right now
I'm more inclined to believe that this is just another TX bug.

ChipID here is 0x5719001 as well but APE firmware NCSI 1.1.15.0.

> Attached is a patch that disables mac polling for the BCM5717, 5719
> and 5720. Tested on BCM5719, BCM5720 and BCM5750 C1.
> 
> I also noticed if_bge.c line 2293:
> BGE_SETBIT(sc, BGE_MI_MODE, BGE_MIMODE_AUTOPOLL|10<<16);
> Does any one have a clue why the (10<<16) is there? it changes the
> MI clock. But why? and why OR 0xA with the default 0xC, instead of
> 0x2? FreeBSD and Linux use the default of 0xC for the MI Clock.
> 

I think it's a remnant of the original code and it should be
BGE_MIMODE_BASE (as in "the other frequency").

> Kind regards,
> 
> David
> 
> Index: sys/dev/pci/if_bge.c
> ===
> RCS file: /cvs/src/sys/dev/pci/if_bge.c,v
> retrieving revision 1.320
> diff -u -r1.320 if_bge.c
> --- sys/dev/pci/if_bge.c  4 Mar 2013 01:33:18 -   1.320
> +++ sys/dev/pci/if_bge.c  21 May 2013 13:18:36 -
> @@ -2288,7 +2288,9 @@
>   /* Enable PHY auto polling (for MII/GMII only) */
>   if (sc->bge_flags & BGE_PHY_FIBER_TBI) {
>   CSR_WRITE_4(sc, BGE_MI_STS, BGE_MISTS_LINK);
> - } else {
> + } else if (!(BGE_ASICREV(sc->bge_chipid) == BGE_ASICREV_BCM5717 ||
> + BGE_ASICREV(sc->bge_chipid) == BGE_ASICREV_BCM5719 ||
> + BGE_ASICREV(sc->bge_chipid) == BGE_ASICREV_BCM5720)) {
>   BGE_STS_SETBIT(sc, BGE_STS_AUTOPOLL);
>   BGE_SETBIT(sc, BGE_MI_MODE, BGE_MIMODE_AUTOPOLL|10<<16);
>   if (BGE_ASICREV(sc->bge_chipid) == BGE_ASICREV_BCM5700)
> @@ -4386,18 +4388,13 @@
>   if_link_state_change(ifp);
>   ifp->if_baudrate = 0;
>   }
> - /*
> -  * Discard link events for MII/GMII cards if MI auto-polling
> disabled.
> -  * This should not happen since mii callouts are locked now, but
> -  * we keep this check for debug.
> -  */
> - } else if (BGE_STS_BIT(sc, BGE_STS_AUTOPOLL)) {
> + } else {
>   /*

I think you've missed some important bits here.

>* Some broken BCM chips have BGE_STATFLAG_LINKSTATE_CHANGED bit
>* in status word always set. Workaround this bug by reading
>* PHY link status directly.
>*/
> - link = (CSR_READ_4(sc, BGE_MI_STS) & BGE_MISTS_LINK)?
> + link = (CSR_READ_4(sc, BGE_TX_STS) & BGE_TXSTAT_LINK_UP)?
>   BGE_STS_LINK : 0;
> 
>   if (BGE_STS_BIT(sc, BGE_STS_LINK) != link) {
> 
> 

Why did you change BGE_MI_STS to BGE_TX_STS?

Please inspect my diff below; I believe it's more complete.

diff --git sys/dev/pci/if_bge.c sys/dev/pci/if_bge.c
index 1d37192..d357443 100644
--- sys/dev/pci/if_bge.c
+++ sys/dev/pci/if_bge.c
@@ -1055,10 +1055,22 @@ bge_miibus_statchg(struct device *dev)
(mii->mii_media_active & IFM_ETH_FMASK) != sc->bge_flowflags) {
sc->bge_flowflags = mii->mii_media_active & IFM_ETH_FMASK;
mii

Re: Crashes with IPsec + GRE (+ crypto softraid)

[Mike Belopuhov]

On Mon, Feb 18, 2013 at 15:45 +0100, Michael wrote:
> Hi all,
> 
> after having a somewhat weird problem for a while now I hope someone can
> help me. _Sorry_ for the really lengthy mail but it is kind of complex
> to describe.
> 
> dmesg and other information can be found at the end.
> 
> The problem in short:
> Server keeps crashing hard (even ddb won't respond) after a more or less
> random time when using a GRE tunnel inside IPsec (transport mode).
> 
> Elaboration:
> The setup consists of 3 OpenBSD boxes. One running OpenBSD 5.1 and the
> other two OpenBSD 5.2 (upgrading from 5.1 to 5.2 didn't fix the issue.
> Each box is directly connected to the internet with a public IP in a
> different physical location (1x US, 2x DE).
> 
> All 3 boxes are connected with an IPsec tunnel, like a triangle (a<->b,
> b<->c, a<->c). Inside the IPsec tunnel is a GRE tunnel with OSPF on top
> for dynamic routing.
> 
> Two of those systems got a softraid0 crypto partition running, the third
> one doesn't. (More on why that might be important later).
> 
> When all 3 boxes are powered up everything is working perfectly fine,
> but after some random interval (can be minutes, can be days) one or two
> of the boxes crash, showing the ddb console but not letting me type
> anything in.
> 
> When the 2 boxes are rebooted, the game starts anew.
> 
> In case only one of the boxes crashed, I can by now predict a 99% change
> that the second one will crash shortly after the first one was fully
> rebooted.
> 
> Now, it is only ever 1 or 2 boxes that crash and so far it never has
> been the box WITHOUT the softraid0 crypto volume.
> 
> Out of curiosity I also created a crypto volume on the third box and put
> it to some use (squid cache parition) and sure enough, now the third box
> sometimes crashed too.
> 
> When doing some tests (with only having two systems using a crypto
> partition) I also noticed that there are no crashes at all if there is
> only a single IPsec tunnel active between two of the boxes (one box with
> crypto partition, the other without) and GRE encrypted inside and the
> other GRE tunnels are unencrypted.
> 
> To not play around too much with the production systems I tried to
> replicate the issue with 3 VirtualBox VM and the latest OpenBSD 5.3
> snapshot, but VirtualBox instantly throws a GURU MEDITATION ERROR
> whenever I try to push a file (1 MB is enough) over an encrypted GRE
> tunnel using scp or netcat from one machine to the other. When I turn of
> IPsec, the transfer works, no crashing.
> 
> I only have console access to two of the boxes and whenever a system
> crashes, it displays a very short message which is always a little
> different, the only consistent part is the mentioning of "Stopped at
> __mp_lock". That system is running OpenBSD 5.1, bsd.mp.
> 
> I hope someone has an idea what might be going on.
> 
> Thanks,
> Michael
> 
> PS: If someone wants to play around with my three VirtualBox test VMs
> you can download them here:
> 
> https://ssl.bsdhost.eu/owncloud/public.php?service=files&t=17b7472546546a617a3358ef9d953a4c
> 

hi,

there appears to be some spls missing in net/if_gre.c code.
netinet/ip_gre.c looks almose sane (gre_input should be
called at splsoftnet from ip_input), yet gre_usrreq calls
rip_usrreq that doesn't do splsoftnet itself (yuck!).
reminds of the recent raw_usrreq change.  anyways, both
diffs are attached.  please try them and see if they help
out.

cheers

diff --git sys/net/if_gre.c sys/net/if_gre.c
index 7a9..84f0f0e 100644
--- sys/net/if_gre.c
+++ sys/net/if_gre.c
@@ -679,12 +679,15 @@ void
 gre_keepalive(void *arg)
 {
struct gre_softc *sc = arg;
+   int s;
 
if (!sc->sc_ka_timout)
return;
 
sc->sc_ka_state = GRE_STATE_DOWN;
+   s = splnet();
gre_link_state(sc);
+   splx(s);
 }
 
 void
@@ -747,6 +750,8 @@ gre_send_keepalive(void *arg)
 void
 gre_recv_keepalive(struct gre_softc *sc)
 {
+   int s;
+
if (!sc->sc_ka_timout)
return;
 
@@ -762,7 +767,9 @@ gre_recv_keepalive(struct gre_softc *sc)
case GRE_STATE_HOLD:
if (--sc->sc_ka_holdcnt < 1) {
sc->sc_ka_state = GRE_STATE_UP;
+   s = splnet();
gre_link_state(sc);
+   splx(s);
}
break;
case GRE_STATE_UP:
diff --git sys/netinet/raw_ip.c sys/netinet/raw_ip.c
index 61285a8..050529f 100644
--- sys/netinet/raw_ip.c
+++ sys/netinet/raw_ip.c
@@ -396,7 +396,7 @@ int
 rip_usrreq(struct socket *so, int req, struct mbuf *m, struct mbuf *nam,
 struct mbuf *control, struct proc *p)
 {
-   int error = 0;
+   int s, error = 0;
struct inpcb *inp = sotoinpcb(so);
 #ifdef MROUTING
extern struct socket *ip_mrouter;
@@ -410,6 +410,7 @@ rip_usrreq(struct socket *so, int req, struct mbuf *m, 
struct mbuf *nam,
goto release;
}
 
+   s = splsoftnet();
   

Re: OpenBSD5.3-beta, kernel panic : pf.conf with once option

[Mike Belopuhov]

On 22 February 2013 14:02, Wesley M.A.  wrote:
> Hi,
>
> I'm running :
> kern.version=OpenBSD 5.3-beta (GENERIC) #33: Fri Feb 15 17:03:34 MST 2013
> dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
>
> 2 network cards : bge0 and axe0
>
> "pfctl -vf /etc/pf.conf" load the ruleset, but just after i do "telnet
> hostname.on.internet 110" this on a workstation, i have a kernel panic on my
> OpenBSD gateway :
>
> uvm_fault(0xd0a51660,0x0, 0, 1) -> e
> Kernel: page fault trap, code=0
> Stopped at pf_purge_rule +0x11: mov 0x10(%ebx),%eax
>
>
> my pf.conf :
> 
> ports_tcp="{80 25 443 587 995 21}"
> set skip on lo
> match out on egress inet from bge0:network to any nat-to egress
> block log all
> pass out
> pass in on bge0 inet proto icmp icmp-type echoreq
> pass in on bge0 inet proto tcp from bge0:network to any port $ports_tcp
> pass in on bge0 inet proto tcp from bge0:network to any port 110 once
> pass in on bge0 inet proto udp from bge0:network to any port domain
> pass in on bge0 inet proto tcp from bge0:network to any port 22
>

short answer: don't do that.  you have to use an anchor.

regarding the actual crash -- i'll look at it asap.

Re: Low latency and interrupt mitigation

[Mike Belopuhov]

On 11 February 2013 13:22, Dan Shechter  wrote:
> Thanks.
>
> If interrupt mitigation disabling won't be enough I will have do
> decide: freebs with netmap or openbsd with if_ethersubr.c
> modification.
>
> netmap looks much easier to code. No need to do the mbuf dance (or I
> hope that with netmap there would be no such need).
>
> BTW, would HW TX IP/UDP checksum reduce the latency?

I don't know exactly, but compared to the amount of work
done by bpf/pcap and syscalls (at least write), I'd say
it's negligible.

> Best regards,
> Dan
>
>
> On Mon, Feb 11, 2013 at 2:11 PM, Mike Belopuhov  wrote:
>> On 11 February 2013 12:53, Dan Shechter  wrote:
>>> Thanks,
>>>
>>> Are there any other tips to reduce latency?
>>>
>>
>> Using pcap means copying packets, so I'd say you want to
>> put your code into the kernel to avoid copying and maybe
>> queueing as well, but this is not something that can be
>> trivially explained and is certainly not something that
>> OpenBSD supports or would want to support out of the box.
>>
>> FreeBSD netmap interface would be helpful in your situation
>> but is not currently supported by OpenBSD.
>>
>> Cheers,
>> Mike
>>
>>>
>>> Best regards,
>>> Dan
>>>
>>>
>>> On Mon, Feb 11, 2013 at 1:43 PM, Mike Belopuhov  wrote:
>>>> On 11 February 2013 12:19, Dan Shechter  wrote:
>>>>> Hi All,
>>>>>
>>>>> I have a very latency sensitive application. I need to move packets
>>>>> from one interface to another
>>>>>
>>>>> I am using a quad 1000/pro Intel NIC. pcidump shows 82571EB
>>>>>
>>>>> My latency sensitive application reads packets from one em interface
>>>>> using libpcap and sends packet to another em interface using udp
>>>>> socket.
>>>>>
>>>>> Does em has interrupt mitigation enabled?
>>>>
>>>> It does.
>>>>
>>>>> If interrupt mitigation is enabled would disabling it will reduce latency?
>>>>
>>>> It would.
>>>>
>>>>> How can I disable the interrupt mitigation for em NICs?
>>>>
>>>> I suggest you try increasing MAX_INTS_PER_SEC define by the factor
>>>> or two in /sys/dev/pci/if_em.h. In case it doesn't suite your needs you
>>>> can try commenting out this line:
>>>>
>>>> E1000_WRITE_REG(&sc->hw, ITR, DEFAULT_ITR);
>>>>
>>>>> What would change in 5.3?
>>>>>
>>>>
>>>> Nothing regarding interrupt moderation.
>>>>
>>>>> Best regards,
>>>>> Dan
>>>>>
>>>>
>>>> Cheers,
>>>> Mike

Re: Low latency and interrupt mitigation

[Mike Belopuhov]

On 11 February 2013 12:53, Dan Shechter  wrote:
> Thanks,
>
> Are there any other tips to reduce latency?
>

Using pcap means copying packets, so I'd say you want to
put your code into the kernel to avoid copying and maybe
queueing as well, but this is not something that can be
trivially explained and is certainly not something that
OpenBSD supports or would want to support out of the box.

FreeBSD netmap interface would be helpful in your situation
but is not currently supported by OpenBSD.

Cheers,
Mike

>
> Best regards,
> Dan
>
>
> On Mon, Feb 11, 2013 at 1:43 PM, Mike Belopuhov  wrote:
>> On 11 February 2013 12:19, Dan Shechter  wrote:
>>> Hi All,
>>>
>>> I have a very latency sensitive application. I need to move packets
>>> from one interface to another
>>>
>>> I am using a quad 1000/pro Intel NIC. pcidump shows 82571EB
>>>
>>> My latency sensitive application reads packets from one em interface
>>> using libpcap and sends packet to another em interface using udp
>>> socket.
>>>
>>> Does em has interrupt mitigation enabled?
>>
>> It does.
>>
>>> If interrupt mitigation is enabled would disabling it will reduce latency?
>>
>> It would.
>>
>>> How can I disable the interrupt mitigation for em NICs?
>>
>> I suggest you try increasing MAX_INTS_PER_SEC define by the factor
>> or two in /sys/dev/pci/if_em.h. In case it doesn't suite your needs you
>> can try commenting out this line:
>>
>> E1000_WRITE_REG(&sc->hw, ITR, DEFAULT_ITR);
>>
>>> What would change in 5.3?
>>>
>>
>> Nothing regarding interrupt moderation.
>>
>>> Best regards,
>>> Dan
>>>
>>
>> Cheers,
>> Mike

Re: Low latency and interrupt mitigation

[Mike Belopuhov]

On 11 February 2013 12:19, Dan Shechter  wrote:
> Hi All,
>
> I have a very latency sensitive application. I need to move packets
> from one interface to another
>
> I am using a quad 1000/pro Intel NIC. pcidump shows 82571EB
>
> My latency sensitive application reads packets from one em interface
> using libpcap and sends packet to another em interface using udp
> socket.
>
> Does em has interrupt mitigation enabled?

It does.

> If interrupt mitigation is enabled would disabling it will reduce latency?

It would.

> How can I disable the interrupt mitigation for em NICs?

I suggest you try increasing MAX_INTS_PER_SEC define by the factor
or two in /sys/dev/pci/if_em.h. In case it doesn't suite your needs you
can try commenting out this line:

E1000_WRITE_REG(&sc->hw, ITR, DEFAULT_ITR);

> What would change in 5.3?
>

Nothing regarding interrupt moderation.

> Best regards,
> Dan
>

Cheers,
Mike

Re: OpenBSD 5.2 isakmpd sequence number problem

[Mike Belopuhov]

On Wed, Jan 30, 2013 at 14:59 +0100, Bartosz Brzozowski wrote:
> Hi Misc.
> 
> I have two location A i B in my lab.
> In the location A there is isakmpd + carp + pfsync + sasync cluster
> on which there is installed OpenBSD 5.2 GENERIC.MP#339 i386
> In the location B there is single OpenBSD 5.2 GENERIC#278 i386 installation.
> 
> I have successfully established IPsec site-to-site connection
> between those two location and everything works fine until failover
> occurs in the cluster in location A ( I am  using ifconfig -g carp
> carpdemote 128 command on the MASTER node to force the failover).
> When failover occurs pfsync increase sequence number on the new
> MASTER node by 16384 and cluster continues sending ESP packets with
> the new, increased value (for example if the last seq number on the
> old MASTER was sent with the value equal to 100 the new MASTER will
> send his first packet with seq number equal to 16484).
> On the OpenBSD 5.2 in location B, using tcpdump I am able to see ESP
> packets, with the new increased seq number, on the phisical
> interface, but I do not see any transmition on enc0 interface. The
> ESP packets are continuously going out of the cluster A and are
> continuously, silently dropped on OpenBSD in location B. All IPsec
> transmition is broken until new SA are established and seq number is
> reset to 0.
> 
> I noted that starting from version OpenBSD 5.2 there is added
> support for Extended Sequence Numbers in the IPsec stack, so I go
> back with the version of OpenBSD on the server located in B from 5.2
> to 5.1 GENERIC#160 i386, copied all configuration files to it,
> established once again IPsec beetwen A and B and from that point
> everything started to work perfectly. Now I am able to switch nodes
> in cluster A (ofcourse seq number, everytime I switch nodes is
> increased by 16384) and OpenBSD 5.1 in location B successfully
> decrypt and encrypt connection. I am able to see packets, going in
> both directions, on interface enc0 as well I am able to see ESP
> packet on phisical interface on OpenBSD 5.1.
> 
> Is there bug in inplementation ESN in new IPsec stack on OpenBSD 5.2
> or do I need additional configuration to make above setup to started
> to work properly.
> Please let me know if You need any additional information.
> 
> Thanks for advise regards,
> Bartosz Brzozowski
> 


hi,

your observations are correct.  the problem should be resolved by
this commit:
http://www.openbsd.org/cgi-bin/cvsweb/src/sys/net/if_pfsync.c.diff?r1=1.189;r2=1.190

trying 5.3 beta might be a good idea as well to make sure it doesn't
have the aforementioned problem.  i'm also waiting for a review of a
change that will improve the anti-replay window handing and merge in
a diff from markus@ to increase the anti-replay window to 2100 packets.
i'll update this thread once the change will be in.  but in case you
have some time to play with it, you could try the diff below.

diff --git sys/netinet/ip_ah.c sys/netinet/ip_ah.c
index 243c82a..9ea6534 100644
--- sys/netinet/ip_ah.c
+++ sys/netinet/ip_ah.c
@@ -149,7 +149,6 @@ ah_init(struct tdb *tdbp, struct xformsw *xsp, struct 
ipsecinit *ii)
 
tdbp->tdb_xform = xsp;
tdbp->tdb_authalgxform = thash;
-   tdbp->tdb_bitmap = 0;
tdbp->tdb_rpl = AH_HMAC_INITIAL_RPL;
 
DPRINTF(("ah_init(): initialized TDB with hash algorithm %s\n",
diff --git sys/netinet/ip_esp.c sys/netinet/ip_esp.c
index 40553c9..77e16ae 100644
--- sys/netinet/ip_esp.c
+++ sys/netinet/ip_esp.c
@@ -242,7 +242,6 @@ esp_init(struct tdb *tdbp, struct xformsw *xsp, struct 
ipsecinit *ii)
}
 
tdbp->tdb_xform = xsp;
-   tdbp->tdb_bitmap = 0;
tdbp->tdb_rpl = AH_HMAC_INITIAL_RPL;
 
/* Initialize crypto session */
@@ -1114,33 +1113,7 @@ esp_output_cb(void *op)
return error;
 }
 
-static __inline int
-checkreplay(u_int64_t *bitmap, u_int32_t diff)
-{
-   if (*bitmap & (1ULL << diff))
-   return (1);
-   return (0);
-}
-
-static __inline void
-setreplay(u_int64_t *bitmap, u_int32_t diff, u_int32_t window, int wupdate)
-{
-   if (wupdate) {
-   if (diff < window)
-   *bitmap = ((*bitmap) << diff) | 1;
-   else
-   *bitmap = 1;
-   } else
-   *bitmap |= 1ULL << diff;
-}
-
-/*
- * To prevent ESN desynchronization replay distance specifies maximum
- * valid difference between the received SN and the last authenticated
- * one.  It's arbitrary chosen to be 1000 packets, meaning that only
- * up to 999 packets can be lost.
- */
-#define REPLAY_DISTANCE (1000)
+#define SEEN_SIZE  howmany(TDB_REPLAYMAX, 32)
 
 /*
  * return 0 on success
@@ -1153,21 +1126,26 @@ checkreplaywindow(struct tdb *tdb, u_int32_t seq, 
u_int32_t *seqhigh,
 int commit)
 {
u_int32_t   tl, th, wl;
-   u_int32_t   seqh, diff;
-   u_int32_t   window = tdb->tdb_wnd;
-   u_int64_t   *bitmap = &tdb->tdb_bitmap;
- 

Re: OpenBSD does not initiate ipsec connection

[Mike Belopuhov]

On Sun, Oct 28, 2012 at 7:49 AM, Erwin Schliske
 wrote:
>>> But now I have figured out what I have to change to bring up the tunnels 
>>> after loading the config with ipsecctl.
>>>
>>> I have to disable sasyncd, which if enabled causes to start isakmpd with 
>>> parameter S. If isakmpd starts without this parameter the tunnels come up 
>>> and work smoothly.
>>>
>>> So the question. Is this a know behaviour, that isakmpd switches to passive 
>>> if sasyncd is enabled? Or is this a bug?
>>
>> I have seen this before. In my experience, in the end the -S parameter
>> works, but it might take a while before isakmpd realises it is running
>> on the master. Never have figured out why it takes long some of the
>> time.
>
> That would be good enough. But on my gateway the tunnel never comes up.
>
> Erwin
>

Hi,

Do you happen to have addresses on the interface below carp?
In case you do, you should instruct isakmpd to use carp address
by adding the following lines to /etc/isakmpd/isakmpd.conf:

[General]
Listen-on=

Make sure to set local address in ike rules in ipsec.conf as well.

Hope this helps.

Cheers,
Mike

Re: Slow VPN Performance

[Mike Belopuhov]

On Tue, Oct 23, 2012 at 10:18 AM, Michael Sideris  wrote:
> While I am not required to comply with any particular crypto
> standards, I have NFS data passing through that link which I would
> classify asfairly sensitive.

hmm, if you're using udp mounts for NFS you might want to try tcp
instead.

> That being said, I am not sure how to
> check the re-keying frequency except watching `ipsecctl -m`.

that depends on the isakmpd settings. right now you're probably
using default ones which are negotiated with your peer. this is
what isakmpd.conf(5) has to say about it:

The Quick Mode lifetime defaults to 20 minutes (minimum 60
seconds, maximum 1 day).

this is rather a broad range, so you might want to shorten it.
look for the "Default-phase-2-lifetime" parameter in the man
page for the /etc/isakmpd/isakmpd.conf.

> I am not
> sure if PFS is enabled by default on a stock OpenBSD 5.1 installation
> so I would appreciate it if you could tell me how I can check that.
>

it is, unless you disable it with the "group none" in the "quick"
configuration options.

> Performance wise I would be happy if I could squeeze ~100 out of the
> 150Mbit/s link. At the moment I am struggling to reach ~100Mbit/s and
> that is with hmac-md5. I would like to find a reasonable balance
> between performance and security but it seems that hmac-sha2-256 is
> too "expensive" for my hardware.

unfortunately it is expensive for any hardware that's why aes-gcm
was developed.

> I really thought dual Xeons @ 2.8GHz would be up to the task.
>

the "dual" part doesn't help as much as it could though.

in any case, i suggest you play with nfs tcp mounts and mss fixups.
otherwise you might be loosing performance where you shouldn't.

trying the snapshot out might also give an opportunity to learn if
some of the performance changes that were committed are helpful
in your setup.

> On Mon, Oct 22, 2012 at 6:41 PM, Mike Belopuhov  wrote:
>> On Mon, Oct 22, 2012 at 4:10 PM, Michael Sideris  wrote:
>>> It seems that changing to hmac-md5 boosted network throughput from
>>> ~50Mbit/s to ~100Mbit/s which is decent and reasonable. I am going to
>>> experiment a bit further with `scrub` options in pf.conf to see if I
>>> can squeeze more performance out of the link. The question now
>>> ishow much is security affected by using hmac-md5 vs the default
>>> hmac-sha2-256?
>>
>> It's more a question of how often do you rekey? You also should not
>> disable Perfect Forward Secrecy that recomputes DH values every
>> time you renew your phase 2 key. And while there are no known
>> serious attacks on HMAC-MD5 it all depends on how important the
>> data that you're protecting is and if you have to be compliant with
>> any regulations that might mandate use of SHA2.
>>
>>>  Should I consider using better CPUs on the servers in
>>> order to gain better performance through a stronger algorithm?
>>>
>>
>> You can get 600-750Mbps (depending on the CPU speed) in the
>> AES-NI enabled setup (using AES-GCM that is).
>>
>>> On Mon, Oct 22, 2012 at 2:58 PM, Mike Belopuhov  wrote:
>>>> On Tue, Oct 16, 2012 at 11:43 AM, Michael Sideris  wrote:
>>>>> Both endpoints run stock OpenBSD 5.1 (amd64). We use the VPN link to
>>>>> manage our platform remotely and perform daily backups. G-VPN runs on
>>>>> a 150Mbit/s link while L-VPN on a 1Gbit/s link. On one hand, our VPN
>>>>> setup runs really nicely. The connections are routed properly, pf is
>>>>> godsent and authpf works wonders. On the other hand, network
>>>>> throughput over the VPN tunnel never exceeds 3.4MB/s (ftp, scp, rsync,
>>>>> etc...)
>>>>>
>>>>> I welcome any suggestions. Keep in mind that this is our production
>>>>> VPN tunnel, so I cannot shut it down at will. Thanks in advance.
>>>>>
>>>>> ---
>>>>> Mike
>>>>>
>>>>
>>>> Hi,
>>>>
>>>> I suggest a couple of changes:
>>>>
>>>>  1) use cheaper hash function (md5 or at least sha1)
>>>>  2) use mss fixup so that your packets don't get fragmented
>>>>
>>>> The first point relates to your "ike" rules in ipsec.conf:
>>>>
>>>> ike esp from $local_net to $remote_net peer $remote_ip \
>>>> quick auth hmac-md5 enc aes
>>>>
>>>> The second point relates to pf rules in pf.conf:
>>>>
>>>> match in scrub (max-mss 1440)
>>>>
>>>> You can experiment with the values in the 1400-1480 range.
>>>>
>>>> Also, please make sure that you don't run tcpbench or any
>>>> other benchmarking on the vpn gates themselves as it offsets
>>>> the measurements.

Re: Slow VPN Performance

[Mike Belopuhov]

On Mon, Oct 22, 2012 at 4:10 PM, Michael Sideris  wrote:
> It seems that changing to hmac-md5 boosted network throughput from
> ~50Mbit/s to ~100Mbit/s which is decent and reasonable. I am going to
> experiment a bit further with `scrub` options in pf.conf to see if I
> can squeeze more performance out of the link. The question now
> ishow much is security affected by using hmac-md5 vs the default
> hmac-sha2-256?

It's more a question of how often do you rekey? You also should not
disable Perfect Forward Secrecy that recomputes DH values every
time you renew your phase 2 key. And while there are no known
serious attacks on HMAC-MD5 it all depends on how important the
data that you're protecting is and if you have to be compliant with
any regulations that might mandate use of SHA2.

>  Should I consider using better CPUs on the servers in
> order to gain better performance through a stronger algorithm?
>

You can get 600-750Mbps (depending on the CPU speed) in the
AES-NI enabled setup (using AES-GCM that is).

> On Mon, Oct 22, 2012 at 2:58 PM, Mike Belopuhov  wrote:
>> On Tue, Oct 16, 2012 at 11:43 AM, Michael Sideris  wrote:
>>> Both endpoints run stock OpenBSD 5.1 (amd64). We use the VPN link to
>>> manage our platform remotely and perform daily backups. G-VPN runs on
>>> a 150Mbit/s link while L-VPN on a 1Gbit/s link. On one hand, our VPN
>>> setup runs really nicely. The connections are routed properly, pf is
>>> godsent and authpf works wonders. On the other hand, network
>>> throughput over the VPN tunnel never exceeds 3.4MB/s (ftp, scp, rsync,
>>> etc...)
>>>
>>> I welcome any suggestions. Keep in mind that this is our production
>>> VPN tunnel, so I cannot shut it down at will. Thanks in advance.
>>>
>>> ---
>>> Mike
>>>
>>
>> Hi,
>>
>> I suggest a couple of changes:
>>
>>  1) use cheaper hash function (md5 or at least sha1)
>>  2) use mss fixup so that your packets don't get fragmented
>>
>> The first point relates to your "ike" rules in ipsec.conf:
>>
>> ike esp from $local_net to $remote_net peer $remote_ip \
>> quick auth hmac-md5 enc aes
>>
>> The second point relates to pf rules in pf.conf:
>>
>> match in scrub (max-mss 1440)
>>
>> You can experiment with the values in the 1400-1480 range.
>>
>> Also, please make sure that you don't run tcpbench or any
>> other benchmarking on the vpn gates themselves as it offsets
>> the measurements.

Re: ikev2 and (ta-da) OpenBSD road warrior host

[Mike Belopuhov]

On Thu, Oct 4, 2012 at 6:13 AM, Erling Westenvik
 wrote:
> Shamefully I must admit what many OpenBSD'ers consider a crime worse
> than intercourse with the devil, namely to follow a so-called "Howto"
> (http://www.mouedine.net/) and within minutes having my daughters
> Windows 7 road "warrior" up and running connected to my OpenBSD gateway
> using IKEv2.
>
> Luckily for those that think natively in man pages, I'm stuck with how
> to achieve the same thing with my OpenBSD real road warriors, so now is
> your chance to tell me off.
>
> On the gateway the following /etc/iked.conf works for the win7 client:
>
> # cat /etc/iked.conf
> ikev2 passive esp \
> from 192.168.3.0/24 to 10.10.10.0/24 local a.b.c.d peer any \
> srcid a.b.c.d \
> config address 10.10.10.7
>
> I've generated certificates for one of my OpenBSD clients:
>
> # ikectl ca vpn certificate t500 create (+ export)
>
> copied them to the client and extracted them according to ikectl(8):
>
> # tar -C /etc/iked laptop -xzpf t500.tgz
>
> which brought /etc/iked on the laptop to contain:
>
> ./ca/ca.crt
> ./certs/t500.crt
> ./crls/ca.crl
> ./export/ca.pfx
> ./export/t500.pfx
> ./private/t500.key
> ./private/local.key
> ./local.pub
>
> What is difficult to derive from the multitude of man pages from this
> point onward, is:
>
> 1) how to add the client to /etc/iked.conf on the gateway.

"config address" cannot be used by the ikev2 client implemented in
the iked.  also you might find out that certificates are not working in
the client setup.  i'm working on fixing the latter right now.

> 2) how to configure and start the client.
>

for now your only option is to use psk and a different server
rule.  please make sure to use different "local" ip addresses
on the server otherwise you won't be able to match multiple
policies.  that's something we need to address as well.

> I think I'll be able to figure out most of it if someone would just
> point me in the right direction.
>
> Cheers,
> Erling

Re: IPSEC VPN performance

[Mike Belopuhov]

On Fri, Sep 28, 2012 at 11:45 AM, Otto Moerbeek  wrote:
> On Thu, Sep 27, 2012 at 05:30:38PM -0400, Jim Miller wrote:
>
>> Hi,
>>
>> I'm trying to determine if the performance I'm seeing between two
>> OpenBSD 5.1 IPSEC VPN endpoints is typical (or expected).  I recognize
>> there are quite a few variables to consider and I'm sure I've not
>> toggled each one but I could use a sanity check regardless.
>>
>> Question:
>> With the configuration below when I disable ipsec I can route traffic
>> between the two hosts (hosts A and B) at about 900mbps.  When I add the
>> VPN I am getting speeds of approx. 40mbps.  The CPU load on the OpenBSD
>> boxes spikes to about 80% on one of the cores but the other 3 are
>> essentially unaffected.  Enabling/Disabling AES-NI in the bios doesn't
>> seem to actually do anything as the cpu message in dmesg still shows the
>> AES flag.
>>
>> The test I'm using is this
>> Host A:
>> # nc -v -l 12345 | /dev/null
>>
>> Host B:
>> # dd if=/dev/zero bs=1000 count=1 | nc -v  12345
>>
>> The reason these performance numbers are concerning to me is that I
>> wanted a solution that would allow me to get decent (a.k.a. 100mbps +/-
>> 10%) without having to buy expensive cisco/juniper devices.
>
> I would start playing with different modes, to see if that makes a
> difference. It could very well be that AES-NI is only used in certain
> modes. Start with the iked defaults for a start.
>

aes-ni is used for all aes-related modes (aes-cbc, aes-ctr, aes-gcm
and aes-gmac)... on amd64.

>>
>> Am I dreaming or have others had better performance?  Also, any recent
>> data on AES-NI optimizations would be helpful.
>>
>> Thanks
>> Jim
>>
>> Hardware Configuration:
>> - (2) identical SuperMicro systems with quad core E31220 w/ AES-NI enabled
>
> amd64 or i386? Why strip info from dmesg? It *might* mkae a difference.
>

wow. it definitely makes a difference: aes-ni is not supported on i386.

> -Otto

Re: IPSEC VPN performance

[Mike Belopuhov]

On Thu, Sep 27, 2012 at 11:30 PM, Jim Miller  wrote:
> Hi,
>
> I'm trying to determine if the performance I'm seeing between two
> OpenBSD 5.1 IPSEC VPN endpoints is typical (or expected).  I recognize
> there are quite a few variables to consider and I'm sure I've not
> toggled each one but I could use a sanity check regardless.
>
> Question:
> With the configuration below when I disable ipsec I can route traffic
> between the two hosts (hosts A and B) at about 900mbps.  When I add the
> VPN I am getting speeds of approx. 40mbps.  The CPU load on the OpenBSD
> boxes spikes to about 80% on one of the cores but the other 3 are
> essentially unaffected.  Enabling/Disabling AES-NI in the bios doesn't
> seem to actually do anything as the cpu message in dmesg still shows the
> AES flag.
>
> The test I'm using is this
> Host A:
> # nc -v -l 12345 | /dev/null
>
> Host B:
> # dd if=/dev/zero bs=1000 count=1 | nc -v  12345
>
> The reason these performance numbers are concerning to me is that I
> wanted a solution that would allow me to get decent (a.k.a. 100mbps +/-
> 10%) without having to buy expensive cisco/juniper devices.
>
> Am I dreaming or have others had better performance?  Also, any recent
> data on AES-NI optimizations would be helpful.
>
> Thanks
> Jim
>
> Hardware Configuration:
> - (2) identical SuperMicro systems with quad core E31220 w/ AES-NI enabled
>
> cpu0: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz ("GenuineIntel" 686-class)
> 3.10 GHz
> cpu0:
> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,LAHF
> cpu1: ..
> cpu2: ...
> cpu3: ...
> - 2GB ram
> - AES-NI enabled in bios
> - (4) Intel PRO/1000 MT (82574L)
>
> Software Configuration:
> VPN A
> /etc/iked.conf
> ikev2 active esp \
> from 172.16.1.0/24 to 172.16.2.0/24 \
> local 10.0.0.1 peer 10.0.0.2 \
> ikesa enc aes-256 auth hmac-sha2-512 group modp4096 \
> childsa enc aes-256-gmac \
> psk "helpmeplease"
>
> VPN B
> (reverse of A config)
>
> Host A -> 172.16.1.2  (behind VPN A)
> Host B- > 172.16.2.2  (behind VPN B)
> VPN A (10.0.0.1) talks to B (10.0.0.2) via a crossover cable.
> No switches/routers/hubs/etc in this test system.  All hosts running
> linux with 1000mb phys.
>

Hi,

I have two suggestions:

1) try -current as forwarding performance was improved;
2) try aes-128-gcm for child sa (traffic encryption). aes-256-gmac-gmac
means don't encrypt, just authenticate.

I must say I'm curious about Xeon E3 AES-NI performance myself as
we have tested only core i5, i7 and previous generation xeons, but
the cpu you've picked should be the right choice.

Cheers,
Mike

Re: 10Gbit Intel 10GbE SFP+ (82599) performance

[Mike Belopuhov]

On Sat, Aug 04, 2012 at 17:43 +0200, Claudio Jeker wrote:
> On Tue, Jul 24, 2012 at 02:22:57PM +0200, Henrik Lund Kramshøj wrote:
> > On 24/07/2012, at 14.16, Otto Moerbeek wrote:
> > 
> > > On Tue, Jul 24, 2012 at 01:52:03PM +0200, Peter Hessler wrote:
> > >
> > >> On 2012 Jul 24 (Tue) at 12:40:00 +0200 (+0200), Henrik Lund Kramsh?j
> > wrote:
> > >> :Current tuning, apart from kernel defaults:
> > >> :and also modified /etc/sysctl somewhat - most changes below are from
> > >> :https://calomel.org/network_performance.html:
> > >>
> > >> Revert all changes recommended by this page, they hurt performance and
> > >> do not help.
> > >>
> > >> calomel.org is only bad juju.
> > >>
> > >>
> > >> --
> > >> In case of atomic attack, the federal ruling against prayer in schools
> > >> will be temporarily canceled.
> > >
> > > Also, the OP didn't say if pf is active or not. By default pf is
> > > enabled these days, switch off by setting pf=NO in rc.conf.local
> > 
> > sorry, but tried both, some difference, some improvement, as shown below
> > 
> > hlk@xpc02:hlk$ sudo pfctl -d
> > pf disabled
> > hlk@xpc02:hlk$  tcpbench -u 10.0.10.30
> > Elapsed:1000 Mbps:1943.217 Peak Mbps:1943.217 Tx PPS:  
> > 165015
> > Elapsed:2000 Mbps:1944.761 Peak Mbps:1944.761 Tx PPS:  
> > 165146
> > Elapsed:3000 Mbps:1942.852 Peak Mbps:1944.761 Tx PPS:  
> > 164984
> > Elapsed:4000 Mbps:1942.710 Peak Mbps:1944.761 Tx PPS:  
> > 164972
> > 
> > hlk@xpc02:hlk$  tcpbench -n 4 10.0.10.30
> >   elapsed_ms  bytes mbps   bwidth
> > 1000  114326840  914.615   28.58%
> > 1000   86697552  693.580   21.67%
> > 1000  7  889.431   27.79%
> > 1000   87886360  703.091   21.97%
> > Conn:   4 Mbps: 3200.717 Peak Mbps: 3200.717 Avg Mbps:  800.179
> > 2001   79059352  631.843   19.21%
> > 2001  127067792 1016.542   30.87%
> > 2001  135531352 1084.251   32.93%
> > 2001   69944248  559.554   16.99%
> > Conn:   4 Mbps: 3292.190 Peak Mbps: 3292.190 Avg Mbps:  823.048
> > 3003  101616720  812.122   25.16%
> > 3003   98757024  789.267   24.45%
> > 3003  103894144  830.323   25.72%
> > 3003   99604280  796.038   24.66%
> > Conn:   4 Mbps: 3227.750 Peak Mbps: 3292.190 Avg Mbps:  806.937
> > 
> > But still way below expectations :-(
> > 
> 
> Then you should adjust your expectations. OpenBSD never said to do 10G
> wirespeed or anything in that regard. Btw. depending on how you plan to
> use the system it may not make sense to run tcpbench on the system under
> test since that will skew the test results.
> 

to elaborate on this point a bit: please make sure you understand what
you're testing!  tcpbench and iperf both test how fast your tcp or udp
server running in userland can receive and transmit information through
the socket interface.  this has nothing to do with router workloads for
example.

> Yes, we would like to be faster but to get more speed large changes are
> needed since currently only one CPU is doing all the work in the kernel.
> 
> -- 
> :wq Claudio

Re: Windows 7 and IkeV2 VPN Issue

[Mike Belopuhov]

On Thu, Jul 26, 2012 at 9:27 PM, Bentley, Dain  wrote:
> Hello fellow OpenBSD users,
>
> I've run into a of couple issues with setting up and IKE IPSEC VPN with a
> windows 7 native client.  Now I've ran through the lists and have found a
> solution to get it working somewhat how I'd like it working.
>
>
> And on my W7 client I have a static IP configured and using machine
> certificates.  I connect there with no issue and everything is kosher...kind
> of.
>
> I want to use a username and password so I have this in my iked.conf:
>
> user "my user ID" "Wouldn't_you_like_to_know?"
>
> When I do this I get an error:
> Error Code 13803 "IKE Negotiation in progress" and it just sits there.  Has
> anyone gotten this to work before?
>

Sure.

>
> Any help would be appreciated.  Is there any setting or something I should
> apply?  I'm running windows with 7 within NAT.  Like I said, certs work fine,
> password and usernames do not.
>

Are you running -current version of iked?  Because you have to.

Re: ipsec between 5.0 & 5.1

[Mike Belopuhov]

On Fri, Jul 27, 2012 at 10:33 AM, lilit-aibolit  wrote:
> Hi misc.
> is it possible?
>

why wouldn't it be?

Re: ipsec tunnel speeds

[Mike Belopuhov]

On Mon, Jun 25, 2012 at 2:53 PM, Mark Romer  wrote:
> Great question Ted
> Does anyone know the answer?

sure.

> Thanks Mark
> On Jun 22, 2012 12:58 PM, "Ted Unangst"  wrote:
>
>> On Fri, Jun 22, 2012 at 12:52, Ryan McBride wrote:
>>
>> > 550Mb/s with aes-128-gcm (requires AES-NI and amd64) on
>> > hw.model=Intel(R) Xeon(R) CPU E5649 @ 2.53GHz
>> > hw.vendor=HP
>> > hw.product=ProLiant DL360 G7
>>
>> what's the reason aes-128-gcm requires amd64?

because the assembly is written for amd64.

>>  we can't add that code to i386?

that specific one? of course not. but the aes-ni and clmul instructions
are part of sse and can be executed by both 32-bit and 64-bit programs.

apart from that, it might be possible that binutils have to be adjusted
(i don't remember if they share the same code) and i386 has to grow
fpu_kernel_{enter,exit}.

Re: ikev2 between openbsd and windows

[Mike Belopuhov]

On Thu, May 31, 2012 at 12:18 +0200, Peter J. Philipp wrote:
> My iked config looks like this:
> 
> ikev2 "win7" passive esp \
> from 172.16.20.0/24 to 0.0.0.0/0 local any peer any \
> srcid 10.0.0.1 \
> eap "mschap-v2" \
> config address 172.16.20.1 \
> config name-server 212.18.3.5 \
> tag "$name-$id"
> 

i've just realised you made a mistake by exchanging "from" and
"to" specifications.  the correct way is:

from 0.0.0.0/0 to 172.16.20.0/24 local any peer any \

it should always read "i provide access from a network behind
MYSELF to a network behind my PEER" regardless of whether you
initiate or a respond.  in other words just like in ipsec.conf.

Re: ikev2 between openbsd and windows

[Mike Belopuhov]

On Thu, May 31, 2012 at 12:47 +0200, Peter J. Philipp wrote:
> On Thu, May 31, 2012 at 12:28:47PM +0200, Mike Belopuhov wrote:
> > > My iked config looks like this:
> > > 
> > 
> > do you have a "user" specification in your iked.conf?
> > which user are you trying to authenticate as?
> > "user" specification occupies a separate line and looks
> > like that:
> > 
> > user "username" "password"
> > 
> > iked can't consult the local password database or radius
> > or any other authentication service at the moment except
> > this internal "database".
> 
> Yes I do have a user entry, right at the top.  I didn't think posting
> it was a good idea.
> 
> > also, have you tried w/o mschap? you need to select the
> > "Computerzertifikate verwenden" radio button to turn eap off.
> 
> I tried that but it had an error, which made me want to try EAP again.
> 
> > > ikev2 "win7" passive esp \
> > > from 172.16.20.0/24 to 0.0.0.0/0 local any peer any \
> > > srcid 10.0.0.1 \
> > > eap "mschap-v2" \
> > > config address 172.16.20.1 \
> > > config name-server 212.18.3.5 \
> > > tag "$name-$id"
> > > 
> > 
> > looks fine except of absent of the "user" specification.
> > i'd ditch the "tag" though as i didn't test it but it shouldn't
> > affect anything.
> 
> Hmm.  What to do... Any hint on how to debug this best?
> 

try to verify that certificates are installed correctly on windows
and are valid.  make sure you didn't install them by doubleclicking
(as i initially wrote) but imported them via mmc into the right
section (under Komputerkonto).

try to change "to 0.0.0.0/0" to something like "to 10.50.0.1",
where you can assign 10.50.0.1 to lo1.

make sure that certificates were created by commands like:

 ikectl ca  certificate 10.0.0.1 create

and host using FQDN.  srcid must match that otherwise windows
refuses to connect.

and at last, please provice output from the iked -dvv.

cheers,
mike

Re: ikev2 between openbsd and windows

[Mike Belopuhov]

On Thu, May 31, 2012 at 12:18 +0200, Peter J. Philipp wrote:
> On Tue, May 29, 2012 at 01:55:45PM +0200, Mike Belopuhov wrote:
> > On Wed, May 16, 2012 at 17:30 +0400, Pavel Shvagirev wrote:
> > > 2. Doesn't work EAP mode - Windows stops on "Checking username and
> > > password" error. Then #13803, 1931...
> > 
> > Hi,
> > 
> > Just to mention it for those not following source-changes@
> > that there was a bug in the message ID handling that prevented
> > EAP from working correctly.  The fix was committed on Friday.
> > 
> > Cheers,
> > Mike
> 
> Hi,
> 
> I still can't get it to work.  I made two screenshots they are here:
> 
> http://ipv4.goldflipper.net/private/iked-eap1.jpg
> 
> and
> 
> http://ipv4.goldflipper.net/private/iked-eap2.jpg
> 
> My iked config looks like this:
> 

do you have a "user" specification in your iked.conf?
which user are you trying to authenticate as?
"user" specification occupies a separate line and looks
like that:

user "username" "password"

iked can't consult the local password database or radius
or any other authentication service at the moment except
this internal "database".

also, have you tried w/o mschap? you need to select the
"Computerzertifikate verwenden" radio button to turn eap off.

> ikev2 "win7" passive esp \
> from 172.16.20.0/24 to 0.0.0.0/0 local any peer any \
> srcid 10.0.0.1 \
> eap "mschap-v2" \
> config address 172.16.20.1 \
> config name-server 212.18.3.5 \
> tag "$name-$id"
> 

looks fine except of absent of the "user" specification.
i'd ditch the "tag" though as i didn't test it but it shouldn't
affect anything.

> I installed the iked from the -current source on top of the 5.0 binary 
> I believe these are the right ones because I see your recent timestamp
> in them:
> 
> ikev2_msg.c:/*  $OpenBSD: ikev2_msg.c,v 1.15 2012/05/30 09:18:14 mikeb Exp $
> 
> Any hint on what I'm doing wrong?  Sorry the screenshots are in german,
> Fehler 13843 is Error 13843.  I googled for that but wasn't any wiser after.
> 
> Regards,
> -peter

Re: ikev2 between openbsd and windows

[Mike Belopuhov]

On Wed, May 16, 2012 at 17:30 +0400, Pavel Shvagirev wrote:
> 2. Doesn't work EAP mode - Windows stops on "Checking username and
> password" error. Then #13803, 1931...

Hi,

Just to mention it for those not following source-changes@
that there was a bug in the message ID handling that prevented
EAP from working correctly.  The fix was committed on Friday.

Cheers,
Mike

Re: ikev2 between openbsd and windows

[Mike Belopuhov]

On Wed, May 16, 2012 at 10:00 PM, Peter J. Philipp  wrote:
> On Mon, May 14, 2012 at 12:53:34PM +0200, Mike Belopuhov wrote:
>> 4) Install the server certificate on the server:
>>
>>ikectl ca vpn certificate 10.1.0.1 install
>>
>> 5) To export the client certificate in a ZIP'ed PFX format, you need
>>to install zip utility (pkg_add -i zip).
>>
>>ikectl ca vpn certificate 10.5.0.1 export
>>
>
> Does the .tgz file need to be extracted at all on the server?

On the server? No. For the server certificate you just do the "install".

> I've tried
> and tried for too long and my certificates are out of sync I think, is
there
> a command to delete everything and just keep the original blank iked
structure
> so that one can start over without old certificates in the way?
>

I guess you can do "ikectl ca vpn delete" and that should remove most
of the stuff that gets in the way.

>> 6) Transfer 10.5.0.1.zip to the Windows host and load the certificates
>>by doubleclicking on them.  Make sure that certificates are valid
>>in the MMC Certificates Snap-In.
>
> This gave me a huge headache.  I tried using MMC (as administrator and
other
> user) but my vpn client stayed at 13806 error.  Perhaps VPN wasn't meant
for
> people like me.
>

As Pavel described, you shouldn't doubleclick as I said because
then windows will install it to the user certificates. Quoting Pavel:
"MMC and the local computer account switch should be used
instead."   I believe he refers to the Certificates snap-in.  It asks
you this question when you add it to the MMC.

Re: ikev2 between openbsd and windows

[Mike Belopuhov]

On Wed, May 16, 2012 at 17:30 +0400, Pavel Shvagirev wrote:
> 
> Thank you very much for the detailed reply. It helped a lot, though I
> have something to add.
> 
> > 6) Transfer 10.5.0.1.zip to the Windows host and load the certificates
> >by doubleclicking on them.
> You should not import the cert by doubleclicking on it - it will import
> to the current user's facility instead of a local computer. That will
> cause 13806 errormessage telling that there is no appropriate computer
> certificate etc. MMC and the local computer account switch should be
> used instead.
> 

Yes, I admit I have just tested the possibility of installing
certificates and hoped that user certs will work just fine.
We have a tool to do the right thing automatically, so I didn't
bother to test this part.  Shame on me (:

> > 7) Configure iked to do RSA auth w/o EAP (for the start):
> >
> > ikev2 "win7" passive esp \
> > from 192.168.0.0/24 to 192.168.1.0/24 local any peer any \
> > srcid 10.1.0.1 \
> > config address 192.168.1.100 \
> > config name-server 192.168.0.1
> >
> >Here, 192.168.0.0/24 is a network client is getting access to,
> >192.168.1.0/24 is a "DHCP"-like network from which client is
> >getting an ip address (192.168.1.100 specifically).  Please
> >note, that the code to turn this awkwardness into real (DHCP-like)
> >address pool specification is not written yet.  Note that srcid
> >has to match the host that the certificate is issued to, otherwise
> >windows will refuse to connect. 
> >
> >Once you do that you can load iked and see that it hooks up the
> >server certificate (in the iked -dvv output that is).
> This is the most intriguing part :)
> 
>ikev2 "win7" esp \
>from 172.16.2.0/24 to 0.0.0.0/0 \
>peer 10.0.0.0/8 local 192.168.56.0/24 \
>eap "mschap-v2" \
>config address 172.16.2.1 \
>tag "$name-$id"
> 
> This example is from the man page. `config address' is in the range of
> `from source', not from the destination subnet. Are you sure it sould be
> like you said?
> 

Yes, I'm sure.  The syntax was changing over time but in the end the
"from" network is always the network behind the host running iked
regardless of whether it's initiator or responder.  Here's a config
I use at the moment for my testing:

ikev2 "win7" passive esp \
   from 10.1.0.2 to 10.1.0.5 local any peer any \
   srcid 172.23.55.126 \
   config address 10.1.0.5 \
   config name-server 10.1.0.2

10.1.0.2 is configured as "ifconfig lo1 10.1.0.2/24" on openbsd.

Man page should be updated.

> How do I manage the `DHCP-like' addresses? Is this address range where
> the client should be granted an IP from OR is that a client's local
> private network? I found that dhcpd cannot run on enc0 interface. How do
> you manage that?
> 

No, DHCP is NOT involved at all. IKEv2 does it itself. I said that you
can only configure one IP address per "ikev2" rule atm and the address
pool code is not written (but it should/will be).

> Now the negotiation seems to be complete but still the connection can
> not be established due to various reasons:
> 
> 1. Windows side stops on error #31 "Attached device is not working
> properly" (looks like a Windows problem though). Have you seen that?
> 

Nope.

> 2. Doesn't work EAP mode - Windows stops on "Checking username and
> password" error. Then #13803, 1931...
> 

Well, as I said, try certificates first.  Disable EAP.

> > If someone thinks that this might be turned into some sort of a
> > howto or FAQ entry or whatever, please feel free to reuse any
> > piece of text.  Attribution is welcomed but not required.
> Your instructoins really did the trick - I got rid of those anoying
> troubles that were caused by strictly following the manuals... I think
> it should have been written in more detail, covering in detail _every_
> network part (with its role) that participate in the negotiation. 'cause
> sometimes it has contradicting points. Probably it is a matter of
> individual perception, nevertheless I had what I had as well as tons of
> others struggling with that in mail lists across the web =)
> 

Everyone is encouraged to contribute to this thread so that we can
work out an unambiguous instruction.

> Thanks.
> 
> -- 
> Best regards,
> Pavel Shvagirev
> skype: pavel.shvagirev

Re: ikev2 between openbsd and windows

[Mike Belopuhov]

On Fri, May 11, 2012 at 20:39 +0400, Pavel Shvagirev wrote:
> Hi everyone.
> 
> Trying to build ikev2 vpn between openbsd 5.1 and windows 7 via
> certificates. Windows stops at #13843 error message - "Invalid payload
> received". Iked -vd output has a 'sa_state: VALID -> ESTABLISHED'
> meaning that 2nd phase is ok but just before that line I have:
> 
> ca_getreq: no valid local certificate found
> 
> What local cert does it mean? 'ikectl show ca certificates' output is ok
> - it returnes all the certs that I have installed/exported/imported on
> windows side
> 
> Did install both obsd's and win7's certificates like
> 
> ikectl ca caname certificate openbsdmachine create | install | export
> ikectl ca caname certificate win7machine create | install | export
> 
> - just like it's said in the man page.
> 
> Google has just one link for that queue - openbsd sources =)
> 
> -- 
> Best regards,
> Pavel Shvagirev
> skype: pavel.shvagirev
> 

OK, this is how you do it.  First of all, fun facts about Windows 7
[simple VPN client]:

0) It CANNOT do PSK;
1) It REQUIRES certificates regardless of whether you want to do EAP
   or not;
2) It has TWO ipsec clients and both(?) can do IKEv2, but these
   instructions are for the simple one found in the Network
   Connnections Center (or whatever it's called).
3) You need to supply windows with two certificates: one for CA,
   one for the server.

Also please make sure that the time on the CA is somewhat sensible
from the clients point of view, so that certificates won't be invalid
because your time is not synchronized.

To setup a CA on the same host as iked do the following:

1) Copy the template /usr/src/usr.sbin/ikectl/ikeca.cnf to /etc/ssl
   and edit as appropriate;

2) Create and install CA certificate:

   ikectl ca vpn create
   ikectl ca vpn install

3) Create certificates for the server and clients.  Make sure that
   host specification matches the SourceID specification in iked.conf.
   You can use IP addressess, but iked defaults to FQDN/hostname.

   ikectl ca vpn certificate 10.1.0.1 create  # server
   ikectl ca vpn certificate 10.5.0.1 create  # host

4) Install the server certificate on the server:

   ikectl ca vpn certificate 10.1.0.1 install

5) To export the client certificate in a ZIP'ed PFX format, you need
   to install zip utility (pkg_add -i zip).

   ikectl ca vpn certificate 10.5.0.1 export

6) Transfer 10.5.0.1.zip to the Windows host and load the certificates
   by doubleclicking on them.  Make sure that certificates are valid
   in the MMC Certificates Snap-In.

7) Configure iked to do RSA auth w/o EAP (for the start):

ikev2 "win7" passive esp \
from 192.168.0.0/24 to 192.168.1.0/24 local any peer any \
srcid 10.1.0.1 \
config address 192.168.1.100 \
config name-server 192.168.0.1

   Here, 192.168.0.0/24 is a network client is getting access to,
   192.168.1.0/24 is a "DHCP"-like network from which client is
   getting an ip address (192.168.1.100 specifically).  Please
   note, that the code to turn this awkwardness into real (DHCP-like)
   address pool specification is not written yet.  Note that srcid
   has to match the host that the certificate is issued to, otherwise
   windows will refuse to connect. 

   Once you do that you can load iked and see that it hooks up the
   server certificate (in the iked -dvv output that is).

7) Now on the windows box, go to the Network Connections Center
   and create an IKEv2 VPN connection with the client.  Make sure
   to check the Certificate radio button on the Security tab in
   the connection properties, so that you won't do EAP.

8) Start the connection.

9) Profit!!!

PS.

If someone thinks that this might be turned into some sort of a
howto or FAQ entry or whatever, please feel free to reuse any
piece of text.  Attribution is welcomed but not required.

Re: Problems reading from (but not writing to?) LTO 5 Ultrium tape from i386 snapshot(s)

[Mike Belopuhov]

On Wed, May 02, 2012 at 11:29 +, Fergus Wilde wrote:
> Hello,
> 
> I wonder if anyone can help me out with the following. Any or all
> help, including 'this has nothing
> to do with OBSD', or options to try passing
> to the dump, happily received. In particular, I wonder
>  if my situation is
> analogous with the facts of the case here:
> http://old.nabble.com/Restore-from-SCSI-tape-on-recent-i386-snapshot-td288246
> 13.html
> 
> although unlike the OP in this case I have *not* had it working right
> beforehand
> as this is a new machine.
> 
> This is with snapshot 5.1 GENERIC.MP#253
> i386 and the two previous snapshots on my 
> local mirror. Dmesg and
> /var/run/dmesg.boot below.
> 
> Machine is an HP 350ML G6 using 3-disk hardware
> RAID 5 with a Quantum Ultrium LTO 5 external drive.
> 
> 
> And so on for the other mounts on the array. The amount of
> time this takes appears realistic, but since
> I can't successfully test the
> dump, I don't know if it is 'really' doing  a backup.
> 
> Attempting to test the
> dump:
> 
> # /sbin/restore -tvs 1 -f /dev/rst0  
> Verify tape and initialize maps
> restore: tape read error: Input/output error
> 
> 
> For interest, I tried using tar
> to backup the root partition. Again, this seemed to work fine from the write
> side, but on attempting to read I again got an Input/output error.
> 
> Thanks for
> any thoughts!
> 
> mpii0 at pci3 dev 0 function 0 "Symbios Logic SAS2008" rev 0x03: msi
> scsibus1 at mpii0: 258 targets
> st0 at scsibus1 targ 5 lun 0:  5, 3180> SCSI4 1/sequential removable naa.00e09e60001d700e

hi,

can you attach a hard drive to the controller (SATA will do)
and run some simple tests with it: partition, newfs, do some
file copying.  see if it works for you.  newer lsi firmwares
are known to be a bit different and don't work very well with
the current driver.

Re: pfsync changes in current?

[Mike Belopuhov]

On Fri, May 4, 2012 at 12:27 PM, Kapetanakis Giannis
 wrote:
> On 03/05/12 22:56, mxb wrote:
>>
>> I'd suggest you to experiment with the src and try to rollback if_pfsync.c
>> to,
>> say rev. 1.179,
>> then roll forward with revisions until you can pinpoint one which breaks
>> it.
>>
>> //maxim
>
>
> Ok, I've traced the problem back to r1.180 of if_pfsync.c
>
> If I remove 1.180 and apply 1.181, 1.182, 1.183, 1.184
> then everything works fine.
>
> According to my understanding 1.180 introduces the following two problems:
>
> 1) When one firewall reboots, the other one is initiating a bulk transfer
> thus demoting his carp/pfsync groups.
> I don't understand why this is needed.
>
> 2) When the firewall boots it sets his demotion counter on groups
> carp/pfsync to 0 (instead of 1) before his bulk transfer
> is finished.
>
> The diff bellow removes 1.180
> Feel free to send me diffs to try.
>
> regards,
>
> Giannis
>

hi,

yes, there's no doubt that this is a problem.

btw, can you try the unpatched kernel and put a switch between
the firewalls, i.e. connect pfsync interfaces through the switch.
will that work for you?

Re: alix2d2 LM86, no hw.sensors

[Mike Belopuhov]

On Sun, Feb 19, 2012 at 00:06 +0100, Mike Belopuhov wrote:
> On Sat, Feb 18, 2012 at 03:09 +1100, Jonathan Gray wrote:
> > On Fri, Feb 17, 2012 at 04:20:25PM +0100, Michal Mazurek wrote:
> > > I have an alix2d2 running OpenBSD 5.0. There are no hw.sensors.
> > > The producer says there is an LM86 on board, which is supported by the
> > > maxtmp driver. It appears the driver is present in generic. I tried 
> > > starting
> > > sensorsd but got:
> > > daemon:Feb 17 13:12:04 T1 sensorsd[10445]: startup, system has 0 sensors
> > > 
> > > How can I read the temperature of my alix2d2 running OpenBSD 5.0?
> > 
> > There is no driver for the CS5535/CS5536 I2C controller the chip is 
> > connected to,
> > it won't work till that is written.
> > 
> 
> [un]surprisingly it's actually the same as gscsio(4).  here's a port
> of that code to glxpcib(4).  i don't have the hardware at hand, but
> i encourage you to test (:
> 
> cheers
> 
> Index: dev/pci/files.pci
> ===
> RCS file: /cvs/src/sys/dev/pci/files.pci,v
> retrieving revision 1.281
> diff -u -p -r1.281 files.pci
> --- dev/pci/files.pci 15 Nov 2011 22:27:53 -  1.281
> +++ dev/pci/files.pci 18 Feb 2012 23:03:52 -
> @@ -807,6 +807,6 @@ attachitherm at pci
>  file dev/pci/itherm.citherm
>  
>  # AMD Geode CS5536 PCI-ISA bridge
> -device   glxpcib: isabus, gpiobus
> +device   glxpcib: isabus, gpiobus, i2cbus
>  attach   glxpcib at pci
>  file dev/pci/glxpcib.c   glxpcib


and i forgot to mention that kernel config has to be patched too.
thanks to shadchin@ for reminding.

Index: arch/i386/conf/GENERIC
===
RCS file: /cvs/src/sys/arch/i386/conf/GENERIC,v
retrieving revision 1.730
diff -u -p -r1.730 GENERIC
--- arch/i386/conf/GENERIC  28 Jan 2012 00:39:15 -  1.730
+++ arch/i386/conf/GENERIC  19 Feb 2012 12:10:56 -
@@ -91,6 +91,7 @@ gscpcib* at pci?  # NS Geode SC1100 PCI-
 gpio*  at gscpcib?
 glxpcib* at pci?   # AMD CS5536 PCI-ISA bridge
 gpio*  at glxpcib?
+iic*   at glxpcib?
 kate*  at pci? # AMD K8 temperature sensor
 km*at pci? # AMD K10 temperature sensor
 amas*  at pci? disable # AMD memory configuration
Index: arch/loongson/conf/GENERIC
===
RCS file: /cvs/src/sys/arch/loongson/conf/GENERIC,v
retrieving revision 1.36
diff -u -p -r1.36 GENERIC
--- arch/loongson/conf/GENERIC  7 Jul 2011 23:41:09 -   1.36
+++ arch/loongson/conf/GENERIC  19 Feb 2012 12:10:56 -
@@ -39,6 +39,7 @@ pci*  at bonito?
 # Lemote Lynloong, Lemote Fuloong 2F and Lemote Yeeloong devices
 glxpcib*   at pci?
 gpio*  at glxpcib?
+iic*   at glxpcib?
 isa0   at glxpcib?
 mcclock0   at isa? port 0x70
 pckbc0 at isa? # Yeeloong only

Re: alix2d2 LM86, no hw.sensors

[Mike Belopuhov]

On Sat, Feb 18, 2012 at 03:09 +1100, Jonathan Gray wrote:
> On Fri, Feb 17, 2012 at 04:20:25PM +0100, Michal Mazurek wrote:
> > I have an alix2d2 running OpenBSD 5.0. There are no hw.sensors.
> > The producer says there is an LM86 on board, which is supported by the
> > maxtmp driver. It appears the driver is present in generic. I tried starting
> > sensorsd but got:
> > daemon:Feb 17 13:12:04 T1 sensorsd[10445]: startup, system has 0 sensors
> > 
> > How can I read the temperature of my alix2d2 running OpenBSD 5.0?
> 
> There is no driver for the CS5535/CS5536 I2C controller the chip is connected 
> to,
> it won't work till that is written.
> 

[un]surprisingly it's actually the same as gscsio(4).  here's a port
of that code to glxpcib(4).  i don't have the hardware at hand, but
i encourage you to test (:

cheers

Index: dev/pci/files.pci
===
RCS file: /cvs/src/sys/dev/pci/files.pci,v
retrieving revision 1.281
diff -u -p -r1.281 files.pci
--- dev/pci/files.pci   15 Nov 2011 22:27:53 -  1.281
+++ dev/pci/files.pci   18 Feb 2012 23:03:52 -
@@ -807,6 +807,6 @@ attach  itherm at pci
 file   dev/pci/itherm.citherm
 
 # AMD Geode CS5536 PCI-ISA bridge
-device glxpcib: isabus, gpiobus
+device glxpcib: isabus, gpiobus, i2cbus
 attach glxpcib at pci
 file   dev/pci/glxpcib.c   glxpcib
Index: dev/pci/glxpcib.c
===
RCS file: /cvs/src/sys/dev/pci/glxpcib.c,v
retrieving revision 1.3
diff -u -p -r1.3 glxpcib.c
--- dev/pci/glxpcib.c   23 Oct 2010 17:42:57 -  1.3
+++ dev/pci/glxpcib.c   18 Feb 2012 23:03:52 -
@@ -28,6 +28,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 #ifdef __i386__
@@ -35,6 +36,8 @@
 #endif
 
 #include 
+#include 
+
 #include 
 #include 
 #include 
@@ -133,6 +136,34 @@
 #define AMD5536_GPIO_IN_INVRT_EN 0x24  /* invert input */
 #defineAMD5536_GPIO_READ_BACK  0x30/* read back value */
 
+/* SMB */
+#define MSR_LBAR_SMB   DIVIL_LBAR_SMB
+#define MSR_SMB_SIZE   0x07
+#define MSR_SMB_ADDR_MASK  0xfff8
+#define AMD5536_SMB_SDA0x00 /* serial data */
+#define AMD5536_SMB_STS0x01 /* status */
+#define AMD5536_SMB_STS_SLVSTOP0x80 /* slave stop */
+#define AMD5536_SMB_STS_SDAST  0x40 /* smb data status */
+#define AMD5536_SMB_STS_BER0x20 /* bus error */
+#define AMD5536_SMB_STS_NEGACK 0x10 /* negative acknowledge */
+#define AMD5536_SMB_STS_MASTER 0x02 /* master */
+#define AMD5536_SMB_STS_XMIT   0x01 /* transmit or receive */
+#define AMD5536_SMB_CST0x02 /* control status */
+#define AMD5536_SMB_CST_MATCH  0x04 /* address match */
+#define AMD5536_SMB_CST_BB 0x02 /* bus busy */
+#define AMD5536_SMB_CST_BUSY   0x01 /* busy */
+#define AMD5536_SMB_CTL1   0x03 /* control 1 */
+#define AMD5536_SMB_CTL1_ACK   0x10 /* receive acknowledge */
+#define AMD5536_SMB_CTL1_INTEN 0x04 /* interrupt enable  */
+#define AMD5536_SMB_CTL1_STOP  0x02 /* stop */
+#define AMD5536_SMB_CTL1_START 0x01 /* start */
+#define AMD5536_SMB_ADDR   0x04 /* serial address */
+#define AMD5536_SMB_ADDR_SAEN  0x80 /* slave enable */
+#define AMD5536_SMB_CTL2   0x05 /* control 2 */
+#define AMD5536_SMB_CTL2_EN0x01 /* enable clock */
+#define AMD5536_SMB_CTL2_FREQ  0x78 /* 100 kHz */
+#define AMD5536_SMB_CTL3   0x06 /* control 3 */
+
 /*
  * MSR registers we want to preserve accross suspend/resume
  */
@@ -151,12 +182,21 @@ struct glxpcib_softc {
 
uint64_tsc_msrsave[nitems(glxpcib_msrlist)];
 
-#if !defined(SMALL_KERNEL) && NGPIO > 0
+#ifndef SMALL_KERNEL
+#if NGPIO > 0
/* GPIO interface */
bus_space_tag_t sc_gpio_iot;
bus_space_handle_t  sc_gpio_ioh;
struct gpio_chipset_tag sc_gpio_gc;
gpio_pin_t  sc_gpio_pins[AMD5536_GPIO_NPINS];
+#endif
+   /* I2C interface */
+   bus_space_tag_t sc_smb_iot;
+   bus_space_handle_t  sc_smb_ioh;
+   struct i2c_controller   sc_smb_ic;
+   struct rwlock   sc_smb_lck;
+
+   /* Watchdog */
int sc_wdog;
int sc_wdog_period;
 #endif
@@ -180,11 +220,23 @@ void  pcibattach(struct device *parent, s
 
 u_int  glxpcib_get_timecount(struct timecounter *tc);
 
-#if !defined(SMALL_KERNEL) && NGPIO > 0
+#ifndef SMALL_KERNEL
+int glxpcib_wdogctl_cb(void *, int);
+#if NGPIO > 0
 void   glxpcib_gpio_pin_ctl(void *, int, int);
 intglxpcib_gpio_pin_read(void *, int);
 void   glxpcib_gpio_pin_write(void *, int, int);
-int glxpcib_wdogctl_cb(void *, int);
+#endif
+intglxpcib_smb_acquire_bus(void *, int);
+void   glxpcib_smb_release_bus(void *, int);
+intglxpcib_smb_send_start(void *, int);
+intglxpcib_smb_send_stop(void *, int);
+void   glxpcib_smb_send_ack(void *, int);
+intglxpcib_smb_initiate_xfer(void *, i

Re: uvm_fault in Dec. 15 amd64 snapshot

[Mike Belopuhov]

On Mon, Dec 19, 2011 at 21:46 +0100, Mike Belopuhov wrote:
> On Sun, Dec 18, 2011 at 18:50 -0800, James A. Peltier wrote:
> > - Original Message -
> > | Hi All,
> > | 
> > | Today is our semester maintenance day and we've upgraded our backup
> > | bridge firewall to the Dec. 15, 2011 snapshot available from
> > | ftp.openbsd.org and I'm getting this odd error when I boot it up.
> > | Oddly enough, this only happens when connected to the switch that
> > | original one is connected to (we swap them out each semester).
> > | 
> > | First, I use the upgrade method to go from snapshot to snapshot and
> > | reboot
> > | I run sysmerge to bring in the new configuration files from etc50.tgz
> > | and xetc50.tgz ( I only have bsd* man* base* xbase* installed) and
> > | reboot.
> > | 
> > | So as you can see the standard running -current and I've done several
> > | upgrades now.
> > | 
> > | On my test switch (HP5304XL) it boots okay and I can reload the
> > | firewall rules with no problem. When I connect it to my HP2910 where
> > | the current firewall is running I cannot fully boot. If I press CTRL+C
> > | during the starting network section it will continue to boot. If I
> > | then run pfctl -e it states that PF is already enabled enabled but if
> > | I run pfctl -Fr -f /etc/pf.conf I get the following.
> > | 
> > | # uvm_fault(0x80d2ff40, 0x0, 0, 1) -> e
> > | kernel: page fault trap, code=0
> > | Stopped at pf_translate+0x154: cmpw %r13w,0(%rsi)
> > | ddb{0}>
> > | 
> > | keyboard is dead, no response at all from console. Any ideas?
> > 
> > Okay, I've gotten some off list requests for more information, which
> > I'm hoping I'll be able to get for those people, but I'm now outside
> > of my maintenance window and will likely need to schedule another
> > outage or figure out how to reproduce it again.  The current bridge
> > firewall running the following version does not exhibit the problem,
> > but I'm not able to get a trace output at this time.  Maybe it's
> > still at least somewhat useful reference for updates that may have
> > happened. ( Yeah right, from Aug 8th until now.  Thousands of
> > commits. ;) )
> > 
> > OpenBSD 5.0 (GENERIC.MP) #57: Mon Aug  8 14:58:00 MDT 2011
> > dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > 
> >
> 
> pf_translate+0x154 corresponds to the condition at pf.c:3765.
> *pd->sport doesn't point to anything.  pd->sport is set to point
> to the extracted header in the pf_setup_pdesc.  the problem is
> that the header extraction happens based on the virtual_proto,
> not proto, which can be different in the case of a fragment.
> 
> now, pf.c got it covered by the condition on line 3476 which
> prevents pf_translate from running on fragments.  the only
> other invocation of pf_translate is in the if_pflog.c:407
> where we don't check for fragments.  therefore i think that
> this is the problem.
> 
> the diff below should fix the problem.  it also doesn't make
> sense to do af translation if we didn't manage to get our
> shit done in the pf_translate and in the subsequent block.
> 
> ok?
> 

ugh, typo has crawled into the diff...
in the meantime, i've confirmed that james is using logging facility.

Index: net/if_pflog.c
===
RCS file: /cvs/src/sys/net/if_pflog.c,v
retrieving revision 1.45
diff -u -p -r1.45 if_pflog.c
--- net/if_pflog.c  21 Oct 2011 15:45:55 -  1.45
+++ net/if_pflog.c  19 Dec 2011 23:13:55 -
@@ -404,7 +404,8 @@ pflog_bpfcopy(const void *src_arg, void 
if (pd.dport)
odport = *pd.dport;
 
-   if ((pfloghdr->rewritten = pf_translate(&pd, &pfloghdr->saddr,
+   if (pd.virtual_proto != PF_VPROTO_FRAGMENT &&
+   (pfloghdr->rewritten = pf_translate(&pd, &pfloghdr->saddr,
pfloghdr->sport, &pfloghdr->daddr, pfloghdr->dport, 0,
pfloghdr->dir))) {
m_copyback(pd.m, pd.off, min(pd.m->m_len - pd.off, pd.hdrlen),
@@ -422,7 +423,7 @@ pflog_bpfcopy(const void *src_arg, void 
pd.tot_len = min(pd.tot_len, len);
pd.tot_len -= pd.m->m_data - pd.m->m_pktdat;
 
-   if (afto)
+   if (pfloghdr->rewritten && afto)
pf_translate_af(&pd);
 
mlen = min(pd.m->m_pkthdr.len, len);

Re: uvm_fault in Dec. 15 amd64 snapshot

[Mike Belopuhov]

On Sun, Dec 18, 2011 at 18:50 -0800, James A. Peltier wrote:
> - Original Message -
> | Hi All,
> | 
> | Today is our semester maintenance day and we've upgraded our backup
> | bridge firewall to the Dec. 15, 2011 snapshot available from
> | ftp.openbsd.org and I'm getting this odd error when I boot it up.
> | Oddly enough, this only happens when connected to the switch that
> | original one is connected to (we swap them out each semester).
> | 
> | First, I use the upgrade method to go from snapshot to snapshot and
> | reboot
> | I run sysmerge to bring in the new configuration files from etc50.tgz
> | and xetc50.tgz ( I only have bsd* man* base* xbase* installed) and
> | reboot.
> | 
> | So as you can see the standard running -current and I've done several
> | upgrades now.
> | 
> | On my test switch (HP5304XL) it boots okay and I can reload the
> | firewall rules with no problem. When I connect it to my HP2910 where
> | the current firewall is running I cannot fully boot. If I press CTRL+C
> | during the starting network section it will continue to boot. If I
> | then run pfctl -e it states that PF is already enabled enabled but if
> | I run pfctl -Fr -f /etc/pf.conf I get the following.
> | 
> | # uvm_fault(0x80d2ff40, 0x0, 0, 1) -> e
> | kernel: page fault trap, code=0
> | Stopped at pf_translate+0x154: cmpw %r13w,0(%rsi)
> | ddb{0}>
> | 
> | keyboard is dead, no response at all from console. Any ideas?
> 
> Okay, I've gotten some off list requests for more information, which
> I'm hoping I'll be able to get for those people, but I'm now outside
> of my maintenance window and will likely need to schedule another
> outage or figure out how to reproduce it again.  The current bridge
> firewall running the following version does not exhibit the problem,
> but I'm not able to get a trace output at this time.  Maybe it's
> still at least somewhat useful reference for updates that may have
> happened. ( Yeah right, from Aug 8th until now.  Thousands of
> commits. ;) )
> 
> OpenBSD 5.0 (GENERIC.MP) #57: Mon Aug  8 14:58:00 MDT 2011
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> 
>

pf_translate+0x154 corresponds to the condition at pf.c:3765.
*pd->sport doesn't point to anything.  pd->sport is set to point
to the extracted header in the pf_setup_pdesc.  the problem is
that the header extraction happens based on the virtual_proto,
not proto, which can be different in the case of a fragment.

now, pf.c got it covered by the condition on line 3476 which
prevents pf_translate from running on fragments.  the only
other invocation of pf_translate is in the if_pflog.c:407
where we don't check for fragments.  therefore i think that
this is the problem.

the diff below should fix the problem.  it also doesn't make
sense to do af translation if we didn't manage to get our
shit done in the pf_translate and in the subsequent block.

ok?

Index: if_pflog.c
===
RCS file: /cvs/src/sys/net/if_pflog.c,v
retrieving revision 1.45
diff -u -p -r1.45 if_pflog.c
--- if_pflog.c  21 Oct 2011 15:45:55 -  1.45
+++ if_pflog.c  19 Dec 2011 20:36:32 -
@@ -404,7 +404,8 @@ pflog_bpfcopy(const void *src_arg, void 
if (pd.dport)
odport = *pd.dport;
 
-   if ((pfloghdr->rewritten = pf_translate(&pd, &pfloghdr->saddr,
+   if (pd->virtual_proto != PF_VPROTO_FRAGMENT &&
+   (pfloghdr->rewritten = pf_translate(&pd, &pfloghdr->saddr,
pfloghdr->sport, &pfloghdr->daddr, pfloghdr->dport, 0,
pfloghdr->dir))) {
m_copyback(pd.m, pd.off, min(pd.m->m_len - pd.off, pd.hdrlen),
@@ -422,7 +423,7 @@ pflog_bpfcopy(const void *src_arg, void 
pd.tot_len = min(pd.tot_len, len);
pd.tot_len -= pd.m->m_data - pd.m->m_pktdat;
 
-   if (afto)
+   if (pfloghdr->rewritten && afto)
pf_translate_af(&pd);
 
mlen = min(pd.m->m_pkthdr.len, len);

Re: OpenBSD 5.0 upgrade: em interface status "no carrier"

[Mike Belopuhov]

On Fri, Nov 18, 2011 at 4:01 AM, Sam Vaughan 
wrote:
> On 18/11/2011, at 12:59 PM, Sam Vaughan wrote:
>
>> Hi,
>>
>> After upgrading from OpenBSD 4.9 to OpenBSD 5.0, the "Intel 82579LM" and
> "Intel PRO/1000 MT (82574L)" devices on one of my servers no longer come
up.
>
> 
>
> If I'd bothered to compare those two dmesg outputs more closely I'd have
> noticed that OpenBSD 5.0 is simply enumerating the two interfaces in the
> opposite order.  What was em0 in 4.9 is now em1 in 5.0 and vice versa.
 Simply
> swapping the cable to the other port and _not_ moving the settings in
ifconfig
> to em1 fixes the problem.  Sorry for the noise.
>
> By the way, is there any reason why I should prefer the 82579LM to the
82574L
> or vice versa?
>
> Thanks,
>
> Sam
>
>

there's no reason you should consider either of them for any serious
task as both of them are desktop low performance versions.

82576EB is known to be a very good one for routing workloads.

http://www.intel.com/content/dam/doc/brochure/ethernet-controllers-phys-broch
ure.pdf

Re: PFSYNC - pf.conf best practice

[Mike Belopuhov]

On Fri, Oct 28, 2011 at 11:25 AM, Mike Belopuhov  wrote:
> On Thu, Oct 27, 2011 at 11:18 AM, Mike Belopuhov  wrote:
>>>> On 26-10-2011 20:32, Maxim Bourmistrov wrote:
>>>>> The side question, after observing 'systat -s1 states', is WHY
"failover"-side
>>>>> doubles exp. time??
>>>>> I'm more expected to have it like a "copy" of the current state of the
>>>>> master.
>>
>> i've started looking into the problem, but it's not yet apparent as to
>> why it happens.
>>
>
> ok, so what i've figured out is that expire timeouts are transferred
> as values of "rule->timeout[state->timeout]" and not absolute ones
> (to compensate for a local time difference).  but when you insert
> the rule into the state table you have to subtract the timeout value
> as pf_state_expires does a good job itself by adding those back.
>
> so here's a diff. i've tried to solve "no adaptive scaling" caveat as well.
>
> ok?
>

need to assign a rule pointer before dereferencing it, right?

Index: if_pfsync.c
===
RCS file: /cvs/src/sys/net/if_pfsync.c,v
retrieving revision 1.169
diff -u -p -u -p -r1.169 if_pfsync.c
--- if_pfsync.c 20 Oct 2011 08:57:26 -  1.169
+++ if_pfsync.c 28 Oct 2011 13:46:12 -
@@ -266,6 +266,8 @@ voidpfsync_bulk_status(u_int8_t);
 void   pfsync_bulk_update(void *);
 void   pfsync_bulk_fail(void *);

+u_int32_t pfsync_expires(const struct pf_state *, u_int32_t, u_int8_t);
+
 #define PFSYNC_MAX_BULKTRIES   12
 intpfsync_sync_ok;

@@ -578,13 +580,7 @@ pfsync_state_import(struct pfsync_state
/* copy to state */
bcopy(&sp->rt_addr, &st->rt_addr, sizeof(st->rt_addr));
st->creation = time_second - ntohl(sp->creation);
-   st->expire = time_second;
-   if (sp->expire) {
-   /* XXX No adaptive scaling. */
-   st->expire -= r->timeout[sp->timeout] - ntohl(sp->expire);
-   }

-   st->expire = ntohl(sp->expire) + time_second;
st->direction = sp->direction;
st->log = sp->log;
st->timeout = sp->timeout;
@@ -603,6 +599,8 @@ pfsync_state_import(struct pfsync_state
st->anchor.ptr = NULL;
st->rt_kif = NULL;

+   st->expire = pfsync_expires(st, ntohl(sp->expire), sp->timeout);
+
st->pfsync_time = time_uptime;
st->sync_state = PFSYNC_S_NONE;

@@ -948,7 +946,8 @@ pfsync_in_upd(caddr_t buf, int len, int
if (sync < 2) {
pfsync_alloc_scrub_memory(&sp->dst, &st->dst);
pf_state_peer_ntoh(&sp->dst, &st->dst);
-   st->expire = ntohl(sp->expire) + time_second;
+   st->expire = pfsync_expires(st, ntohl(sp->expire),
+   sp->timeout);
st->timeout = sp->timeout;
}
st->pfsync_time = time_uptime;
@@ -1019,10 +1018,12 @@ pfsync_in_upd_c(caddr_t buf, int len, in
else
pf_state_peer_ntoh(&up->dst, &st->dst);
}
+
if (sync < 2) {
pfsync_alloc_scrub_memory(&up->dst, &st->dst);
pf_state_peer_ntoh(&up->dst, &st->dst);
-   st->expire = ntohl(up->expire) + time_second;
+   st->expire = pfsync_expires(st, ntohl(up->expire),
+   up->timeout);
st->timeout = up->timeout;
}
st->pfsync_time = time_uptime;
@@ -2330,4 +2331,37 @@ pfsync_sysctl(int *name, u_int namelen,
default:
return (ENOPROTOOPT);
}
+}
+
+u_int32_t
+pfsync_expires(const struct pf_state *st, u_int32_t expire, u_int8_t ttype)
+{
+   u_int32_t   start;
+   u_int32_t   end;
+   u_int32_t   states;
+   u_int32_t   result;
+   u_int32_t   timeout;
+
+   if (expire == 0 || ttype >= PFTM_MAX)
+   return (time_second);
+
+   timeout = st->rule.ptr->timeout[ttype];
+   result = time_second - timeout + expire;
+   start = st->rule.ptr->timeout[PFTM_ADAPTIVE_START];
+   if (start) {
+   end = st->rule.ptr->timeout[PFTM_ADAPTIVE_END];
+   states = st->rule.ptr->states_cur;
+   } else {
+   start = pf_default_rule.timeout[PFTM_ADAPTIVE_START];
+   end = pf_default_rule.timeout[PFTM_ADAPTIVE_END];
+   states = pf_status.states;
+   }
+   if (end && states > start && start < end) {
+   if (states < end)
+   return (result + timeout * (end - states) /
+   (end - start));
+   else
+   return (time_second);
+   }
+   return (result);
 }

Re: PFSYNC - pf.conf best practice

[Mike Belopuhov]

On Thu, Oct 27, 2011 at 11:18 AM, Mike Belopuhov  wrote:
>>> On 26-10-2011 20:32, Maxim Bourmistrov wrote:
>>>> The side question, after observing 'systat -s1 states', is WHY 
>>>> "failover"-side
>>>> doubles exp. time??
>>>> I'm more expected to have it like a "copy" of the current state of the
>>>> master.
>
> i've started looking into the problem, but it's not yet apparent as to
> why it happens.
>

ok, so what i've figured out is that expire timeouts are transferred
as values of "rule->timeout[state->timeout]" and not absolute ones
(to compensate for a local time difference).  but when you insert
the rule into the state table you have to subtract the timeout value
as pf_state_expires does a good job itself by adding those back.

so here's a diff. i've tried to solve "no adaptive scaling" caveat as well.

ok?

Index: if_pfsync.c
===
RCS file: /cvs/src/sys/net/if_pfsync.c,v
retrieving revision 1.169
diff -u -p -u -p -r1.169 if_pfsync.c
--- if_pfsync.c 20 Oct 2011 08:57:26 -  1.169
+++ if_pfsync.c 28 Oct 2011 09:24:09 -
@@ -266,6 +266,8 @@ voidpfsync_bulk_status(u_int8_t);
 void   pfsync_bulk_update(void *);
 void   pfsync_bulk_fail(void *);

+u_int32_t pfsync_expires(const struct pf_state *, u_int32_t, u_int8_t);
+
 #define PFSYNC_MAX_BULKTRIES   12
 intpfsync_sync_ok;

@@ -578,13 +580,9 @@ pfsync_state_import(struct pfsync_state
/* copy to state */
bcopy(&sp->rt_addr, &st->rt_addr, sizeof(st->rt_addr));
st->creation = time_second - ntohl(sp->creation);
-   st->expire = time_second;
-   if (sp->expire) {
-   /* XXX No adaptive scaling. */
-   st->expire -= r->timeout[sp->timeout] - ntohl(sp->expire);
-   }
+   st->expire = pfsync_expires(st, ntohl(sp->expire),
+   sp->timeout);

-   st->expire = ntohl(sp->expire) + time_second;
st->direction = sp->direction;
st->log = sp->log;
st->timeout = sp->timeout;
@@ -948,7 +946,8 @@ pfsync_in_upd(caddr_t buf, int len, int
if (sync < 2) {
pfsync_alloc_scrub_memory(&sp->dst, &st->dst);
pf_state_peer_ntoh(&sp->dst, &st->dst);
-   st->expire = ntohl(sp->expire) + time_second;
+   st->expire = pfsync_expires(st, ntohl(sp->expire),
+   sp->timeout);
st->timeout = sp->timeout;
}
st->pfsync_time = time_uptime;
@@ -1019,10 +1018,12 @@ pfsync_in_upd_c(caddr_t buf, int len, in
else
pf_state_peer_ntoh(&up->dst, &st->dst);
}
+
if (sync < 2) {
pfsync_alloc_scrub_memory(&up->dst, &st->dst);
pf_state_peer_ntoh(&up->dst, &st->dst);
-   st->expire = ntohl(up->expire) + time_second;
+   st->expire = pfsync_expires(st, ntohl(up->expire),
+   up->timeout);
st->timeout = up->timeout;
}
st->pfsync_time = time_uptime;
@@ -2330,4 +2331,37 @@ pfsync_sysctl(int *name, u_int namelen,
default:
return (ENOPROTOOPT);
}
+}
+
+u_int32_t
+pfsync_expires(const struct pf_state *st, u_int32_t expire, u_int8_t ttype)
+{
+   u_int32_t   start;
+   u_int32_t   end;
+   u_int32_t   states;
+   u_int32_t   result;
+   u_int32_t   timeout;
+
+   if (expire == 0 || ttype >= PFTM_MAX)
+   return (time_second);
+   timeout = st->rule.ptr->timeout[ttype];
+   start = st->rule.ptr->timeout[PFTM_ADAPTIVE_START];
+   if (start) {
+   end = st->rule.ptr->timeout[PFTM_ADAPTIVE_END];
+   states = st->rule.ptr->states_cur;
+   } else {
+   start = pf_default_rule.timeout[PFTM_ADAPTIVE_START];
+   end = pf_default_rule.timeout[PFTM_ADAPTIVE_END];
+   states = pf_status.states;
+   }
+   result = time_second - timeout + expire;
+   if (end && states > start && start < end) {
+   if (states < end)
+   return (result + timeout * (end - states) /
+   (end - start));
+   else
+   return (time_second);
+   }
+
+   return (result);
 }

Re: PFSYNC - pf.conf best practice

[Mike Belopuhov]

On Wed, Oct 26, 2011 at 9:51 PM, Maxim Bourmistrov
 wrote:
>
> Well, it is idle so far as it is not able to take care of dhcp-clients -
dhcpd listens on CARP which is not available at the moment.
> This box is a slave to the named too, but updates of zone are not so
frequent due to the LAN-side.
>
> I'll try to boot back origin bsd, but as of my knowledge, lager updates
should not(if any) increase number of states on the other side,
> rather than remove many as requested and update many as requested.
>
> I had my thoughts about MCLBYTES affecting the rest of the code, but not
checked it out at all.
>
> pf.conf is ALLWAYS in sync on both sides. This is even stated clearly in
comments, incase any should take over after me.
>
> //maxim
>
>
> On Oct 26, 2011, at 8:50 PM, Camiel Dobbelaar wrote:
>
>> On 26-10-2011 20:32, Maxim Bourmistrov wrote:
>>> The side question, after observing 'systat -s1 states', is WHY
"failover"-side
>>> doubles exp. time??
>>> I'm more expected to have it like a "copy" of the current state of the
>>> master.
>>
>> Yes, the number of states should be roughly in sync on both firewalls.
>> I'd keep pf.conf in sync too (including all settings).
>>
>> Is the backup firewall really idle on all interfaces?
>>
>> Does it happen without the pfsync mtu changes too?  What does "netstat
>> -s -p pfsync" say?
>>
>> What do you see if you capture "pfctl -ss | sort" on both firewalls (at
>> the same time) and diff the outputs?
>>
>
>

this has nothing to do with mtu.  this bug^Wbehavior is there since
quite some time.  but given the code like this:

st->expire = time_second;
if (sp->expire) {
/* XXX No adaptive scaling. */
st->expire -= r->timeout[sp->timeout] - ntohl(sp->expire);
}

st->expire = ntohl(sp->expire) + time_second;

this is not something we can't anticipate.

i've started looking into the problem, but it's not yet apparent as to
why it happens.

cheers

Re: pfsync0 MTU

[Mike Belopuhov]

On Mon, Oct 24, 2011 at 12:18 PM, Maxim Bourmistrov
 wrote:
>
> Hi,
>
> I patched on side of this tandem

do you mean 'one'? then you should obviously patch both.
i mean, come on, you wanted to do some research on
your own, so do it.

> and had following setup:
>
> fw1: em0 mtu 9000, pfsync0 mtu 2048
> fw2: em0 mtu 9000, pfsync0 mtu 9000
>
> This produced "pfsync: failed to receive bulk update".
> If I change back to mtu 2048 states get propagated.
>
> I also changed hardmtu as dlg@ suggested.
>

it's not immediately clear if you have to change it so
you have to verify both situations.

>
> //maxim

Re: pfsync0 MTU

[Mike Belopuhov]

On Sat, Oct 22, 2011 at 20:14 +0200, Maxim Bourmistrov wrote:
> 
> On both sides I use em(4) with MTU 9000.
> Then tried to set the same value to the pfsync with success (ifconfig pfsync0 
> mtu 9000), but the actual value I see is 2048.
> 

ugh. i thought you've fixed up the source code.
i'm curious if it'll still work with a smaller mtu on the physical
interface :-)

Index: net/if_pfsync.c
===
RCS file: /cvs/src/sys/net/if_pfsync.c,v
retrieving revision 1.169
diff -u -p -r1.169 if_pfsync.c
--- net/if_pfsync.c 20 Oct 2011 08:57:26 -  1.169
+++ net/if_pfsync.c 22 Oct 2011 18:17:44 -
@@ -1294,8 +1294,8 @@ pfsyncioctl(struct ifnet *ifp, u_long cm
s = splnet();
if (ifr->ifr_mtu <= PFSYNC_MINPKT)
return (EINVAL);
-   if (ifr->ifr_mtu > MCLBYTES) /* XXX could be bigger */
-   ifr->ifr_mtu = MCLBYTES;
+   if (ifr->ifr_mtu > 65536)
+   ifr->ifr_mtu = 65536;
if (ifr->ifr_mtu < ifp->if_mtu)
pfsync_sendout();
ifp->if_mtu = ifr->ifr_mtu;

Re: pfsync0 MTU

[Mike Belopuhov]

On Thu, Oct 20, 2011 at 10:40 +0200, Maxim Bourmistrov wrote:
> Hi list,
> is there any reason for MTU on pfsync0 to be limited to 2048?

yes, when pfsync(4) was written, there was only one mbuf cluster
pool: MCLBYTES (2048) sized one.  now we have several.

> Any benefit from having it lager, say up to 9000?
> 

it should be possible to send out more updates at once therefore
calling output routines less often.  there might be delay concerns
though -- this should be investigated.

> I enabled MTU 9000 on syncdev and tried on pfsync0.

but does it work? are you getting state updates? you should have
lots of states to verify that huge packets actually get sent out.

btw, what's mtu size of your syncdev interface?

> As seen in tcpdump now, sync pkts are large but not as large as
> 9000(2048 limit).
> 
> //maxim

Re: anybody working on support for Cavium CN16xx crypto accelerator?

[Mike Belopuhov]

not that i'm aware of. but we'd be interested if someone would provide
hardware and documentation.

On Tue, May 17, 2011 at 8:46 AM, patrick.oesch...@bluewin.ch
 wrote:
> http://www.caviumnetworks.com/processor_security_nitroxPX.html
> thank you
> /pat

Re: IPSEC/SSL accelerator

[Mike Belopuhov]

On Tue, May 17, 2011 at 3:45 PM, Stuart Henderson  wrote:
> On 2011-05-17, patrick.oesch...@bluewin.ch  
> wrote:
>> i am looking for a IPSEC accelerator board for a company firewall to 
>> terminate multiple IPSEC tunnels (branches)
>>
>> expected IPSEC traffic: ~500mbit/s (100 tunnels)
>> any recommendations for a *pci express 4x* board handling this amount
>> of traffic?
>> thank you
>> /pat
>>
>>
>
> there hasn't been support for any newer bus-based accelerators
> added recently (overheads for these are typically rather high).
>
> currently if you want fast AES, you should be looking at the
> newer intel cpus with AESNI (and OpenBSD 4.9 or newer), but this
> doesn't fit your pcie 4x requirements.
>
>

unfortunately, aesni won't help you much here as you still have
to do not accelerated hmac which will cap the maximum throughput.

Re: Dell R310 - H200 Raid performance problem

[Mike Belopuhov]

On Thu, Feb 10, 2011 at 14:25 +0100, Lukasz Czarniecki wrote:
> Hi
> 
> I've bought a Dell R310 with H200 raid controller reported in dmesg as:
> Symbios Logic SAS2008. It uses mpii driver and has two hard drives
> configured in RAID 1.
> 
> Now it seems to work fine but i still have a problem with its
> performance. Raid is fully initialized.
> 
> How can I help to resolve this problem?
> 
> I'm doing simple benchmark:
> wget ftp.spline.de/pub/OpenBSD/4.8/sys.tar.gz
> time tar xzf ./sys.tar.gz
> 
> On the same hardware Linux unpacks it in less then two seconds.
> 
> Numbers for OpenBSD:
> 4.8 amd64 sp: 3m40.95s real 0m0.65s user 0m0.71s system
> 4.8 amd64 mp-stable: 3m43.36s real 0m0.48s user 0m0.98s system
> 4.9 amd64 sp: 3m47.72s real 0m0.51s user 0m0.69s system
> 4.9 i386 rd : 3m45.11s real 0m1.03s user 0m1.19s system
> 

Lukasz and me have figured out that disk write cache gets turned
off by the Dell firmware when you create a volume (it doesn't get
disabled if you use single drives):

http://support.dell.com/support/edocs/storage/storlink/h200/en/ug/html/features.htm#wp1062398

H200 doesn't have and there's no possibility to install an onboard
memory and the battery, so the device becomes pretty much useless
unless the operating system takes care of it.  Apparently Linux
does.  Should OpenBSD do the same?  In my opinion yes.

Lukasz has tested the patch below and it works fine for him.  I don't
have the hardware myself, so I'm not going to push it for the release,
but if someone thinks it's worth it, please speak up.


Index: mpii.c
===
RCS file: /home/cvs/src/sys/dev/pci/mpii.c,v
retrieving revision 1.37
diff -u -p -r1.37 mpii.c
--- mpii.c  29 Dec 2010 03:55:09 -  1.37
+++ mpii.c  17 Feb 2011 15:15:25 -
@@ -981,6 +981,52 @@ struct mpii_msg_sas_oper_reply {
u_int32_t   ioc_loginfo;
 } __packed;
 
+struct mpii_msg_raid_action_request {
+   u_int8_taction;
+#define MPII_RAID_ACTION_CHANGE_VOL_WRITE_CACHE(0x17)
+   u_int8_treserved1;
+   u_int8_tchain_offset;
+   u_int8_tfunction;
+
+   u_int16_t   vol_dev_handle;
+   u_int8_tphys_disk_num;
+   u_int8_tmsg_flags;
+
+   u_int8_tvp_id;
+   u_int8_tvf_if;
+   u_int16_t   reserved2;
+
+   u_int32_t   reserved3;
+
+   u_int32_t   action_data;
+#define MPII_RAID_VOL_WRITE_CACHE_DISABLE  (0x01)
+#define MPII_RAID_VOL_WRITE_CACHE_ENABLE   (0x02)
+
+   struct mpii_sge action_sge;
+} __packed;
+
+struct mpii_msg_raid_action_reply {
+   u_int8_taction;
+   u_int8_treserved1;
+   u_int8_tchain_offset;
+   u_int8_tfunction;
+
+   u_int16_t   vol_dev_handle;
+   u_int8_tphys_disk_num;
+   u_int8_tmsg_flags;
+
+   u_int8_tvp_id;
+   u_int8_tvf_if;
+   u_int16_t   reserved2;
+
+   u_int16_t   reserved3;
+   u_int16_t   ioc_status;
+
+   u_int32_t   action_data[5];
+
+   struct mpii_sge action_sge;
+} __packed;
+
 struct mpii_cfg_hdr {
u_int8_tpage_version;
u_int8_tpage_length;
@@ -1972,6 +2018,8 @@ int   mpii_req_cfg_page(struct mpii_softc
 
 intmpii_get_ioc_pg8(struct mpii_softc *);
 
+void   mpii_cache_enable(struct mpii_softc *);
+
 #if NBIO > 0
 intmpii_ioctl(struct device *, u_long, caddr_t);
 intmpii_ioctl_inq(struct mpii_softc *, struct bioc_inq *);
@@ -2175,6 +2223,9 @@ mpii_attach(struct device *parent, struc
goto free_dev;
}
 
+   /* enable write cache */
+   mpii_cache_enable(sc);
+
/* we should be good to go now, attach scsibus */
sc->sc_link.adapter = &mpii_switch;
sc->sc_link.adapter_softc = sc;
@@ -3206,6 +3257,45 @@ mpii_cfg_coalescing(struct mpii_softc *s
}
 
return (0);
+}
+
+void
+mpii_cache_enable(struct mpii_softc *sc)
+{
+   struct mpii_msg_raid_action_request *req;
+   struct mpii_device  *dev;
+   struct mpii_ccb *ccb;
+   int i;
+
+   ccb = scsi_io_get(&sc->sc_iopool, 0);
+   if (ccb == NULL)
+   return;
+
+   for (i = 0; i < sc->sc_max_devices; i++) {
+   if (sc->sc_devs[i] == NULL ||
+   !ISSET(sc->sc_devs[i]->flags, MPII_DF_VOLUME))
+   continue;
+
+   dev = sc->sc_devs[i];
+
+   ccb->ccb_state = MPII_CCB_READY;
+   ccb->ccb_rcb = NULL;
+   ccb->ccb_done = mpii_empty_done;
+
+   req = ccb->ccb_cmd;
+   bzero(req, sizeof(*req));
+   req->function = MPII_FUNCTION_RAID_ACTION;
+   req->action = MPII_RAID_ACTION_CHANGE_VOL_WR

Re: Migrating from isakmpd to iked: interface name not recognized

[Mike Belopuhov]

On Mon, Dec 13, 2010 at 18:50 +0100, Axel Rau wrote:
> Hi all,
> 
> in the man page for iked.conf, I read:
> "Addresses can be specified in CIDR notation (matching netblocks), as
> symbolic host names, interface names, or interface group names."
> 
> In my iked.conf, I have
>local pppoe0
> but iked -vn complains:
>   no IP address found for pppoe0
>   /etc/iked.conf: 26: could not parse host specification
> .
>   ifconfig pppoe0 | grep inet
> shows:
>   inet 79.243.41.99 --> 87.186.224.28 netmask 0x
> 

iked is still in the works.  you shouldn't try to migrate to it unless
you can come up with fixes for such problems yourself.

mask2prefixlen functions are taken from bgpd.  OK?

Index: parse.y
===
RCS file: /home/cvs/src/sbin/iked/parse.y,v
retrieving revision 1.14
diff -u -p -u -p -r1.14 parse.y
--- parse.y 17 Nov 2010 16:43:45 -  1.14
+++ parse.y 14 Dec 2010 15:57:27 -
@@ -266,6 +266,8 @@ struct ipsec_addr_wrap  *host_v4(const ch
 struct ipsec_addr_wrap *host_dns(const char *, int);
 struct ipsec_addr_wrap *host_if(const char *, int);
 struct ipsec_addr_wrap *host_any(void);
+u_int8_tmask2prefixlen(struct sockaddr_in *);
+u_int8_tmask2prefixlen6(struct sockaddr_in6 *);
 voidifa_load(void);
 int ifa_exists(const char *);
 struct ipsec_addr_wrap *ifa_lookup(const char *ifa_name);
@@ -1712,6 +1714,65 @@ host_any(void)
return (ipa);
 }
 
+u_int8_t
+mask2prefixlen(struct sockaddr_in *sa_in)
+{
+   in_addr_t ina = sa_in->sin_addr.s_addr;
+
+   if (ina == 0)
+   return (0);
+   else
+   return (33 - ffs(ntohl(ina)));
+}
+
+u_int8_t
+mask2prefixlen6(struct sockaddr_in6 *sa_in6)
+{
+   u_int8_t l = 0, *ap, *ep;
+
+   /*
+* sin6_len is the size of the sockaddr so substract the offset of
+* the possibly truncated sin6_addr struct.
+*/
+   ap = (u_int8_t *)&sa_in6->sin6_addr;
+   ep = (u_int8_t *)sa_in6 + sa_in6->sin6_len;
+   for (; ap < ep; ap++) {
+   /* this "beauty" is adopted from sbin/route/show.c ... */
+   switch (*ap) {
+   case 0xff:
+   l += 8;
+   break;
+   case 0xfe:
+   l += 7;
+   return (l);
+   case 0xfc:
+   l += 6;
+   return (l);
+   case 0xf8:
+   l += 5;
+   return (l);
+   case 0xf0:
+   l += 4;
+   return (l);
+   case 0xe0:
+   l += 3;
+   return (l);
+   case 0xc0:
+   l += 2;
+   return (l);
+   case 0x80:
+   l += 1;
+   return (l);
+   case 0x00:
+   return (l);
+   default:
+   fatalx("non continguous inet6 netmask");
+   }
+   }
+
+   return (l);
+}
+
 /* interface lookup routintes */
 
 struct ipsec_addr_wrap *iftab;
@@ -1721,6 +1782,8 @@ ifa_load(void)
 {
struct ifaddrs  *ifap, *ifa;
struct ipsec_addr_wrap  *n = NULL, *h = NULL;
+   struct sockaddr_in  *sa_in;
+   struct sockaddr_in6 *sa_in6;
 
if (getifaddrs(&ifap) < 0)
err(1, "ifa_load: getifaddrs");
@@ -1737,17 +1800,13 @@ ifa_load(void)
if ((n->name = strdup(ifa->ifa_name)) == NULL)
err(1, "ifa_load: strdup");
if (n->af == AF_INET) {
-   n->af = AF_INET;
-   memcpy(&n->address, ifa->ifa_addr,
-   sizeof(struct sockaddr_in));
-   memcpy(&n->mask, ifa->ifa_addr,
-   sizeof(struct sockaddr_in));
+   sa_in = (struct sockaddr_in *)ifa->ifa_addr;
+   memcpy(&n->address, sa_in, sizeof(*sa_in));
+   n->mask = mask2prefixlen(sa_in);
} else if (n->af == AF_INET6) {
-   n->af = AF_INET6;
-   memcpy(&n->address, ifa->ifa_addr,
-   sizeof(struct sockaddr_in6));
-   memcpy(&n->mask, ifa->ifa_addr,
-   sizeof(struct sockaddr_in6));
+   sa_in6 = (struct sockaddr_in6 *)ifa->ifa_addr;
+   memcpy(&n->address, sa_in6, sizeof(*sa_in6));
+   n->mask = mask2prefixlen6(sa_in6);
}
n->next = NULL;
n->tail = n;

Re: kernel hangs by many connections (reproducable)

[Mike Belopuhov]

On Sun, Sep 12, 2010 at 11:26 +0600, Anton Maksimenkov wrote:
> Hello.
> I use my OBSD machine to test some server on another machine. They are
> connected by pathcord, 1Gbit network cards are used.
> Test program (uses kqueue) do many (I want thousands) connections to
> server. Write query, read answer.
> And it tries to keep that much connections by doing as much new
> connections as needed.
> 
> When number of connections kept below 100 - all ok. But if I raise
> them (upto about 500-1000) the program start these connections, do
> some write/read (show about 10-20 successful reads) and the kernel
> hangs. 1-2 sec after start.
> Tweaks - kern.maxfiles=16384 and openfiles-cur/max=8192 for my user.
> 
> Info from ddb (see dmesg below):
> 
> ddb> show panic
> the kernel did not panic
> 
> ddb> trace
> Debugger(0,3f8,0,0,1) at Debugger+0x4
> comintr(d1571000) at comintr+0x287
> Xrecurse_legacy4() at Xrecurse_legacy4+0xb3
> --- interrupt ---
> pool_do_get(d0a10b60,0,0,0,60) at pool_do_get+0x2c2
> pool_get(d0a10b60,0,8000,0,0) at pool_get+0x54
> m_gethdr(1,1,8000,369e99,0) at m_gethdr+0x39
> m_clget(0,1,d1526054,800,d03e1aeb) at m_clget+0x10a
> re_newbuf(d1526000,10,d999eb48,d02b30cc,d1526000) at re_newbuf+0x35
> re_rx_list_fill(d1526000,20,60,58,d1520010) at re_rx_list_fill+0x21
> re_rxeof(d1526000,d9799800,3e,10,10) at re_rxeof+0x37c
> re_intr(d1526000) at re_intr+0x12a
> Xrecurse_legacy11() at Xrecurse_legacy11+0xb7
> --- interrupt ---
> filt_soread(d9a5bdc0,0,0,d9a5bd98,d9a5bd98) at filt_soread+0x1
> selwakeup(d9a5bdbc,d9b08300,d9b08200,d9b08300,d9a5bd98) at selwakeup+0x22
> sowakeup(d9a5bd4c,d9a5bd98,14,d999ed24,1) at sowakeup+0x1d
> tcp_input(d9b08300,14,0,0,6) at tcp_input+0x26ac
> ipv4_input(d9b08300,0,d999ede8,d0202089,d03d0058) at ipv4_input+0x42a
> ipintr(d03d0058,d09e0010,10,d5d10010,d09e72c0) at ipintr+0x49
> Bad frame pointer: 0xd999ede8
> 

there's a forgotten splx in the driver.  that explains why system
appears to be hung.  OK?

Index: re.c
===
RCS file: /home/cvs/src/sys/dev/ic/re.c,v
retrieving revision 1.128
diff -u -p -u -p -r1.128 re.c
--- re.c27 Aug 2010 17:08:00 -  1.128
+++ re.c5 Oct 2010 08:27:05 -
@@ -2042,8 +2042,10 @@ re_init(struct ifnet *ifp)
if (sc->sc_hwrev != RL_HWREV_8139CPLUS)
CSR_WRITE_2(sc, RL_MAXRXPKTLEN, 16383);
 
-   if (sc->rl_testmode)
+   if (sc->rl_testmode) {
+   splx(s);
return (0);
+   }
 
mii_mediachg(&sc->sc_mii);

Re: OpenBSD on DELL R310

[Mike Belopuhov]

2010/9/29 Bjvrn Sandell :
> Hi,
>
> Is anyone running OpenBSD on a DELL R310? With a H200 raid controller? Any
> issues?
>
> Thanks,
> --
> Bjvrn Sandell 
>
>

according to the mpii(4) manpage, this is supposed to be LSI SAS2008
that is well supported in OpenBSD 4.8+.

cheers.

Re: [dera...@cvs.openbsd.org: Re: I would like to send this to misc@ and security-announce@, from me.]

[Mike Belopuhov]

C'mon, ftp.kd86.com was delisted from the ftp.html page on Mon Apr 6.
Can you just stop bashing Wim?  It doesn't make anyone happier (except
Theo probably).  Or maybe we should rush searching the whole fscking
internet for the incorrect OpenBSD mirrors?  Chill out, dudes.

On Thu, Apr 30, 2009 at 11:21 -0600, Bob Beck wrote:
>  
>   Users are cautioned about rogue ftp sites claiming to have OpenBSD.
>  
>   The best place to get OpenBSD is from an official CD set, produced in
> a secured location
>  
>   It has come to our attention that some ftp sites (ftp.kd85.com) which
> are not official OpenBSD mirrors are purporting to serve OpenBSD 4.5
> at this time. We have noted that what is actually present in the 4.5
> directory is not 4.5, but rather a late development cycle snapshot which
> they have moved into place claiming it is 4.5. 
>  
>   While we have no problem with anyone mirroring OpenBSD for the good 
> of the user community, we do believe that people who offer up the wrong
> thing are being deceptive and will hurt the userbase - particularly when
> the packages being offered up are not the release versions. 
>  
>   please ensure you look at http://www.openbsd.org/ftp.html when
> choosing to do an ftp install, and don't be fooled by someone "phishing"
> for your ftp traffic.

Re: Please test this on all available nVidia AHCI-capable controllers

[Mike Belopuhov]

On Sat, Dec 13, 2008 at 13:12 +, Mike Belopuhov wrote:
> Including MCP65, MCP67, MCP73, MCP77 and MCP79 families.
> 
> If you see something like this in your dmesg:
> 
> pciide1 at pci0 dev 9 function 0 "NVIDIA MCP77 AHCI" rev 0xa2: \
> DMA (unsupported)
> 
> please make sure you've switched your SATA controller to the native
> (or AHCI mode) in the BIOS.
> 
> ** ACHTUNG! ACHTUNG! ***
> 
> If you switch your controller from compatible to native mode, disk
> names will be changed too!  From wdX to sdX.  Please take this into
> account.
> 
> 
> 
> The patch should apply cleanly on 4.3, 4.4 and -current.
> 
> Please report back any breakage and success if you've switched to AHCI
> mode.
> 
> Cheers.
> 

i was too smart about this one-liner and giovanni@ was the only one
to respond.

please test this on any motherboards with an nvidia chipset.

Index: dev/pci/ahci.c
===
RCS file: /cvs/src/sys/dev/pci/ahci.c,v
retrieving revision 1.145
diff -u -p -r1.145 ahci.c
--- dev/pci/ahci.c  13 Dec 2008 11:49:54 -  1.145
+++ dev/pci/ahci.c  16 Dec 2008 16:15:12 -
@@ -648,6 +648,12 @@ ahci_pci_attach(struct device *parent, s
}
}
 
+   /*
+*  nVidia AHCI controllers need special handling
+*/
+   if (PCI_VENDOR(pa->pa_id) == PCI_VENDOR_NVIDIA)
+   sc->sc_flags |= AHCI_F_IGN_FR;
+
if (pci_intr_map(pa, &ih) != 0) {
printf(": unable to map interrupt\n");
return;

Please test this on all available nVidia AHCI-capable controllers

[Mike Belopuhov]

Including MCP65, MCP67, MCP73, MCP77 and MCP79 families.

If you see something like this in your dmesg:

pciide1 at pci0 dev 9 function 0 "NVIDIA MCP77 AHCI" rev 0xa2: \
DMA (unsupported)

please make sure you've switched your SATA controller to the native
(or AHCI mode) in the BIOS.

** ACHTUNG! ACHTUNG! ***

If you switch your controller from compatible to native mode, disk
names will be changed too!  From wdX to sdX.  Please take this into
account.



The patch should apply cleanly on 4.3, 4.4 and -current.

Please report back any breakage and success if you've switched to AHCI
mode.

Cheers.

Index: ahci.c
===
RCS file: /cvs/src/sys/dev/pci/ahci.c,v
retrieving revision 1.145
diff -u -p -u -r1.145 ahci.c
--- ahci.c  13 Dec 2008 11:49:54 -  1.145
+++ ahci.c  13 Dec 2008 12:58:32 -
@@ -648,6 +648,12 @@ ahci_pci_attach(struct device *parent, s
}
}
 
+   /*
+*  nVidia AHCI controllers need special handling
+*/
+   if (ad->ad_vendor == PCI_VENDOR_NVIDIA)
+   sc->sc_flags |= AHCI_F_IGN_FR;
+
if (pci_intr_map(pa, &ih) != 0) {
printf(": unable to map interrupt\n");
return;

Re: tcpdump | more doesn't produce output

[Mike Belopuhov]

On Thu, Jul 28, 2005 at 15:26 -0700, Michael Favinsky wrote:
> Has anyone tried a tcpdump | more ? Or a tcpdump | grep?
> 
> When I try to pipe tcpdump output to either more or grep I don't get any
> network data output. Anyone have any explanation for this behavior?
>

use stdout line buffered mode: tcpdump -l.

Re: static route files

[Mike Belopuhov]

On Thu, Jul 14, 2005 at 23:22 -0700, Reza Muhammad wrote:
> dear All,
> 
> In which file should i put my static route entry ?
> 

in hostname.if(5).

> regards
> reza
> 
> 
>   
> 
> Start your day with Yahoo! - make it your home page 
> http://www.yahoo.com/r/hs 

Re: built php4 from ports, but no mysql or postgresql support...how to enable support?

[Mike Belopuhov]

On Thu, May 12, 2005 at 14:13 -0400, Rick Barter wrote:
> I have Googled and searched the archives, but I'm not finding what I'm 
> looking for.
> 
> I have installed php4 from ports and realize after researching an erro 
> I was getting and viewing info.php that I don't have support for mysql 
> and postgresql.
> 
> Is there a way I can add support for these databases after the fact? 
> If not, how should I go about recompiling with these options.  I see 
> in the Makefile that these "flavors" are available, but I can't figure 
> out how to tell it I want to use a certain flavor.
> 

I'd go with something like this (for mysql):

$ cd /usr/ports/www/php4/extensions
$ sudo env SUBPACKAGE=-mysql FLAVOR="no_curl no_dba no_dbase \
no_ldap no_pgsql no_snmp no_mcrypt no_imap no_domxml \
no_mhash no_sybase_ct no_xmlrpc no_xslt" make install \
clean CLEANDEPENDS=Yes

> Can someone point me to a man page or some documentation?
> 
> rvb

Re: Mail from doubleclick.com sites

[Mike Belopuhov]

On Thu, May 12, 2005 at 11:45 -0500, Dave Feustel wrote:
> Is there any straightforward way to
> block email from locations that whois shows
> to have been created by doubleclick.com?
> 

something like this maybe?...

LOCAL_CONFIG
Kcheckblocked regex [EMAIL PROTECTED]
(doubleclick.com)

LOCAL_RULESETS
HReceived: $>+CheckReceived

SCheckReceived
R$+ $: $(checkblocked $1 $)
[EMAIL PROTECTED]   $#error $@ 5.7.1 $: 550 Access denied