Re: Meaning of file flags

[Rocky Hotas]

On mag 23  7:56, Greg Troxel wrote:
> I just pulled out "The Design and Implementation of the 4.4BSD Operating
> System".  It talks about immutable, append-only, and nodump on page 263.
> There is no mention of "archive".
[...]

I can not give much contribution to this, but thank you so much for your
research, the sources, and your considerations: they are useful for anyone
getting in touch with this feature for the first time.

Rocky

Re: Meaning of file flags

[Rocky Hotas]

First of all, thanks to both for your help. The meaning of nodump is
clear now.

On mag 22 18:45, Jan Schaumann wrote:
>
> Does the attached diff help?

Yes! I still have a couple of doubts.

The difference between "system flag" and "user flag". The former can be
applied by the super-user and the latter by the non-privileged user. Is
there any other difference between them?

> +The
> +.Va arch
> +flag is only used in connection with certain
> +filesystems (e.g., MS-DOS), where it indicates whether
> +a file has been modified since it was last backed up.

Considering the first lines of the section `Usage' here:

 

I suggest considering to write: ``in connection with certain filesystems
(e.g., MS-DOS), where it is related to the creation or modification of a
file before its backup'', or something similar: there is no homogeneous
use of the archive bit across these filesystems.

Bye,

Rocky

Meaning of file flags

[Rocky Hotas]

Hello!
In a recent and very interesting thread here, file flags are mentioned.
Manpage chflags(1) shows their list. But what about their meaning?
I've found this old message:

 

where short descriptions of flags are shown. However, I still can't
understand the meaning of `nodump' and `arch'.
Is there another reference/document/message about them?
Bye,

Rocky

Re: Assembly example (nasm) for amd64 in NetBSD

[Rocky Hotas]

On apr 14 12:10, RVP wrote:
[...]
> 
> No, the stuff in /usr/pkg/share/doc/nasm.
> 
> HTH,

Thank you so much for all the clarifications: they solved my doubts and
they definitely have been a huge help.

Rocky

Re: Assembly example (nasm) for amd64 in NetBSD

[Rocky Hotas]

On apr 14  8:40, RVP wrote:
> 
> Ah, the `version' should be int32 not int64.

Even if this is intended to be amd64 code, built as `nasm -f elf64'?
IIUC, you replaced `db' (``define byte'') and `dq' (``define quadword'',
that should be 8 bytes) with `dd' (``define double word'', that is 4
bytes, 32 bit). So the section only contains now `dd's.

I can't understand the ``NetBSD string'', which is the concatenation of
the 6 characters `NetBSD', the NULL character, and `$'. Is this a
conventional string? Where can I find it?
Also, in the code comments, it is stated to be ``8 bytes'' and it is in
the first version of your code, but now it is only 6 bytes, `NetBSD'. Is
it the same?

> And, looking at the
> nasm docs. suggests that sections can be annotated to match what
> the toolchain adds, This is better:
[...]
> +section .note.netbsd.ident note alloc noexec nowrite align=4

You added `note alloc noexec nowrite align=4'. I didn't know much about
the toolchain, so I'm not able to understand well these new strings.
By `nasm docs' you refer to the manpage nasm(1)?

Bye!

Rocky

Re: Assembly example (nasm) for amd64 in NetBSD

[Rocky Hotas]

On apr 13 21:25, Jason Mitchell wrote:
> Rocky,
> 
> The error means that the executable can't run on the CPU you have. Definitely 
> NOT bad code. Running "ldd testcpu.o" might give you more of a idea of what's 
> wrong.
> 
> HTH,
> 
> Jason M.

The error it gives is

$ ldd testcpu.o
ldd: testcpu.o: invalid ELF class 2; expected 1

But after adding the `.note.netbsd.iden' section in the assembly code,
as suggested by RVP, the file runs and works as expected.

Rocky

Re: Assembly example (nasm) for amd64 in NetBSD

[Rocky Hotas]

On apr 14  1:46, RVP wrote:
> 
> Try adding a ".note.netbsd.ident" section:
> 
> ```
> $ diff -u testcpu.s.orig testcpu.s
> --- testcpu.s.orig2023-04-14 01:42:08.984238899 +
> +++ testcpu.s 2023-04-14 01:41:06.204866303 +
> @@ -23,3 +23,11 @@
>  mov rax, 1
>  mov rbx, 0
>  syscall
> +
> +section .note.netbsd.ident
> +
> +   dd 7  ; ELF_NOTE_NETBSD_NAMESZ
> +   dd 4  ; ELF_NOTE_NETBSD_DESCSZ
> +   dd 1  ; ELF_NOTE_TYPE_NETBSD_TAG
> +   db 'NetBSD',0,'$' ; NetBSD string
> +   dq 90300  ; NetBSD version 9.3.0
> $
> ```
> 
> -RVP

It works, thank you!

Rocky

Assembly example (nasm) for amd64 in NetBSD

[Rocky Hotas]

Hello!
I was looking for a simple assembly code, a `Hello world' for example,
which runs on NetBSD, written and compiled with nasm.
I tried a code to print the CPU vendor string through cpuid: the entire
code is copied after the text of this e-mail. It works on FreeBSD. On
NetBSD 9.0 I built it with

nasm -f elf64 -o testcpu.o testcpu.s
ld -s -o testcpu testcpu.o

This gives no error, however the file can not execute. If I try to run
the file, I obtain:

/bin/sh: Cannot execute ELF binary ./testcpu

The only example I found is

 https://wiki.netbsd.org/examples/netbsd_assembly/

However, it is quite confusing and it is for i386, while I would like to
use amd64.
I can't understand if the error is due to a bad linking or building, or
if the code itself has some wrong instruction.
Is there a NetBSD nasm amd64 example? Or some hints that I should follow?
Bye!

Rocky


Here is my code, that is the contents of file testcpu.s:


section .bss

buff resb 12

section .text
global _start

_start:
  
mov eax, 0
cpuid

mov dword [buff+0], ebx 
mov dword [buff+4], edx 
mov dword [buff+8], ecx 

mov rax, 4
mov rdi, 1
mov rsi, buff
mov rdx, 12
syscall

mov rax, 1
mov rbx, 0
syscall

Re: Option -p in apropos(1)

[Rocky Hotas]

On giu 03 21:35, David H. Gutteridge wrote:

> This (along with other fixes) has now been pulled up to the netbsd-9
> branch, so the next releng build of 9.2_STABLE will include it.

Great! Thank you :)

Rocky

Re: Option -p in apropos(1)

[Rocky Hotas]

On mag 16 21:51, David H. Gutteridge wrote:
> It does accept $PAGER as a definition of what tool to use for the
> purpose, though that wasn't documented in the apropos(1) man page. I
> just added it.

FTR, for some reason in my default env(1) variable PAGER is not set: it
is empty. However, if (for example) I try to set it to `more', apropos(1)
behaves as described yesterday, so there are no improvements.

Rocky

Re: Option -p in apropos(1)

[Rocky Hotas]

On mag 16 20:33, RVP wrote:
> 
> apropos should call pclose(). Please try this patch below:
> 

I added the line in my repository sourcecode. Building a kernel is
straightforward, but what about this case? How to build and use apropos
from the sourcecode?

Rocky

Re: Option -p in apropos(1)

[Rocky Hotas]

On mag 16 21:05, Steffen Nurpmeso wrote:
> 
> I have no idea but does the program combo honour $LESS

Sorry, I can understand what you are meaning. Variable $LESS in my shell
appears to be empty.

> does it contain use of alternate screen and quit-at-eof?

I don't know how to verify this.

> What does "LESS= apropos X" do?

It prints the apropos(1) results in the default appearance, without a
pager, as `apropos X' would do.

Rocky

Re: Option -p in apropos(1)

[Rocky Hotas]

On mag 16 18:11, Rocky Hotas wrote:
> 
> Am I doing something wrong?

Some very funny updates.

First, I'm on NetBSD 9.2 amd64 (but I also tested the same things in
another machine with NetBSD 9.0 amd64).

The first time I run `apropos' with option `-p', it doesn't work (as
written in the previous message). In a newly created terminal (an
st(1) instance) it may print:

...
perl5320delta (1)   what is new for perl v5.32.0
...Configure  4 For clang++, add "#include " to Configure's
probes for "futimes", "strtoll", "strtoul", "strtoull", "strtouq",
otherwise the probes would fail to compile.  4 Use a compile and run
test for "lchown" to satisfy clang++ which should more...

--More--(byte 3792)myhostname$

Where `myhostname$' is my bash prompt. So, more(1) tries to start, but
it immediately exits.

If I run apropos(1) for a second time, changing the argument, then it
works as expected: more(1) is active and the screen can be scrolled.

The argument change is essential, because if I run apropos(1) a second
time with the same argument, the output ends again with

--More--(byte 3792)myhostname$

as before. Also, if I run

apropos -p argument1
apropos -p argument2
apropos -p argument1

the last time I get again

--More--(byte 3792)myhostname$

as if this was related to some dislike towards argument1.

Rocky

Option -p in apropos(1)

[Rocky Hotas]

Hello!

I was trying to run apropos(1) with option -p. According to the man
page, it should

 -p  Turn on pager formatting, and pipe the results through a pager
 (defaulting to more(1)).

Pager formatting is enabled by default (so, it doesn't need neither
option -p, nor option -P), but the pager is not activated. Results
are presented as if the option was not specified.

Am I doing something wrong?

Rocky

Re: sysctl and screen brightness

[Rocky Hotas]

On set 10 10:39, Benny Siegert wrote:
> It depends on what the exact machine is. Try looking through the
> output of "sysctl hw" or "sysctl machdep" for options with
> "brightness", "backlight", or similar in their names.

I have (in a laptop)

000:02:0: Intel Sandy Bridge (mobile) GT2 Integrated Graphics Device (VGA 
display, revision 0x09)
001:00:0: ATI Technologies Radeon HD 6400M (VGA display)

I think the first one is not used. According to your suggestion, the
only relevant items in the whole `sysctl -a' seem to be:

hw.acpi.acpiout1.brightness = 100
hw.acpi.acpiout2.brightness = 100
hw.acpi.acpiout6.brightness = 100
hw.drm2.i915.invert_brightness = 0
hw.drm2.radeon_backlight = -1

If I try (as root) to change the values of any of them, the operation is
permitted, but it has no effect (hw.acpi.acpiout* still remains at their
value 100), or it is not permitted:

sysctl: hw.drm2.i915.invert_brightness: Operation not permitted

sysctl: hw.drm2.radeon_backlight: Operation not permitted

In fact, I am not able to modify the brightness of my screen: if I press
the F2-F3 keys which should do this, nothing changes.

This is just FYI, as another example of hardware. Maybe the NetBSD driver
still doesn't offer a full flexibility with some configurations like
this one.

Bye!

Rocky

A proposal for code explanation sessions

[Rocky Hotas]

Hello!

During this year's NetBSD Annual General Meeting on IRC (May 22nd),
several participants suggested the activation of some code tutorial
sessions performed by the developers.

NetBSD sourcecode is in fact still a mysterious place for the new
users, or just for those who read it for the first time. According
to the chat, it would be great if some developer could show how they 
work on the sourcecode, for example with a live session, with any
mean: a Youtube live, or even just an IRC live session.

This arouse several difficulties: not only as regards the lack of
time, but also the difficulties involved in understanding what a
developer is doing (some automated operations, with obvious meaning
for a skilled developer, are meaningless for someone else).

I would like to re-open this discussion and, in particular, I would
like to suggest something maybe simpler. Instead of explaining why
some portion of code is being updated and how, it would be very
useful to show and explain the code just as it is. For example, a
volunteer developer could take an important, well known sourcecode
file, and explain it line by line; as a starting trivial file,
something like src/bin/hostname/hostname.c can be considered, then
moving to something more complicated like src/sbin/init/init.c.

While I'm sure it's not possible to actually explain every line,
the principles used to write the code "that way", to place that
function in that point, to call that function with those argument
values, could be shown.

I believe that some gentle introduction to kernel code would be
extremely useful to the whole community, giving anyone more means
to craft the code by himself, to improve it, to learn and eventually
(why not?) to become a developer. It would be a huge enhancement
for an "open-source" community, because the code would not only be
available, but also comprehensible.

What do you think about it? Would some developer be available for
this?

Bye!

Rocky

System bootstrap and process with pid 0

[Rocky Hotas]

Hello!
In this book

 

(Sec. 11.1, p. 239) it is stated that

``After bootstrap, the system initialization is started in main
function of sys/kern/init_main.c'', which creates a process with
pid 0 (as specified in the comments at the beginning of the function
code).

On the other hand, in this blog post

 

it is stated that the init ``process owns all the other processes
in the computer, either directly or indirectly''; moreover, in the
subsequent examples (and also in the images) it seems that init is
the actual first process executed by the OS and that any other
process is originated from it.

Shouldn't then init be that process launched by init_main.c? Instead, I
found:

$ ps -aux -p 0
USER PID %CPU %MEM VSZ   RSS TTY STAT STARTEDTIME COMMAND
root   0  0.0  0.2   0 15172 ?   OKl  Mon10AM 9:57.64 [system]

$ ps -aux -p 1
USER PID %CPU %MEM   VSZ  RSS TTY STAT STARTEDTIME COMMAND
root   1  0.0  0.0 22700 1244 ?   Is   Mon10AM 0:00.01 init

So, what exactly is that process with pid 0 and what is its relation
with init?

Bye!

Rocky

Re: Some questions about build.sh, machine, -u and tools

[Rocky Hotas]

Sorry if you are receiving this message for the second time. I can't see
it in the ML, so I'm resending it.

On mag 05 13:21, Martin Husemann wrote:

> No, but you may have the choice wich kernel/userland you want to run
> on the same physical hardware. An example is my cubietruck:
> I run it big-endian as:
> 
> [~] martin@space-truckin > uname -p -m
> evbarm earmv7hfeb
> 
> but most people use exactly the same hardware to run earmv7hf (little endian)
> instead.
> 
> On this board the difference is a simple "set endianess" instruction early
> in the kernel, but on some other machines it is a physical jumper on the 
> board.
> 
> This is not a very typical case though.

It's the first time I "hear" (actually, read) about it. It offers
great flexibility!

> The "compat" cases are more common: on arm machines you can execute
> binaries for eabi (the default) and oabi. On aarch64 machines you could
> run aarch64 (native 64bit), eabi (new 32bit abi) or oabi (old 32bit abi).

This way, MACHINE seems to represent the entirety of the hardware
devices which compose a specific system. It doesn't (it can't!)
change, of course.  Then, given this "static" hardware, you may make some
choices about endianness (as in the cubietruck example) and/or ABI (as
in this compat example) and/or some other features: this determines
the architecture, the instruction set you choose to build the kernel
for (the instruction set you choose to run on the actual CPU), and
this architecture is exactly MACHINE_ARCH.

> I guess now you see the complex spread sheet and understand why you have
> been confused initally - hope it got better during this thread ;-)

Oh, I think (I hope!) yes, it's becoming to be more clear, I really
couldn't figure it out before.

Thank you so much :)

Rocky

Re: Some questions about build.sh, machine, -u and tools

[Rocky Hotas]

Sorry if you are receiving this message for the second time. I can't see
it in the ML, so I'm resending it.

On mag 05 15:34, Valery Ushakov wrote:

> Yes.  It's just formatted using mdoc, but is not installed as part of
> system manual pages.

Ok! This is what produced that header.

> I think Martin explained it pretty well in another followup.

Yes, it was very useful :)

> If by the "directory tools" you mean the one in the build directory
> (it took me a while to realize that), then yes.  It's called an objdir
> and is a standard make(1) feature.  Please check make(1).

Sorry, it was not clear. Assume that you run, on a NetBSD 9.99.81 amd64
host system,

./build.sh -U -O ~/obj -j2 -m evbarm -a aarch64 tools

(as in par. 32.1 of The NetBSD Guide)

then you will have, in ~/obj, two subdirectories:

tooldir.NetBSD-9.99.81-amd64
tools

By "directory tools" I was meaning the last one listed above,
`~/obj/tools'. I hope that ~/obj is in this case what you mean by
"build directory".

As regards ~/obj/tools and running make(1) in this directory,

 .OBJDIR A path to the directory where the targets are built.

Now I understand why you wrote ``tools is the objdir for the
src/tools, a place where tools are built'' in the previous message.

> Yes, we build our own make, sed, etc as part of the tools build and
> install them in $TOOLDIR/bin.  We add nb- prefix to the installed
> versions to avoid name clashes with the host programs.  E.g. GNU
> programs (can) do the same, using g- prefix (and in some cases that g-
> even fused into the name (gcc, gawk)).

Oh, amazing! And I guess `nb' stand for `(N)et(B)SD'.

Thank you so much,

Rocky

Re: Some questions about build.sh, machine, -u and tools

[Rocky Hotas]

On mag 05 13:21, Martin Husemann wrote:

> No, but you may have the choice wich kernel/userland you want to run
> on the same physical hardware. An example is my cubietruck:
> I run it big-endian as:
> 
> [~] martin@space-truckin > uname -p -m
> evbarm earmv7hfeb
> 
> but most people use exactly the same hardware to run earmv7hf (little endian)
> instead.
> 
> On this board the difference is a simple "set endianess" instruction early
> in the kernel, but on some other machines it is a physical jumper on the 
> board.
> 
> This is not a very typical case though.

It's the first time I "hear" (actually, read) about it. It offers
great flexibility!

> The "compat" cases are more common: on arm machines you can execute
> binaries for eabi (the default) and oabi. On aarch64 machines you could
> run aarch64 (native 64bit), eabi (new 32bit abi) or oabi (old 32bit abi).

This way, MACHINE seems to represent the entirety of the hardware
devices which compose a specific system. It doesn't (it can't!)
change, of course.  Then, given this "static" hardware, you may make some
choices about endianness (as in the cubietruck example) and/or ABI (as
in this compat example) and/or some other features: this determines
the architecture, the instruction set you choose to build the kernel
for (the instruction set you choose to run on the actual CPU), and
this architecture is exactly MACHINE_ARCH.

> I guess now you see the complex spread sheet and understand why you have
> been confused initally - hope it got better during this thread ;-)

Oh, I think (I hope!) yes, it's becoming to be more clear, I really
couldn't figure it out before.

Thank you so much :)

Rocky

Re: Some questions about build.sh, machine, -u and tools

[Rocky Hotas]

On mag 05 15:34, Valery Ushakov wrote:

> Yes.  It's just formatted using mdoc, but is not installed as part of
> system manual pages.

Ok! This is what produced that header.

> I think Martin explained it pretty well in another followup.

Yes, it was very useful :)

> If by the "directory tools" you mean the one in the build directory
> (it took me a while to realize that), then yes.  It's called an objdir
> and is a standard make(1) feature.  Please check make(1).

Sorry, it was not clear. Assume that you run, on a NetBSD 9.99.81 amd64
host system,

./build.sh -U -O ~/obj -j2 -m evbarm -a aarch64 tools

(as in par. 32.1 of The NetBSD Guide)

then you will have, in ~/obj, two subdirectories:

tooldir.NetBSD-9.99.81-amd64
tools

By "directory tools" I was meaning the last one listed above,
`~/obj/tools'. I hope that ~/obj is in this case what you mean by
"build directory".

As regards ~/obj/tools and running make(1) in this directory,

 .OBJDIR A path to the directory where the targets are built.

Now I understand why you wrote ``tools is the objdir for the
src/tools, a place where tools are built'' in the previous message.

> Yes, we build our own make, sed, etc as part of the tools build and
> install them in $TOOLDIR/bin.  We add nb- prefix to the installed
> versions to avoid name clashes with the host programs.  E.g. GNU
> programs (can) do the same, using g- prefix (and in some cases that g-
> even fused into the name (gcc, gawk)).

Oh, amazing! And I guess `nb' stand for `(N)et(B)SD'.

Thank you so much,

Rocky

Re: Some questions about build.sh, machine, -u and tools

[Rocky Hotas]

For some reason, my messages are not delivered to this ML. I try to
merge them into a single reply both for Martin and Valery, from another
address. I apologize for any inconvenience and/or double messages you
may have received.


Part 1

On mag 05 13:21, Martin Husemann wrote:

> No, but you may have the choice wich kernel/userland you want to run
> on the same physical hardware. An example is my cubietruck:
> I run it big-endian as:
>
> [~] martin@space-truckin > uname -p -m
> evbarm earmv7hfeb
>
> but most people use exactly the same hardware to run earmv7hf (little endian)
> instead.
>
> On this board the difference is a simple "set endianess" instruction early
> in the kernel, but on some other machines it is a physical jumper on the 
> board.
>
> This is not a very typical case though.

It's the first time I "hear" (actually, read) about it. It offers
great flexibility!

> The "compat" cases are more common: on arm machines you can execute
> binaries for eabi (the default) and oabi. On aarch64 machines you could
> run aarch64 (native 64bit), eabi (new 32bit abi) or oabi (old 32bit abi).

This way, MACHINE seems to represent the entirety of the hardware
devices which compose a specific system. It doesn't (it can't!)
change, of course.  Then, given this "static" hardware, you may make some
choices about endianness (as in the cubietruck example) and/or ABI (as
in this compat example) and/or some other features: this determines
the architecture, the instruction set you choose to build the kernel
for (the instruction set you choose to run on the actual CPU), and
this architecture is exactly MACHINE_ARCH.

> I guess now you see the complex spread sheet and understand why you have
> been confused initally - hope it got better during this thread ;-)

Oh, I think (I hope!) yes, it's becoming to be more clear, I really
couldn't figure it out before.

Thank you so much :)


Part 2

On mag 05 15:34, Valery Ushakov wrote:

> Yes.  It's just formatted using mdoc, but is not installed as part of
> system manual pages.

Ok! This is what produced that header.

> I think Martin explained it pretty well in another followup.

Yes, it was very useful :)

> If by the "directory tools" you mean the one in the build directory
> (it took me a while to realize that), then yes.  It's called an objdir
> and is a standard make(1) feature.  Please check make(1).

Sorry, it was not clear. Assume that you run, on a NetBSD 9.99.81 amd64
host system,

./build.sh -U -O ~/obj -j2 -m evbarm -a aarch64 tools

(as in par. 32.1 of The NetBSD Guide)

then you will have, in ~/obj, two subdirectories:

tooldir.NetBSD-9.99.81-amd64
tools

By "directory tools" I was meaning the last one listed above,
`~/obj/tools'. I hope that ~/obj is in this case what you mean by
"build directory".

As regards ~/obj/tools and running make(1) in this directory,

 .OBJDIR A path to the directory where the targets are built.

Now I understand why you wrote ``tools is the objdir for the
src/tools, a place where tools are built'' in the previous message.

> Yes, we build our own make, sed, etc as part of the tools build and
> install them in $TOOLDIR/bin.  We add nb- prefix to the installed
> versions to avoid name clashes with the host programs.  E.g. GNU
> programs (can) do the same, using g- prefix (and in some cases that g-
> even fused into the name (gcc, gawk)).

Oh, amazing! And I guess `nb' stand for `(N)et(B)SD'.

Thank you so much,

Rocky

Re: Some questions about build.sh, machine, -u and tools

[Rocky Hotas]

On mag 05 11:13, Martin Husemann wrote:

> A MACHINE is a concrete thing, it belongs to a broader group of similar
> (but not identical) other machines of the same MACHINE_ARCH.

Ok! So, the reference and the "glue" is not MACHINE (as I thought), but
MACHINE_ARCH. This also explains what Greg was writing before.

It seems that the distinction between MACHINE and MACHINE_ARCH is
created for this purpose. As far as a userland program meets the same
MARCHINE_ARCH, even across different MACHINEs, it should work.

> How wide the range of MACHINE_ARCH goes, depends on details, e.g. there are
> lots of very different MACHINE (mac68k, atari, amiga, sun3, next68k, 
> mvme68k...)
> for MACHINE_ARCH = m68k. This is because we made userland compatible.
> Similar for MACHINE_ARCH = powerpc.
[...] 
> So in short: MACHINE_ARCH may be shared with other MACHINEs. Userland
> programs (ignoring device specific ioctl helpers) do not care about
> MACHINE but only require the MACHINE_ARCH.

So that they are compatible with different MACHINEs if they have the
same MACHINE_ARCH.

Thank you!

Rocky

Re: Some questions about build.sh, machine, -u and tools

[Rocky Hotas]

On mag 05 10:57, Rocky Hotas wrote:

> There are two different kinds of tools:
> 
> 1.
> The ones in obj_dir/tools, which are the result of the building of
> /usr/src/tools.
[...]
> 2.
> The ones in obj_dir/tooldir.NetBSD-9.99.81-amd64, which are the actual
> toolchain.

Apparently, this is completely wrong. See my reply to Valery for an
update:

 <http://mail-index.netbsd.org/netbsd-users/2021/05/05/msg026966.html>

Re: Some questions about build.sh, machine, -u and tools

[Rocky Hotas]

On mag 03 16:42, Valery Ushakov wrote:
> TL;DR: src/BUILDING explains most of these things.

Thanks! I'm trying to read it and it clarifed some doubts, but not
all.
It has the header of a Section 8 manpage, but it is not accessible as
such (`man BUILDING', `man 8 BUILDING' or with lower keys). Is this
normal?

> On Mon, May 03, 2021 at 12:18:54 +0200, Rocky Hotas wrote:

> Roughly speaking MACHINE determines with kernel, bootloader, etc the
> system uses.  You can have multiple MACHINE_ARCH for the same MACHINE
> when the machine can have different ABIs (e.g. old arm, earm) or run
> in different endianness (big, little) or bitness (32, 64).

Ok! Even if two MACHINEs with different ABIs or endiannes have really
few common features, maybe just the commercial brand and the
partitioning scheme.  Sorry, this is extreme of course, but the common
MACHINE idea still looks weird to me.

> tooldir* is default TOOLDIR, a place where the cross tools are
> installed, you can set it with -T option to build.sh.  tools is the
> objdir for the src/tools, a place where tools are built.  The choice
> to put tooldir inside objdir by default might be a bit confusing, IMO.
> Personally, I just use explicit -T.

Ok! My previous guess was wrong, then. So, IIUC: src/tools are the cross
tools. The directory tools is just an intermediate step in the building
of the cross tools. Once they are built, their executables are finally
placed in the TOOLDIR.

I was confused by the apparently different names that are present in
tools and in tooldir.NetBSD-9.99.81-amd64. Instead, most of the names
in tooldir.NetBSD-9.99.81-amd64 are just the names in tools, with `nb'
prefix. Also, running from tooldir.NetBSD-9.99.81-amd64 a trivial (but
effective, for a newbie)

diff nbsed ../../tools/sed/sed
diff nbcat ../../tools/cat/cat

shows that the two files match. Correct me if I'm wrong.

> Yes, -u is an "update" build.  build.sh is not magic, it's just a
> wrapper for plain old make.  Roughly speaking, -u tells it to not nuke
> the existing objdir and destdir and just run make and let it do the
> make thing, figuring out which things are out of date.
> 
> If you have a tree you've already built.  You can hack on, say,
> src/games/trek and then just run $TOOLDIR/bin/nbmake-vax to compile
> your changes, where nbmake-vax is just a script that invokes nbmake
> host tool (i.e. good old make, compiled for the host you are on) with
> a bunch of variables pointing to the right places to find sources, the
> toolchain, etc.

Thank you so much!

Rocky

Re: Some questions about build.sh, machine, -u and tools

[Rocky Hotas]

According to the paper

 

let's use the terms `build host', `host' and `target'.

On mag 03  8:30, Greg Troxel wrote:

> I keep forgetting this and having to page it in, but a big concept is
> that you can share userland among systems built for the same CPU type
> but that are different kinds of computers.  Basically MACHINE describes
> the computer and MACHINE_ARCH the CPU.  It may be helpful to read
> build.sh.

I thought it was the opposite: a userland shared by several MACHINEs of
the same type but different CPUs...

MACHINE is defined in the paper as a platform (par. 5.1).  Intuitively,
yes, MACHINE seems related to a whole computer (so, a collection
of hardware devices with its standards and conventions) and
MACHINE_ARCH is somewhat more specific, being related to a specific
CPU, but MACHINE is still a somewhat blurry element to me.

I scrolled the whole build.sh searching for the string MACHINE, but
(at least at a first glance) I didn't find something new.

> The tooldir is a prefix that has bin/lib and so on, which has programs
> that run on the build host, and are cross for targets.
> 
> There is also an obj dir for building the cross tools, usually the tools
> subdir within the objdir.

This is the exact fact which confuses me. I tried to understand this as
follows (but I'm not sure it's right and the par. 5.5 in the paper did
not shed a light).

There are two different kinds of tools:

1.
The ones in obj_dir/tools, which are the result of the building of
/usr/src/tools.
These are the tools used by the `build host' to build the second type of
tools, the toolchain.

2.
The ones in obj_dir/tooldir.NetBSD-9.99.81-amd64, which are the actual
toolchain.
They are used by the `host' to build the sourcecode (kernel and/or
userland) for the `target' system.

> -u basically means "just run make, and don't clean first",
> operationally.  This only works if the makefile rules are correct and
> nothing has changed outside the scope of what make copes with.
> 
> If you have changed a few lines of code, -u almost always works.
> 
> If you did cvs update along netbsd-9, it is highly likely that -u will
> work.
> 
> If you updated along current, it often works.   The shorter the time
> period, the more likely.

Ok! This is straght and simple.

Thank you!

Rocky

Some questions about build.sh, machine, -u and tools

[Rocky Hotas]

Hello!
Chapter 32 of The NetBSD Guide deals with Crosscompiling NetBSD with
build.sh.

1.
With `list-arch' the full list of the available machines and CPU
types is shown, named respectively MACHINE and MACHINE_ARCH. Correct
me if I'm wrong, MACHINE_ARCH represents the CPU with its specific
instructions set. In other words, this one:

 

I struggle instead about MACHINE. Some of them (sparc, vax) have
just one MACHINE_ARCH with the same name. Some of them have just
one MACHINE_ARCH with a different name (amd64). This allows to
specify just one between `-m' or `-a', when building with build.sh.
Then, there are the cases where a single MACHINE has several
MACHINE_ARCH. So, what does MACHINE exactly refer to?

2.
(For `host', `build host', etc. refer to the paper

 )
After fetching the sources, and after building the tools for the
first time with `-O ~/obj', two directories are created in `~/obj':

tooldir.NetBSD-9.99.81-amd64
tools

`tooldir.NetBSD-9.99.81-amd64' actually contains the cross-compiler
with its tools that have just been built: they are the actual `tools'
created by build.sh and they are supposed to run on the host platform.

What exactly is, instead, the contents of the directory `tools'?
Are they supposed to run on the build host platform to build the
cross-compiler and its tools?

3.
In Chapter 32, option `-u' for build.sh is used both when building
`tools' and when building the kernel itself.
Assume to run build 1 as:

./build.sh -U -j2 -O ~/obj -m evbarm -a aarch64 tools kernel=MYKERNEL

Then, some modifications to the cross-compiler tools and/or to the
kernel sourcefile(s) are performed. In the end, assume to run build
2 as:

./build.sh -U -u -j2 -O ~/obj -m evbarm -a aarch64 tools kernel=MYKERNEL

IIUC, `-u' means that if anything between tools and kernel has already
been built and its code has not been modified after build 1, now it
won't be built again. Is this correct?

Bye!

Rocky

Re: skills to document the netbsd kernel?

[Rocky Hotas]

> Sent: Tuesday, April 13, 2021 at 9:31 AM
> From: "Mayuresh Kathe" 
> To: netbsd-users@netbsd.org
> Subject: skills to document the netbsd kernel?

Hello!

> while i possess decent technical skills, i lack knowledge of
> ansi-c which i believe would be a must to read the source of
> the netbsd kernel and most of the userland.

Yes, knowing the C language is a necessary condition. Which part of C
language? Anything, but above all pointers, double pointers, function
pointers, structs, preprocessor macros (like `#define').

> my question is; what topics should i acquire command over to
> understand the netbsd kernel source with the intent of
> documenting it? i guess i should be well versed in operating
> systems theory and practise, but what else?
> thank you.

I have a very similar intention and my questions and doubts are exactly
the same as yours. According to the (very few) hints I collected so far,
first try to understand the kernel source tree: how it is internally
organized, where drivers are positioned, where machine-dependent code is
positioned, where other fundamental OS components are placed, and so on. I
think you may (and should!) surely have an overall idea of the kernel,
but you can not know everything about it with the same deep detail. I
think it's humanly impossible.  Therefore, I would suggest you to focus
on some area you are interested in (device drivers? Schedulers? Memory
management? TCP/IP stack?) and explore the relevant sourcefiles. Also try
the documentation (section 9 of the manpages), because while it may be
not enough to understand a topic, it is anyway very helpful.  Of course,
download the sourcetree of the kernel and try to build it, following
the instructions in Chapter 30 and Chapter 31 of The NetBSD Guide:

 
 

Also, try to make some elementary modifications to your kernel (for
example, to the devices included in the GENERIC file, or to the messages
generated during the autoconfiguration) just to make some practice with
its files. Try to google something you do not know (an absolutely random
example search: `netbsd scheduler'); you could find some old message or
some guide (there are few) about what you are dealing with.

Last but not least, if you are working (or would like to work) on
something specific, and you have doubts or questions, let's talk about
it! Through the Mailing List or the Freenode IRC channel #netbsd.

Bye!

Rocky

Re: libusb not working properly on 9.0/9.1

[Rocky Hotas]

On gen 20 18:44, Greg Troxel wrote:
> 
> I think libusb on NetBSD does not do async IO.  Could that be it?

Unfortunately I am not able to determine if this is relevant.

Re: libusb not working properly on 9.0/9.1

[Rocky Hotas]

On gen 20 13:25, Craig Kulesa wrote:
> 
> Hi Rocky,

Hi Craig :)!
I share the quotes from you message here, hoping that this does not bother
you: I think it could be very useful for anyone interested.

> I'm using libusb in NetBSD 9 and am not having trouble with it.  But as
> Manuel points out, I think only ugen devices are accessible.  This limits
> how useful libusb can be.

OK! This was also the guess from Manuel.

> From the posting you made for uhubctl, I think what is happening is that
> your hub is already attaching as uhub0, not ugen0, and so libusb cannot see
> it.
> 
> This is a common phenomenon -- a driver like uhid or uhub attaches and then
> ugen cannot.

I was not sure about how ugen is used from the system (and never actually
dealt with them before), now it's more clear.

> FreeBSD gets around this, I believe, by attaching both ugen and uhid. That
> way, you can use libusb on any device without having to do anything.

It's a workaround, but it's effective :).

> An alternative would be to allow detaching as uhub or uhid, and reattaching
> as ugen.
> 
> Hope that helps, though it doesn't solve your problem just yet...

Thanks a lot for your suggestion. If it worked, there's no problem for me as
regards following this procedure "by hand". The real issue is maybe with
the connected devices: a USB key, a mobile phone, a mouse... Would I be able
to use them as before?

AFAIK from drvctl(8), `drvctl -d uhub0' should detach uhub. How can then
I reattach it as ugen?

Bye!

Rocky

libusb not working properly on 9.0/9.1

[Rocky Hotas]

Hello!
I would like to run this executable

 

on NetBSD, but even on a supported hardware platform (Raspberry Pi 3
B+) it is unusable, due to libusb not properly working. I was using
libusb1-1.0.24 from pkgsrc 2020Q4.

As suggested here:

 

(bypassing pkgsrc) I cloned the repository of libusb and built from it.
After successfully completing the operations, the basic tools
examples/listdevs and example/testlibusb do not work.

This happens both with:

- NetBSD 9.1 on Raspberry Pi 3 B+ (evbarm);
- NetBSD 9.0 on a laptop (amd64).

What could it be the reason?

Rocky

Error in dmesg(8) for USB hub in Raspberry Pi 3 B+

[Rocky Hotas]

Hello!
In the dmesg, after the configuration of the on-board USB hubs, this
line is printed in NetBSD 9.1:

uhub0: autoconfiguration error: illegal enable change, port 1

The relevant parts of the dmesg are:

[ 1.00] NetBSD 9.1 (GENERIC64) #0: Sun Oct 18 19:24:30 UTC 2020
[ 1.00] 
mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/evbarm/compile/GENERIC64
[ 1.00] total memory = 933 MB
[ 1.00] avail memory = 900 MB
...
[ 1.00] simplebus0 at armfdt0: Raspberry Pi 3 Model B Plus Rev 1.3
...
[ 1.00] cpu0 at cpus0: Cortex-A53 r0p4 (Cortex V8-A core)
...
[ 2.081312] dwctwo0: Core Release: 2.80a (snpsid=4f54280a)
[ 2.081312] usb0 at dwctwo0: USB revision 2.0
...
[ 2.101183] uhub0 at usb0: NetBSD () DWC2 root hub (), class
9/0, rev 2.00/1.00, addr 1
[ 2.82] uhub0: 1 port with 1 removable, self powered
...
[ 3.963921] uhub1 at uhub0 port 1: vendor 0424 (0x424) product 2514
(0x2514), class 9/0, rev 2.00/b.b3, addr 2
[ 3.973922] uhub1: multiple transaction translators
[ 3.983924] uhub1: 4 ports with 3 removable, self powered
[ 5.284044] uhub2 at uhub1 port 1: vendor 0424 (0x424) product 2514
(0x2514), class 9/0, rev 2.00/b.b3, addr 3
[ 5.294046] uhub2: multiple transaction translators
[ 5.304048] uhub2: 3 ports with 2 removable, self powered
[ 5.634091] uhub0: autoconfiguration error: illegal enable change,
port 1
[ 5.644092] WARNING: 2 errors while detecting hardware; check system
log.
...
[ 7.054290] mue0 at uhub2 port 1
[ 7.054290] mue0: SMSC (0x424) LAN7800 USB 3.1 gigabit ethernet
device (0x7800), rev 2.10/3.00, addr 4
[ 7.354341] mue0: LAN7800 id 0x7800 rev 0x2

I found this same `autoconfiguration error' in an older message in this
ML regarding the same board:

http://mail-index.netbsd.org/port-arm/2019/02/26/msg005527.html

It is generated here:

https://nxr.netbsd.org/xref/src/sys/dev/usb/uhub.c#564

but I am not able to understand what causes it. 

What does it mean, and what it is related to? I tried to connect a mouse
and a keyboard, and they seem to work fine.

Bye!

Rocky

Re: GPIO programs for Raspberry Pi in Python and C

[Rocky Hotas]

On gen 14 22:30, Rocky Hotas wrote:
> 
> I never used ioctl(2) and maybe it's not immediate to integrate it with
> gpio(4). If you have some piece of code performing even a trivial
> operation like turning ON a pin, and you are OK with sharing it, it
> would be welcome.

It's easier than I thought, but I would like to share the procedure if
anyone else is wondering how to do it.

First, GPIO must be configured:

1) add a line `gpio=YES' in /etc/rc.conf;
2) specify in /etc/gpio.conf the pins you want to use. For example,

gpio0 21 set out just_a_test

See gpio.conf(5) for more information, and the following useful
page:

 <https://github.com/catskillmarina/netbsd-gpio-doc>

Then, add to your non-privileged user `_gpio' as a secondary group:

# usermod -G _gpio exampleuser

In fact, the device representing GPIO pins is:

$ ls -l /dev/gpio0
crw-rw-r--  1 root  _gpio  173, 0 Oct 19 04:20 /dev/gpio0

This will allow also `exampleuser' to read and write the configuration
of GPIO pins. Root can always do this, also with gpioctl(8).

See gpio(4) for the available macros and for the data structures to be
used. Also, obviously, see ioctl(2). A simple example:


#include 
#include 
#include 
#include 
#include 

struct gpio_info mytest;
struct gpio_req mystatus;
int myfd;

int main() {
if ( (myfd = open("/dev/gpio0", O_RDWR)) == -1 ) {
perror("Can't open GPIO device file\n");
return 1;
}

if (ioctl(myfd, GPIOINFO, &mytest) == -1) {
perror("ioctl GPIOINFO failed");
}

printf("Number of currently active GPIO pins: %d\n", mytest.gpio_npins);

mystatus.gp_pin = 21;

if (ioctl(myfd, GPIOREAD, &mystatus) == -1) {
perror("ioctl GPIOREAD failed");
}

printf("Requested pin: %d\n", mystatus.gp_pin);
printf("Name of requested pin: %s\n", mystatus.gp_name);
printf("Value of requested pin: %d\n", mystatus.gp_value);

mystatus.gp_value = GPIO_PIN_LOW;

if (ioctl(myfd, GPIOWRITE, &mystatus) == -1) {
perror("ioctl GPIOWRITE failed");
}

printf("Value of requested pin was: %d. It is now 0.\n", 
mystatus.gp_value);
}


The last statement can be verified, for example, from the root shell:

# gpioctl gpio0 21

Thanks again for your help!

Rocky

Re: GPIO programs for Raspberry Pi in Python and C

[Rocky Hotas]

Thanks to all for your very useful hints and suggestions.

On gen 14 12:12, Greg Troxel wrote:
> 
> Benny Siegert  writes:
> 
> > NetBSD uses the gpio(4) device to talk to the GPIO pins:
> > http://man.netbsd.org/gpio.4
> >
> It is really straightforward in C.  I wrote a program to watch a pin and
> do things, and all the complexity was in writing rate limiting.

I never used ioctl(2) and maybe it's not immediate to integrate it with
gpio(4). If you have some piece of code performing even a trivial
operation like turning ON a pin, and you are OK with sharing it, it
would be welcome.

> As for python, if there is a python library for gpio pins and it has
> some semblance of portability, it should be  relatively easy to add
> #ifdef NetBSD and do the read/write the NetBSD way.   If it's written as
> if there is only one operating system in the world, it's probably a bit
> harder.

The library esists, but I have no clues about its portability. However,
there's also a LUA interface for GPIO pins, shown in gpio(3lua) (which
basically uses ioctl(2) and gpio(4)).

C is not a limitation for me, if Python is not available ATM.

Rocky

GPIO programs for Raspberry Pi in Python and C

[Rocky Hotas]

Hello!
I successfully installed NetBSD 9.1 evbarm (arm64.img) on a Raspberry Pi
3 B+. In the Raspberry documentation, several example regarding the
handling of GPIO pins are available as Python programs:

https://www.raspberrypi.org/documentation/usage/gpio/python/README.md

They use a module named `gpiozero'. However, in pkgsrc I did not find
such a module for NetBSD.
Is there something equivalent? Or is there some plan to make `gpiozero'
available in future?

As an alternative, are there some C libraries available for NetBSD, to
manage the GPIO pins?

In pkgsrc current, excluding the games and a video player, I only found:

misc/raspberrypi-userland-20170109nb3

but I am not sure what it is about, and it is not present in 2020Q4 (the
repository actually used by my Raspberry).

Rocky

Re: Block a single connection with npf

[Rocky Hotas]

On dic 12 12:48, Greg Troxel wrote:

> First, you are talking about "gateway" so I am guessing you have the
> usual computer with two interfaces, one to your ISP with a single IPv4
> address and one that is your home LAN, where is is .1, does dhcp, and is
> running npf.  If not please explain.

You perfectly guessed and depicted the scenario. (I omitted this because
I took this typical configuration as granted, but maybe it was necessary
to briefly mention it).

> Normally, people set up nat from LAN to WAN

Yes, exactly, and that's my case, too.

> and a firewall for incoming
> WAN packets that blocks most things except for what they want, and
> usually a stateful pass out rule so that matching packets from outgoing
> things are allowed.

I jumped straight to the second step, not having (so far) specific needs
to block something from the outside (there are no services exposed to my
public IP address):

pass stateful out all

> I am not an npf expert yet -- still on the steep part of the learning
> curve

Me neither. I'm even behind you, probably, on the very beginning of that
steep part :).

> -- but I think stateful rules only on syn packets and then apply
> to others.

Sorry, I can't understand what you meant here.

> This is a normal desire, to stop "phoning home" and "exfiltration" by
> your adversary-controlled proprietary-software-infested IOT things :-)

Yes exactly! :)

> Steppping, back, the real goal is to stop the packet from going to the
> ISP and onwards to the destination.So you can block the packet
> inbound to if_mylan or outbound on if_mywan.  But outbound on wan, you
> have to be careful about if NAT is applied when the rule is evaluated so
> you can match.   Or you can just block target_address entirely.

I think that blocking the packet inbound to $if_mylan is the best
solution. This action should be as narrow as possible, blocking only
this connection (blocking entirely  seems to be overkill).

> I would add
> 
>   block in family inet4 proto tcp from  to  
> port 443
> 
> instead.  Basically you want to drop all packets on that port from
>  to  and you don't care if the connection
> is open or not.
> Generally I find I want to use stateful rules only to allow a reverse flow,
> and this isn't one of those times.

Ok, so basically (comparing it with my solution) you would not use
`stateful'. This makes sense.

> Another strategy, that works with ipfilter, is to have (pseudocode for 
> example)
> 
>   block out log on wan from 10.0.0.IOT1 to any
>   block out log on wan from 10.0.0.IOT1 to 
>   pass out on wan from 10.0.0.IOT1 to [exactly what I want to allow]
> 
> With ipf, the rule is evaluated before nat.

I think that it would be before NAT even with npf.

> However, for block-all pass-some, you need to do it outbound, because
> you might want these devices to be able to do DNS or NTP to your router
> box, even if you don't want them to communicate externally in general.

This would be a very careful strategy. However, in this specific case I
don't want to block anything, so I prefer the "block-some" approach. 

Another issue I'm struggling with is to block not a single IP, but a
custom bunch of IPs, so not a ip/subnet couple, but a range:

block in family inet4 proto tcp from  to 
-

This gives a syntax error in npf. Also, I would like to use the
namespace, for example:

block in family inet4 proto tcp from  to *.netbsd.org

But this gives error as well. I still don't know if this is just a
syntax error, or if npf does not support these two features at all.

If you dealt with this, let me know.
Thanks for your thorough message.

Rocky

Re: Block a single connection with npf

[Rocky Hotas]

On dic 12 17:13, Rocky Hotas wrote:
>
> group "internal" on $if_mylan {
> pass in all
> pass out all
> 
> block stateful out family inet4 proto tcp from  to 
>  port 443
> }

[... ]
> Is the order of the rules wrong? Or something else?

Ok, maybe it is more trivial than it seemed. Replacing the direction
`out' with `in',

block stateful in family inet4 proto tcp from  to 
 port 443

this seems to have the desired effect.
Thank you and sorry for the noise.

Rocky

Block a single connection with npf

[Rocky Hotas]

Hello!
In npf.conf(5), the group defining the rules for the LAN interface
$if_mylan of my gateway is as follows:

group "internal" on $if_mylan {
pass in all
pass out all

block stateful out family inet4 proto tcp from  to 
 port 443
}

I would like $if_mylan to be essentially a transparent interface, for
any traffic from and to the internet, except in one case.

A host inside the LAN, with IP , often sends data to
, port 443, through a TCP connection. I would like to
block this stream, without affecting any other connection between the LAN
and the internet. I would like that, when the data arrives from
 to $if_mylan, it is blocked.

I tried as above, adding a `block' line. But this seems not to have any
effect (at least, observing the stream for several seconds on iftop,
after having updated /etc/npf.conf and launched `npfctl reload').

Is the order of the rules wrong? Or something else?

Bye!

Rocky

Re: npf syntax: port ranges, negation of a condition, and map rules

[Rocky Hotas]

On ott 29 10:16, Michael van Elst wrote:

> The "port forwarding case" only handles single ports. You need
> separate map commands for each port.

After several errors, I was wondering in fact if this was the reason.
Thanks for clarifying this!

Rocky

Re: npf syntax: port ranges, negation of a condition, and map rules

[Rocky Hotas]

On ott 28 12:22, Martin Husemann wrote:

> There is an example in /usr/share/examples/npf/host-npf.conf:
> 
> # Allow being tracerouted
> pass in proto udp to any port 33434-33600

This is a single use (but thanks! I forgot this example file). My case
is double, because I would like to map a port range into another port
range (both ranges contain the same number n of ports).

> > but none of them seems to work in
> > map $myif dynamic proto tcp $myhost port XXX <- $myif port XXX

In particular, for example,

map $myif dynamic proto tcp $myhost port 33434-33435 <- $myif port 55000-55001

doesn't work:

# npfctl reload
/etc/npf.conf:41:92: port range is not valid near '55001'

Replacing 55000-55001 with a single port or with 33434-33435 themselves
gives the same error.

Rocky

npf syntax: port ranges, negation of a condition, and map rules

[Rocky Hotas]

Hello!
After browsing the available documentation for npf, I did not find any
hints about some issues.

1) How to represent port ranges? I tried with

beginning_port:end_port (which for example is used in OpenBSD's pf)
beginning_port,end_port
beginning_port-end_port

$port_range = { beginning_port, end_port }

but none of them seems to work in

map $myif dynamic proto tcp $myhost port XXX <- $myif port XXX

However, `http://www.netbsd.org/~rmind/pub/npf_manual_netbsd_6.pdf'
mentions in par. 3.4 that a ``TCP/UDP port or range' can be speficied in
a rule.

2) I would like to block all the IPs from a subnet, except the ones
listed in a variable $host_group. Is this possible?

block in final proto tcp from NOT $host_group to $destination port $some_port

3) In the `map' section of npf.conf(5) it is stated that

``Explicit filter criteria can be specified using pass criteria ... as an
  additional option of the mapping''.

So, assume I have: 

map $myif dynamic proto tcp $myhost port $port1 <- $myif port $port2
pass stateful in family inet4 proto tcp to $myif port $port2

How to merge them?

Checking the source code, trying to get some more information about
this, is not feasible for me.

Thank you anyway!

Rocky

Re: altq filter acts before or after npf NAPT?

[Rocky Hotas]

On ott 23 15:47, BERTRAND Joël wrote:
> and altqd runs as expected. Only a few bugs. Don't try to launch
> /etc/rc.d/altqd reload, altqd will take 100% of a CPU. Don't try to stop
> altqd, you will obtain a kernel panic.

Some updates: if altqd.conf has only a queing discipline per each
interface, I can (with 9.0 release) do `service altqd stop', `service
altqd start' with no issues.

If altqd.conf contains only a traffic conditioner for a single interface,
it is impossibile to stop altqd. `service altqd stop' does have no
effect: shell hangs in `sleep' mode; a Ctrl+C will restore the shell, but
altqd will still be running (however, `altqstat' does not show anything,
so maybe altqd is not fully functional).
Most important thing: a `shutdown' will not be able to stop `altqd'. It
blocks here:

System shutdown time has arrived

About to run shutdown hooks...
Stopping cron.
Stopping inetd.
Stopping altqd.

Another important introduction to altq, if anyone is interested, is:

 

``Managing Traffic with ALTQ'', from Kenjiro Cho, published in 1999

A couple of observations:
- ``An interface can have one queueing discipline attached at a time''
  (par. 3.1.2), so it's not possible to define multiple queueing
  disciplines for the same network interface.
- Maybe the issues regarding the kernel timer resolution, assumed to be
``10msec in most UNIX system'' (par. 3.2.1), are outdated.

Bye!

Rocky

Re: Use of disklabel, MBR and GPT

[Rocky Hotas]

On ott 23 16:12, Bruce Lilly wrote:
> On Fri, Oct 23, 2020 at 3:34 PM Martin Husemann  wrote:
> 
> > There is no point in ever using a disklabel on a disk that has a GPT.

This is so important to know. The first time I used GPT I tried in any
way to install a disklabel, too. Then, it became clear it is not
possible.

> The various bits of relevant documentation should state so,
> clearly and unambiguously.

I quote this.

On one hand, it's not simple to re-write the documentation and maybe
this is the reason why it has not been updated so far (also given the
fact that GPT integration is relatively recent); on the other hand,
this complicates a lot the work of a new user (who doesn't know NetBSD),
which I think should be guided as much as possible.

> Instead (for example, not intended to be a complete list of issues), the
> current Guide (http://netbsd.org/docs/guide/en/chap-exinst.html#exinst-intro)
> has sections for MBR (3.5) and disklabel (3.6), but no mention (at all!)
> of GPT (and refers to "BIOS" but has no mention of EFI or UEFI).
> 3.6 misleadingly states:
> "NetBSD uses its own partition scheme, called a disklabel, which is
> stored at the start of the MBR partition: "
> strongly implying that a physical disklabel is necessary for NetBSD
> in all cases, presumably including installation to GPT disks.

Yes, and this is the reason why I tried to install disklabel on a GPT
partition! Also, the reason why I created this thread.

Rocky

Re: Use of disklabel, MBR and GPT

[Rocky Hotas]

On ott 22 11:48, Bruce Lilly wrote:
> A few additional points:  
> 
> "disklabel" is a term used by all 3 of the major BSD variants
   
> (FreeBSd, NetBSd, and OpenBSD), but  
> with significant differences.  
  
A huge "thank you" for the massive amount of information and
observations, for exploring several different cases with pros/cons.

Also, this clarifies several incompatibilities between the *BSD
systems.   
  
> `gpt` is the most usable and stable part of the NetBSD installer;
> other bits seem to fail  

It's a very good tool.  

> (for one thing, this particular disk spins at 5400 RPM, not 3600;
> NetBSD disklabel  
> also claims 3600 "rpm" for SSDs and flash drives).

Yes, I noticed in my tests that this false value is recurrent.
Thanks also for the comparison of the whitespace tolerance in fstab.

> So GPT works reasonably well across various OSes, to the extent that
> each OS supports (and documents support for) it; MBR for multiboot of
> more than a very few OSes tends to be a problem, and BSD-specific
> disklabel is largely an anachronism (and is incompatible between BSD
> variants).
  
This is a very useful recap. As regards the integration of gpt in
NetBSD, I can confirm: it is very well handled by this OS. And your
examples show that gpt is well supported in many other OSs as well.

Yes, maybe disklabel is strongly outdated, but I wanted to share some
notes about it, because I think it was largely used in the past, along
with MBR, with all the issues (and the confusion) we highlighted in
this thread.  
  
Thank you!  

Rocky

Re: altq filter acts before or after npf NAPT?

[Rocky Hotas]

On ott 23 15:47, BERTRAND Joël wrote:

>   If I remember, altq is applied after npf.
> My /etc/altqd.conf contains :
> 
> legendre:[~] > cat /etc/altq.conf
> #tbrconfig
> interface wm0 bandwidth 8M priq
> 
> class priq wm0 high_class NULL priority 1
> class priq wm0 low_class NULL priority 0 default

[...]

A huge thank you for this example. It's a big help.

> and altqd runs as expected. Only a few bugs. Don't try to launch
> /etc/rc.d/altqd reload, altqd will take 100% of a CPU.

Ok! I never tried :).

> Don't try to stop
> altqd, you will obtain a kernel panic.

I stopped it several times and then started it again. It all seems OK
(with NetBSD 9.0 release amd64, obviously rebuilt with the ALTQ kernel
options uncommented).

>   In my example :
> - wm0 is WAN interface ;
> - agr0 is LAN interface ;
> - tap0 and tap1 are two interfaces of a bridge used for SIP and video
> calls (jitsi).

Thank you so much!

Rocky

Re: altq filter acts before or after npf NAPT?

[Rocky Hotas]

On ott 23  8:19, Greg Troxel wrote:
> You should  be aware that not that many people use altq.

Yes, I was guessing this.

> (I used to use
> it heavily, even adding new queueing disciplines, but that was a very
> long time ago.)

The only important thing is if it's still a good tool for traffic
shaping. If it is old, there's no problem (as far as it's still good).

> So my advice is:
> 
>   don't assume that others know the answers
> 
>   read the source code
> 
> I don't mean to discourage you from asking - that's 100% fine.  It's
> just that I expect that if you want to do what you are describing (which
> seems entirely reasonable), I think you will end up digging in to the
> code yourself.

It is a very honest advice/encouragement.

To avoid going off-topic, I'll be concise: you are right, in particular
for a couple of questions I asked here lately. Sometimes I don't feel
fully independent; in this case, there are two layered difficulties:
comprehension of network protocols/procedures, and comprehension of
source code. Each of them alone is beyond my capabilities.

However, I get the suggestion and will try to do my best.

Rocky

Re: 9.0/9.1 packages symlink for amd64/aarch64/i386 updated to 2020Q3

[Rocky Hotas]

On ott 22 22:54, nia wrote:
> hi,
> 
> if you run pkgin upgrade and were using the standard set of 9.0 packages,
> you will now be on 2020Q3
> 
> thanks, and sorry for the delay

Thanks to you for this info. It's great! I'm upgrading.

Rocky

altq filter acts before or after npf NAPT?

[Rocky Hotas]

Hello!
Consider a NetBSD 9.0 (release) system using both npf and altqd. This
machine has two NICs and it is a gateway. First NIC is connected to a
LAN, with multiple hosts, and second NIC is connected to the modem.
With npf, the following is applied:

map $if_second_nic dynamic $internal_lan -> $second_nic_ipv4

This is a NAPT, so when a packet from a LAN host is outgoing from the
second NIC, its source address is mapped into the second NIC's IP.

One of the few (or maybe the only) examples about altqd configuration
is here:

 

It creates this filter:

filter bge0 high_class 0 0 0 0 17

where the third `0' means `any source IP address'. Assume that bge0 is
the second NIC. This is for packets outgoing from it.

Does altqd act:
1) before
or
2) after npf?

Because in case 1) the `source IP address' will be the one of the LAN
hosts who generated the packet, which is the real source IP address of
that packet.
Instead, in case 2), any packet outgoing from bge0 will always have
bge0's IP address, thus making impossible any source IP distinction
between the packets in this filter.
Does anyone know which is the case?

Rocky

Re: npf, map and pass stateful out all

[Rocky Hotas]

On ott 20  9:03, Greg Troxel wrote:

> I am not an pf expert, but...

Don't worry, me neither :).

> > As it is written here, and exactly as Martin said in a previous
> > thread, `pass stateful out all' ``allows the packet to go out, and
> > also creates a NAT state entry so any answers are allowed back in''.
> 
> This creates a state entry.   That is a record that packets with some
> protocol, srcaddr/port, dstaddr/port when out, and this creates implicit
> rules that will allow the reverse packets back in.  It does not
> configure NAT.

You got the point. Yes, and I can confirm this. ``Also creates a NAT''
confused me. Maybe Martin was only meaning that the connection is
tracked, but not that an actual NAT is created. NAT can only be created
through `map'.

I put `map $ext_if dynamic $localnet -> $ext_if' in npf.conf. Then, I
considered the `SSH_CLIENT' env variable for two ssh connections from
different hosts in a LAN connected to the gateway, to a host which is
outside $ext_if. In both cases, the env variable showed the IP address
of the gateway. This did not happen without the `map' line: in that
case, `SSH_CLIENT' had the LAN IP of the host where I started the ssh
connection (so, they were different in the two connections).

> Yes.  That does, and it probably doesn't create a state entry - just
> maps the outgoing packet.  Or if it does create state, that state
> probably won't allow an incoming packet (when there is a block all in
> rule).

I am not able to say which of the two options is the right one, but
(using the same ssh example as above), the host outside $ext_if, without
a `pass stateful out all' is not reachable.

ssh: connect to host  port 22: Network is unreachable

> > Given the `map' line which creates a NAT, is `pass stateful out all'
> > (which allows any packet creating a NAT) really necessary? A NAT has
> > already been created with `map'.
> 
> I believe it is.

Yes, I can confirm this.

> Absolutely.   A firewall that does not do NAT is a very sensible
> concept.  You are only configuring NAT becuase you don't have real
> addresses on your LAN (you and ~everybody else at home).

Exactly.
Thank you!

Rocky

npf, map and pass stateful out all

[Rocky Hotas]

Hello!
Between the documentation about npf, another very useful document is:

 

As it is written here, and exactly as Martin said in a previous
thread, `pass stateful out all' ``allows the packet to go out, and
also creates a NAT state entry so any answers are allowed back in''.

But also this line enables NAT:

map $ext_if dynamic $localnet -> $ext_if

So, assume that my npf.conf has:

map $ext_if dynamic $localnet -> $ext_if

group (name "external", interface $ext_if) {
pass stateful out all
}

Given the `map' line which creates a NAT, is `pass stateful out all'
(which allows any packet creating a NAT) really necessary? A NAT has
already been created with `map'.
Or, viceversa, with `pass stateful out all', is `map' really
necessary?

Rocky

Re: Configure NetBSD as a gateway for LAN hosts

[Rocky Hotas]

On ott 13 12:08, Martin Husemann wrote:
> With above routing table this should already happen - no concrete local subnet
> matching, so it will pick "default".

Yes, I tried and it does!

> > With a routing table
> > entry, or with a rule (the `pass stateful out all' in soho_gw-npf.conf)
> > in npf?
> 
> That rule does not change routing, it just allows the packet to go out,
> and also creates a NAT state entry so any answers are allowed back in.

I checked npf.conf(5) and also

 

but I wasn't able to determine this. Thank you, it is exactly as you
said: I tried with ssh, ping and also a random client/server
communication in a random port with nc(1).

> In general it is best to get packet flow working first and then start caring
> about filtering, but with NAT this is tricky.

Why is this tricky with NAT? Because when a request exits from the
gateway, it exits from a port determined by the NAT, but when the answer gets
back to the gateway, it is hard to recognize it?

I still can't figure it out.

If you think there's a better way, let me know. Also, so far, I still
didn't try with the `map' keyword in npf.com (which I thought was the
only way to perform NAT).

Thanks a lot!

Rocky

Re: Use of disklabel, MBR and GPT

[Rocky Hotas]

On ott 16 12:47, Greg Troxel wrote:

> >  - if you need the disk to be accessible from other systems, you need to
> >use a partitioning scheme they understand. MBR works for DOS and older
> >windows, GPT works for nearly everything newer. Disklabel only works
> >for older Unix systems or NetBSD.
> 
> Very true, but also: If the disk needs to be readable by some  device
> which of course has a microcontroller but isn't a "real computer", it's
> very likely it can only deal with MBR.  Examples are Garmin GPS
> receivers, cameras, and things like that.

Amazing! I never worked with that, so could't guess it. A very good
point!

Rocky

Re: Use of disklabel, MBR and GPT

[Rocky Hotas]

On ott 16 12:52, Greg Troxel wrote:
> > Obviously, disklabel creates custom partitions only in the portion of
> > the disk which is related to NetBSD, the only one where it is intended

[...]

> This isn't really true.  While the installer might only create
> partitions within the NetBSD MBR partition, it is entirely feasible and
> reasonable to create a disklable partition that refers to some other MBR
> partition.

Yes, this is the same as observed by Robert Elz. As he stated, I should
have written "disklabel should create", "is intended to create", as in
fact I write below, speaking about MBR for the same problem.

> This is the default on RPI, where you have (skipping uninteresting lines)

[...]

Oh, it's a really concrete example about how diskalbel can act on disk
areas other than the NetBSD MBR primary partition. Yes, maybe it should
be discouraged to avoid confusion, but it has been a necessary
clarification.
Thanks!

Rocky

Re: Use of disklabel, MBR and GPT

[Rocky Hotas]

On ott 16 19:44, Robert Elz wrote:
>   | Obviously, disklabel creates custom partitions only in the portion of
>   | the disk which is related to NetBSD,
> 
> That's not correct, though if written as a "should create" it would
> perhaps be good advice for most people.
> 
> In NetBSD, the MBR is used to find the NetBSD partition, and then if
> that contains a valid disklabel, that label is used (solely) thereafter.
> 
> The label can reference anywhere on the disk, accessing Windows or

[...]

> If you're determined to make a mess of the drive, you can create NetBSD
> partitions that fit in the middle of a different OS partition - destroying
> any data

[...]

Thank you for pointing all this out.

> Note that NetBSD is different to FreeBSD here, in FreeBSD each FreeBSD
> MBR partition has a disklabel of its own which accesses just the MBR
> partition allocated to it.

This was my guess, in fact (``I don't know if FreeBSD also manages in a
different way the coexistence of MBR and disklabel'', without mentioning
the details).

Thanks for all your clarifications!

Rocky

Re: Use of disklabel, MBR and GPT

[Rocky Hotas]

On ott 16 12:03, Rocky Hotas wrote:
> When the machine is booted with NetBSD, disklabel is the only way
NetBSD  
> looks at the disk, the *entire* disk; MBR in this case is not
considered  
> at all.  
  
Obviously, disklabel creates custom partitions only in the portion of
the disk which is related to NetBSD, the only one where it is intended
to act. It leaves unchanged any other area of the disk. If MBR did not
exist, the extension of disklabel partitions `a', `b', `e', `f', etc.
could span the whole disk, and in fact this is the case of my last
example, where disklabel only is used, without MBR.
 
In a similar way, when MBR is used from other OSs, it is intended to
create and modify all the primary partitions, except the one where
NetBSD is installed (unless a full disk wipe must be performed): in
other words, it is indended to modify the disk, except the portion
used by NetBSD. This way, NetBSD can coexist with a PC/IBM,
DOS-compatible system.

Re: Use of disklabel, MBR and GPT

[Rocky Hotas]

On ott 16 12:25, Martin Husemann wrote:
> I have a few clarifications...

Thanks for writing them.

> > Maybe this is trivial, but during the installation of NetBSD a
> > disklabel is actually nested inside a primary MBR partition.
> 
> This is true for some architectures (like amd64), but others do use plain
> disklabel on a disk without any MBR. This also works on amd64 when you
> do not need to boot from the disk.

Yes, exactly. I tried to specify this (``Note that the above behaviour
is about amd64; for other ports it may change''), maybe I could have
better positioned the sentence.

> > - to grant the compatibility with PC/DOS (and other OSs like Windows
> >   this way can still use the same disk);
> > - to grant a regular boot on PC/IBM hardware (where, without a valid
> >   MBR, the BIOS could not even try to search a bootloader on the disk).
> 
> Yes and no - newer sysinst versions allow to kill the whole disk content
> (including the MBR and whatver partitions exists), and start from scratch
> (as if the disk had been totaly wiped before). Use with care - this kills
> all data on the disk.

Oh, ok, maybe I did not dig enough in newer installers.

> The (single) disklabel can then describe partitions outside of the NetBSD
> partition too (as seen when setting up (U)EFI booting on disks with MBR
> partitioning -  the EFI boot partition shows up in the disklabel).

Thanks for pointing out this.

> That is kind of correct once the kernel is running.

Yes, I was meaning this.

> This is not true - you can select GPT for boot disks with both firmwares.

Ops, again my fault about newer installers.

> You did not stress the (from my POV) most important points of all this mess:
> 
>  - if you need the disk to be accessible from other systems, you need to
>use a partitioning scheme they understand. MBR works for DOS and older
>windows, GPT works for nearly everything newer. Disklabel only works
>for older Unix systems or NetBSD.

I did not specify this about GPT, but as regards MBR DOS compatibility... the
whole message is about this topic. Maybe a recap was missing.

>  - if your disk is larger than 2TB you can not use MBR (you may have luck
>with some systems upto 4TB, but the "portability" is lost), nor can
>you use disklabel. Select GPT (sysinst knows about the size limits
>and will avoid non-working selections).
> 
> The topic is confusing, thanks for summarizing!

Thanks to you for all the clarifications and integrations.

Rocky

Use of disklabel, MBR and GPT

[Rocky Hotas]

Hello!
After partitioning a new disk with MBR, creating two primary partitions,
I was trying to define a disklabel in each of them. Quite soon I
realized it was not possible. After an IRC conversation, which was very
instructive yesterday, here are some considerations.

An important premise is that disklabel, MBR and GPT are data structures
intended to describe a whole disk, not a single partition: internally,
they then divide the disk into partitions. These single partitions can
not host themselves a disklabel, an MBR or a GPT to further subdivide
them, because they are not an entire disk.

Maybe this is trivial, but during the installation of NetBSD a
disklabel is actually nested inside a primary MBR partition. So, I would
like to share some further clarification.

Assume that you have this example MBR:

1. NTFS, Windows system partition
2. NTFS, Windows data
3. (empty)
4. (empty)

(for simplicity, I will not consider the case of extended partitions,
but it would be similar).

Assume that, when installing NetBSD with sysinst(8),

 

partition 3. is chosen for NetBSD. sysinst(8) does not change MBR in
this case (thus preserving primary partitions 1. and 2.).

There are several other options:

- sysinst(8) may create a brand new MBR (warning: this will destroy all
  the existing data on the disk), with primary partitions defined by the
  user, who can then choose which of them will be used for NetBSD;
- through sysinst(8) the user may also modify an existing MBR, and in
  this case it's his responsibility to preserve the partitions dedicated
  to other OSs and to manage only the free space in the disk for NetBSD;
- sysinst(8) also offers the chance to `Use the entire disk': in this
  case, it creates anyway a MBR, with a primary partition which extends
  to the whole disk, used for NetBSD.

As you may note, the use of MBR is never abandoned. This is done for two
reasons:

- to grant the compatibility with PC/DOS (and other OSs like Windows
  this way can still use the same disk);
- to grant a regular boot on PC/IBM hardware (where, without a valid
  MBR, the BIOS could not even try to search a bootloader on the disk).

Note that the above behaviour is about amd64; for other ports
(https://www.netbsd.org/ports/) it may change.

Whatever choices the user makes, in the end there will be an MBR primary
partition dedicated exclusively to NetBSD. Inside it, sysinst(8) installs
disklabel. disklabel(8) allows to further divide this NetBSD-dedicated
MBR partition, originating disklabel partitions `a' (which is generally
used for the NetBSD root directory /), `b' (generally used for swap), `e'
(it may be /home, for example), `f' (for example, /usr), `g' (for
example, /var), etc.; in amd64, partitions `c' and `d' are reserved, to
represent respectively the NetBSD MBR primary partition and the whole
disk, as shown here:

 

It may seem that this disklabel, nested into an MBR primary partition,
is used to further divide internally this partition: so, for another MBR
primary partition on the same disk, there may be another disklabel which
can be used the same way. This is wrong.

Disklabel is an alternative, and not an integration, to MBR.

In the case of an amd64 NetBSD installation inside a PC/DOS disk, the
disklabel is placed inside the MBR primary partition dedicated to NetBSD,
in the second block (and the first block is used for the NetBSD
bootloader). This is done to make sure that the disklabel data structure
is preserved and not altered or deleted by the action of other OSs: in
fact, the MBR primary partition dedicated to NetBSD is not used by DOS
or similar OSs, but it is recognized by them.
Without this issue, disklabel (whose location is flexible) could be
placed in the second sector of the disk (while MBR is in the first), not
the second sector of a specific partition.

When the machine is booted with NetBSD, disklabel is the only way NetBSD
looks at the disk, the *entire* disk; MBR in this case is not considered
at all. (Obviously, during the installation, a reference to the NetBSD
bootloader is placed in the boot sector of MBR, so that NetBSD can be
booted). Disklabel provides exactly all the information MBR provides,
but in a way NetBSD can understand. It provides the full size of the
disk (disklabel partition `d' for amd64), the size and location of the
usable portion of the disk (disklabel partition `c', for example, which
represents in amd64 the NetBSD primary partition), and the structure of
this portion of the disk (disklabel custom partitions `a', `b', `e',
`f', `g', etc.). Thinking about MBR, it actually provides the same
information, but in a way the other OSs can understand: the full size
of the disk, the size and location of the usable portion(s) of the disk
and their structure (all the primary partitions whose type is
DOS-compatible).

Disklabel and MBR are then just tw

Re: Configure NetBSD as a gateway for LAN hosts

[Rocky Hotas]

On ott 12 13:55, John Nemeth wrote:

>  This is a very common configuration, so there is lots of
> documentation on the Internet about how to do it.  For NetBSD in
> particular, take a look at the Guide:
> 
> http://www.netbsd.org/docs/guide/en/
> http://www.netbsd.org/docs/guide/en/part-net.html

I had already checked out Section `IV. Networking and related issues',
but it seems sometimes to be outdated (for example, when speaking about
ipfilter) or without examples (I'll try to better explain this below).

>  Yes, you will need this, unless you have 'options GATEWAY' in
> your kernel config.

Ok! (I have a GENERIC, so I it's necessary).

> } - put `gateway_enable="YES"' in /etc/rc.conf.
> 
>  I don't know what the source of this is, but it doesn't do
> anything on NetBSD.

Oh, ok. It comes from:

 

>  You will need to use one of the packet filters in order to do
> NAT.  The example above is overkill for your needs (l2tp
[...]
> You would probably be better off starting with soho_gw-npf.conf

This is all very useful.

> Note
> that the examples in the Guide use ipfilter which will work for
> now, but will likely be deleted at some point.

This is why Section `24.5. Setting up an Internet gateway with IPNAT'
gave me only a partial help. I should use npf instead of ipnat, through
a normal Ethernet connection to the modem, instead of a PPP direct
connection with the ISP. There are several adaptations to be made for my
case I can't figure out.

>  No, assuming the standard ISP setup where you're provided with
> a single IPv4 address, a bridge won't work.

Yes, the modem has a single IPv4 address.

> See the Guide for how to configure routing.

The Guide deals with this in Section `23.5. Subnetting and Routing', but
only with a theoretical example, with no route(8) commands.

However, maybe I'm overlapping the roles of routing with the role of npf.

I probably have no difficulties in configuring the routing as regards the
netbsd_gateway host itself. Something like:

Internet:
DestinationGatewayFlagsRefs  UseMtu
Interface
defaultmodem_IP   UG  --  -  NIC2
127/8  localhost  UGR --  33624  lo0
localhost  lo0UHl --  33624  lo0
subnet2link#2 U   --  -  NIC2
subnet1link#1 UHl --  -  NIC1

But when netbsd_gateway receives a packet from a host in subnet1, whose
destination is a remote host in the internet, how must it be instructed
to forward the packet to modem_IP through NIC2? With a routing table
entry, or with a rule (the `pass stateful out all' in soho_gw-npf.conf)
in npf?

Bye and thank you!

Rocky

Re: Configure NetBSD as a gateway for LAN hosts

[Rocky Hotas]

On ott 12 14:55, Jason Mitchell wrote:

> You have a choice between routing and bridging.

[...]

> b) Do NAT/PAT so that the modem only sees traffic coming from the NIC
> 2 network

In case of routing, I would prefer this solution, because I would like
the modem only sees traffic coming from the NIC 2 network. The
netbsd_gateway should completely manage the traffic.

> I'm also assuming that the modem is a router and NAT device as well. If
> it's not, then you need to do b). 

Yes, it is, even if choosing b) maybe doesn't require this.

> I think NAT/PAT is easier anyway and I think routing is easier to
> troubleshoot.

This is good news!

> But bridging would be simpler to set up.

You only mentioned bridging: how would it be accomplished?

> HTH,

It does!! Thanks :)

Rocky

Configure NetBSD as a gateway for LAN hosts

[Rocky Hotas]

Hello!
Thanks to your suggestions for a NIC (in particular, thanks to Martin:
Realtek worked), I configured a second NIC in a NetBSD 9.0 (release)
machine.
I would like to use it as a 1) gateway and 2) DHCP server, but didn't
find much documentation as regards problem 1).

Assume that the machine's hostname is netbsd_gateway and its two NICS
are NIC1 and NIC2.

My intention is to create two subnets: subnet1 for all the LAN hosts,
included NIC1, and subnet2 just for NIC2 and the modem. This second
subnet should never be directly accessible from the LAN hosts.

In this moment, netbsd_gateway should simply forward the packets
(sent from LAN hosts to the external internet) to the modem and the
packets from the modem (coming from internet) to the proper LAN
destination host.

(As a further step, I would like to use a traffic shaping tool, to tweak
the available bandwidth and priority for single hosts, but this is a
separate problem).

IIUC, some preliminary operations are:

- put `net.inet.ip.forwarding=1' in /etc/sysctl.conf;
- put `gateway_enable="YES"' in /etc/rc.conf.

But then I don't know how to proceed. Which is the correct approach?
Should I use npf? I found that /usr/share/examples/npf/l2tp_gw-npf.conf
depicts something similar to what I'm trying to do, but it includes
several filterings and protocols.
Should I build a bridge? And how to configure the routing tables?

I'm aware that these are many questions.
Of course, if anyone knows about a tutorial or guide, it's hugely
welcome!

Thank you in any case,

Rocky

Re: Suggestions for a discrete NIC compatible with NetBSD 9.0

[Rocky Hotas]

On ott 09 16:29, Martin Husemann wrote:
> 
[...]
> The others are all -current:
> re0 at pci2 dev 0 function 0: RealTek 8168/8111 PCIe Gigabit Ethernet (rev. 
> 0x15)
> re0: interrupting at msix3 vec 0
> re0: RTL8168H/8111H (0x5400)
> re0: Ethernet address 70:4d:7b:66:23:a2
> re0: using 256 tx descriptors
> rgephy0 at re0 phy 7: RTL8251 1000BASE-T media interface, rev. 0
> re0 at pci4 dev 0 function 0: RealTek 8168/8111 PCIe Gigabit Ethernet (rev. 
> 0x15)
> re0: interrupting at msix3 vec 0
> re0: RTL8168H/8111H (0x5400)
> re0: Ethernet address 00:d8:61:a4:18:9d
> re0: using 256 tx descriptors
> rgephy0 at re0 phy 7: RTL8251 1000BASE-T media interface, rev. 0
> 
> 
> No problems so far with any of those.

Ok! This is great. Thank you :)

Rocky

Re: Suggestions for a discrete NIC compatible with NetBSD 9.0

[Rocky Hotas]

On ott 08 12:59, Martin Husemann wrote:
> Most random cheap PCIe cards that you can get are re(4) using a realtek
> chipset and work well with NetBSD (in my experience).

Thanks! I found almost only cards with some flavours of chip RTL8111:
RTL8111E, RTL8111G, RTL8111H. re(4) mentions only 8111 (with no suffix)
and just in the name of the manpage.

src/sys/dev/pci/pcidevs has this line:

product REALTEK RT8168  0x8168  8168/8111 10/100/1000 Ethernet

I wonder if the suffixes E/G/H are important, and if these flavours of
Realtek 8111 chip would actually be supported by re(4).

Rocky

Re: Suggestions for a discrete NIC compatible with NetBSD 9.0

[Rocky Hotas]

On ott 08 13:40, Brad Spencer wrote:

> > They were faster for gig than the Realtek stuff I tried. (To be fair,
> > probably going back 10+ years though).

> Yes, I agree the Intel Pro cards work pretty well..  I tend to pick up
> used "Intel PRO/1000 PT (82571EB)", which are dual nics (probably server
> pulls).  Very often I will end up flashing newer firmware to them, as
> the used ones I have gotten have run older firmware.  Simply use a thumb
> drive booting DR-DOS with the Intel updater program on it.  Depending on
> your case, you may want to add a small fan to the heat sink if they are
> busy as they can get warm.  I had one in a case that did not quite
> circulate the air well enough to keep it as cool as I liked and a small
> fan running off spare +5v power dealt with that.

Thanks for your advices and for sharing your experience. It has been
very important. I don't actually need a double interface card. The
firmware and the heat sink are some additional tweakings I would like to
avoid; but regardless of this, I would like to buy a brand new card, and
unfortunately couldn't find much Intel chips. 

I'll try to search more for them.

Rocky

Suggestions for a discrete NIC compatible with NetBSD 9.0

[Rocky Hotas]

Hello!
I am running NetBSD 9.0 (release) on a system with an integrated NIC,
which is, from pcictl(8):

Intel PCH LAN (82578DM) Controller (ethernet network, revision 0x05)

It works. But I would like to add a network interface.
Maybe some of you remember my recent issues with NICs: therefore, before
blindly purchasing one, I would like to check out some candidates with
you: the card should be compatible with NetBSD.

I am looking for a discrete card, which is at least 10/100 Mbps full
duplex. Gigabit ethernet is optional but welcome. The case is Small Form
Factor, so the card should be Low Profile.

Available slots are: PCI, PCI Express x1, x4, x16. I am not sure about
the PCI Express version, but according to dmidecode it should be 2:

Type: x1 PCI Express 2
...
Type: x16 PCI Express 2

Any suggestions?
Bye!

Rocky

Re: Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

Hi!
Sorry for the huge delay.

On set 18 15:48, Vincent DEFERT wrote:
> Engrampa is the archive manager of MATE.

Oh, ok, I used MATE only a couple of times and didn't remeber it.

> The files besides the .bin have .exe, .dll and .sys suffixes, likely the
> firmware update utility for Windows users.
> _If your BIOS has a firmware update functionality_, it will be able to do
> the job safely, or will issue an error message it it can't.
> I have applied this technique to an Intel NUC and it is what I've observed.
> The only risk then is to use an improper firmware and brick the laptop, but
> I think you have already checked that.

Ok, thanks, now it's all more clear (I had not much experience with BIOS
updates).

> You may also want to check the release notes of the firmware update to see
> if it is worth applying.

A very good advice. After some tricky Google search, they should be:

https://whp-aus2.cold.extweb.hp.com/pub/softpaq/sp66501-67000/sp66770.html

The only relevant line I find is:

- Provides improved security for UEFI EDK2.

(and then HP ``strongly recommends'' it, maybe due to the security). I
have installed NetBSD without UEFI, thus using the old interface.
If this is the only feature carried by this update, it seems not useful for
my issue.
If you don't agree, or you would suggest anyway to make the update, let
me know.

Rocky

Re: Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

On set 17 13:13, Vincent DEFERT wrote:
> I have extracted it using Engrampa and made a tgz archive with the files in
> it.
> You can download it here: http://defert.com/tmp/sp66770.tgz
> 
> There's a .bin file in it, I think it's all you need on your USB stick.

Thanks, I didn't know Engrampa. I'm quite hesitant, being this is a
non-standard procedure: what if, during the BIOS installation, the program
requires the missing files and doesn't find them?
I'm not familiar with BIOS updates, the only thing I know is if something
goes wrong, the laptop could become useless.

Re: Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

On set 17 11:19, Martin Husemann wrote:

> The ACPI component in the kernel is newer, so it might trigger different
> bugs in the ACPI parts of your firmware.

Got it (and it absolutely makes sense). Is sounds like something which is
difficult to isolate and identify.

Re: Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

On set 17 19:03, SAITOH Masanobu wrote:

> I think it would be worth to try the following change:

I tried. Modified src/sys/dev/ic/rtl8169.c as suggested, then rebuilt
the 9.0-RELEASE kernel. Unfortunately the problem is still the same.
Here is the new dmesg:

https://termbin.com/qszzr

Thank you anyway!

Rocky

Re: Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

On set 16 22:50, Rocky Hotas wrote:

> Full dmesg from current (9.99.72) kernel:
> 
> https://termbin.com/uhaa

Sorry, of course this was from 9.0 (formal release) kernel. If you also
need dmesg from 9.99.72, let me know.

Rocky

Re: Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

On set 17  8:55, Vincent DEFERT wrote:
> You could try opening the firmware update's executable as an archive
> (they are often SFX - self-extracting ZIP archives),
> extract its contents to a FAT-formatted USB pen drive,
> reboot the laptop with the pen drive attached
> and enter the BIOS firmware flash utility (often the F7 key).

This is a good advice, which can be useful for similar cases. As regards
my laptop, the file is

https://ftp.hp.com/pub/softpaq/sp66501-67000/sp66770.exe

(yes, the Compaq BIOS update is distributed by HP, it sounded weird to
me, too).
I tried to extract it with GUI `Ark' on Linux, with CLI `unrar' and by
executing the file itself, after renaming it as `sfx', as suggested
here:

https://unix.stackexchange.com/a/79074
https://unix.stackexchange.com/a/115655

It didn't work, so maybe it's a different format.

> If you found a firmware update from 2014, your laptop may not be
> so old it can't update from a USB stick. :)

No, in fact IIRC it should also be able to boot from a USB stick (maybe
just for one of the external connectors, not all of them).

If the NIC never worked, a BIOS update could be meaningful. But this is
a different case. If with 8.1 the NIC worked, it should also work with
9.0.

Of course, a driver design requires a massive work I can't imagine and I
understand that simple needs of the user are not that much simple from
the developer's perspective.

Rocky

Re: Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

On set 16 12:22, Martin Husemann wrote:
> Can you please provide full dmesg, and output and from
> 
>   pcictl dump -b 3 -d 0 -f 0

Full dmesg from 8.1 kernel:

https://termbin.com/t57g

Full dmesg from current (9.99.72) kernel:

https://termbin.com/uhaa

Output of `pcictl pci0 dump -b 3 -d 0 -f 0':

https://termbin.com/udgn

> Could be anyhting, like newer ACPI version or whatever. Have you checked
> for firmware updates for your machine?

This is a very old laptop. I just found a 2014 firmware update (which
I'm almost sure had not been installed), but it's only available for
Windows, so, having only NetBSD, I couldn't install it.
About ACPI, I really don't know, but I hope that the two dmesg can
provide some more information.

Rocky

Re: Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

On set 15 14:18, Greg Troxel wrote:
> 
> I suspect MSI vs MSI-X is the big deal.

As in the message to Martin, yes, I have the same suspect.

Re: Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

On set 15 17:37, Martin Husemann wrote:

> Good point, can you try disabling rlphy in your kernel config?

As in my first message:

[With 9.0] ``I also tried to boot disabling rlphy(4) (`userconf
disable rlphy' from the boot prompt), and ukphy(4) is used instead,
but nothing changed. The same happens with netbsd-9 (stable)''.

with -current (NetBSD 9.99.72), the PHY is not even considered, the
last kernel lines regarding the NIC are just:

[ 1.004075] re0 at pci3 dev 0 function 0: RealTek
8100E/8101E/8102E/8102EL PCIe 10/100BaseTX (rev. 0x05)
[ 1.004075] re0: interrupting at msix3 vec 0
[ 1.004075] re0: reset never completed!
[ 1.004075] re0: Ethernet address ff:ff:ff:ff:ff:ff
[ 1.004075] re0: using 256 tx descriptors
[ 1.004075] ifmedia_set: no match for 0x20/0xfff

This also happens when booting 9.99.72 with `userconf disable rlphy'.
I don't know the underlying code, but maybe the shift to MSI-X is
more significant than PHY?

Rocky

Re: Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

Hello!
Sorry, I wasn't able to read this message till now, for some (recurring)
issues with this e-mail address (this is why I sometimes use
rockyho...@post.com instead).

On set 11  9:33, Greg Troxel wrote:

> It sounds like the only issue is the re0 card, and everything else is
> ok.  If that's not true, would be good to send an update with more
> details.

It really seems that everything else is ok, included the WiFi card.

> I would suggest comparing the 8.1 dmesg to the 9 one carefully
> especially around interrupts.  It sort of seems like maybe the tx done
> and/or receive interupts are not arriving.

Yes, this was similar to the suggestion from Martin. The difference I
see is that 8.1 used MSI interrupts, while 9.0 uses MSI-X.
Also, 8.1 used ukphy(4), while 9.0 uses rlphy(4), which is specific for
Realtek.

Rocky

Re: Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

Hi!

> Sent: Sunday, September 13, 2020 at 8:28 PM
> From: "Martin Husemann" 
> To: "Rocky Hotas" 
> Cc: "Valery Ushakov" , "NetBSD Users Mailing List" 
> 
> Subject: Re: Unusable Realtek NIC after upgrading to NetBSD 9
...
> With the old working kernel, what interrupt target did you get?
> The MSI / MSI-X interrupt support is new and may not work on your board
> (for whatever reasons we would still need to analyze).

Here is the dmesg using 8.1 (formal release) kernel:

re0 at pci3 dev 0 function 0: RealTek 8100E/8101E/8102E/8102EL PCIe 
10/100BaseTX (rev. 0x05)
re0: interrupting at msi2 vec 0
re0: Unknown revision (0x40c0)
re0: Ethernet address 28:92:4a:29:53:5a
re0: using 256 tx descriptors
ukphy0 at re0 phy 7: OUI 0x00e04c, model 0x0008, rev. 2
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto

So, it is MSI instead of MSI-X.
This is from ifconfig:

re0: flags=0x8843 mtu 1500
capabilities=3f80
capabilities=3f80
enabled=0
ec_capabilities=3
ec_enabled=0
address: 28:92:4a:29:53:5a
inet 192.168.1.27/24 broadcast 192.168.1.255 flags 0x0
inet6 fe80::2a92:4aff:fe29:535a%re0/64 flags 0x0 scopeid 0x1

I checked again and the NIC is perfectly working in this case.

Rocky

Re: Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

Hello! 

Sent: Sunday, September 13, 2020 at 8:50 AM
From: "Clay Daniels" 
To: "Valery Ushakov" 
Cc: netbsd-users@netbsd.org
Subject: Re: Unusable Realtek NIC after upgrading to NetBSD 9

> I would highly recommend the 9.0 Stable snapshot at:
> http://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/[http://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/]
> The snapshot of 12 Sept works great. Give it a try.
> 
> Clay

I did as suggested, tried to boot (same procedure I mentioned in
the reply to Valery in this thread) with this kernel:

http://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/202009120220Z/amd64/binary/kernel/netbsd-GENERIC.gz

This should be from September, 12. However, if I try `uname -a':

NetBSD hostname 9.0_STABLE NetBSD 9.0_STABLE (GENERIC) #0: Fri Sep  4 18:54:33 
UTC 2020  mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC 
amd64

The NIC has the same issue with this kernel:

[ 1.053809] re0 at pci3 dev 0 function 0: RealTek 8100E/8101E/8102E/8102EL 
PCIe 10/100BaseTX (rev. 0x05)
[ 1.053809] re0: interrupting at msix3 vec 0
[ 1.053809] re0: Ethernet address 28:92:4a:29:53:5a
[ 1.053809] re0: using 256 tx descriptors
[ 1.053809] rlphy0 at re0 phy 7: RTL8201E 10/100 media interface, rev. 2
[ 1.053809] rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
...
[   402.940784] re0: watchdog timeout
[   407.962941] re0: watchdog timeout
 
re0: flags=0x8843 mtu 1500
capabilities=3f80
capabilities=3f80
enabled=0
ec_capabilities=3
ec_enabled=0
 address: 28:92:4a:29:53:5a
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 192.168.1.27/24 broadcast 192.168.1.255 flags 0x0
inet6 fe80::2a92:4aff:fe29:535a%re0/64 flags 0x0 scopeid 0x1

PING www.google.it (216.58.208.131): 56 data bytes
64 bytes from 216.58.208.131: icmp_seq=0 ttl=114 time=7026.321123 ms
64 bytes from 216.58.208.131: icmp_seq=1 ttl=114 time=6021.678435 ms
64 bytes from 216.58.208.131: icmp_seq=2 ttl=114 time=5009.271834 ms

www.google.it PING Statistics
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 5009.271834/6019.090464/7026.321123/1008.527135 
ms

Rocky

Re: Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

Hello Valery,

> Sent: Friday, September 11, 2020 at 9:37 PM
> From: "Valery Ushakov" 
> To: netbsd-users@netbsd.org
> Subject: Re: Unusable Realtek NIC after upgrading to NetBSD 9
...
> Data point.  I had something that looked similar enough to this with
> my USL-5p (landisk)
>
> re0 at pci0 dev 0 function 0: RealTek 8139C+ 10/100BaseTX (rev. 0x20)
> re0: interrupting at irq 5
> re0: Ethernet address 00:a0:b0:65:15:6c
> re0: using 64 tx descriptors
> rlphy0 at re0 phy 0: Realtek internal PHY
> rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
>
> It was mostly sitting idle then one day it popped this watchdog
> timeout and lost its network iirc.

Thanks for sharing the behaviour of your card. I don't know it, but the
re(4) and rlphy(4) drivers are the same as mine.

> So, I'd try with booting a current'ish kernel on your machine.  Enough
> compat is enabled by default in GENERIC, so you can just drop in
> current as netbsd.new and boot it one off manually with the existing
> 9.0 install.

This is a great advice, which saves a lot of time. I did as suggested.
I updated the current source and built the kernel, then I selected `3' in
the boot menu to access the boot prompt and wrote `boot netbsd.new'.
It is NetBSD 9.99.72.

Things got even worse in this case:

[ 1.004075] re0 at pci3 dev 0 function 0: RealTek 8100E/8101E/8102E/8102EL 
PCIe 10/100BaseTX (rev. 0x05)
[ 1.004075] re0: interrupting at msix3 vec 0
[ 1.004075] re0: reset never completed!
[ 1.004075] re0: Ethernet address ff:ff:ff:ff:ff:ff
[ 1.004075] re0: using 256 tx descriptors
[ 1.004075] ifmedia_set: no match for 0x20/0xfff
...
[ 7.727294] entropy: ready
[ 8.174418] re0: reset never completed!
[ 8.194411] re0: reset never completed!
...
[19.964983] re0: watchdog timeout
[19.974843] re0: reset never completed!


re0: flags=0x8843 mtu 1500
capabilities=3f80
capabilities=3f80
enabled=0
ec_capabilities=3
ec_enabled=0
address: ff:ff:ff:ff:ff:ff
media: Ethernet none (none)
inet6 fe80::a3e:8eff:fe6a:527a%re0/64 flags 0x0 scopeid 0x1

I also tried to `shutdown -p now' and powered on again (instead of
`shudown -r now') but with no difference.

With the 9.0 formal release the NIC latency is very high, but it
communicates with DNS servers and the internet (though with a 10 s Google
ping). In this case, instead, with the unusual ff:ff:ff:ff:ff:ff MAC
address, the NIC card doesn't work at all.

This is IIRC the 4th bad experience with a NIC on NetBSD and, at least
as regards the issues I encountered, with -current (instead of the
formal release) the issue was never solved or mitigated. Maybe this is
due to some modifications in PCI or in the driver file, starting from
NetBSD-9.0, which are permanently now in the code, also in newer
versions.

Rocky

Unusable Realtek NIC after upgrading to NetBSD 9

[Rocky Hotas]

Hello!
Using NetBSD 8.1 in a laptop, both its Ethernet and WiFi NICs worked.
Then, I made a NetBSD 9 (formal release) fresh install and the
Ethernet NIC is almost unusable.

Here, the relevant dmesg part:

[ 1.055234] re0 at pci3 dev 0 function 0: RealTek
8100E/8101E/8102E/8102EL PCIe 10/100BaseTX (rev. 0x05)
[ 1.055234] re0: interrupting at msix3 vec 0
[ 1.055234] re0: Ethernet address 28:92:4a:29:53:5a
[ 1.055234] re0: using 256 tx descriptors
[ 1.055234] rlphy0 at re0 phy 7: RTL8201E 10/100 media interface,
rev. 2
[ 1.055234] rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
auto

The card is detected and configured by the system, but it is extremely
slow: a Google ping may take up to 10 seconds (yes, 1 ms). An ssh
connection is extremely slow, making it unusable.

It seems that, if I use the card, the dmesg gets populated by this
repeated message:

[ 20690.751223] re0: watchdog timeout
[ 20695.773292] re0: watchdog timeout
[ 20706.809817] re0: watchdog timeout
[ 20720.853570] re0: watchdog timeout
[ 20730.887686] re0: watchdog timeout

I also tried to boot disabling rlphy(4) (`userconf disable rlphy' from
the boot prompt), and ukphy(4) is used instead, but nothing changed.
The same happens with netbsd-9 (stable).
This problem did not appear with NetBSD 8.1, so it is a regression.
Is there anything I can do?
Bye!

Rocky


P.S.1
>From `ifconfig -a' (note that the Ethernet cable is currently
disconnected, because I can't use the card):

re0: flags=0x8802 mtu 1500

capabilities=3f80
capabilities=3f80
enabled=0
ec_capabilities=3
ec_enabled=0
address: 28:92:4a:29:53:5a
media: Ethernet autoselect (none)
status: no carrier
inet 192.168.1.27/24 broadcast 192.168.1.255 flags 0x4
inet6 fe80::2a92:4aff:fe29:535a%re0/64 flags 0x8
scopeid 0x1


P.S.2

Note that this is not the same issue as the previous thread:

 

which has been solved.

Re: Use network printer from NetBSD

[Rocky Hotas]

On giu 27  9:02, Brett Lymn wrote:
 
> what if you try:
> 
> telnet printerip 9100
> 
> then enter
> 
> test
> 
> followed by control-m control-j control-l
> 
> that should do a carriage return, newline, new page resulting in a page 
> coming out
> of the printer.
 
It does! A new blank page is "printed". Maybe it is not because the Ctrl-M
Ctrl+J Ctrl+L sequence is correctly parsed by the printer, but (just as
in previous tests) because a blank page is "printed" for any data I send
as plain text. If there's some more attempt I can make, let me know.
 
Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

Hi!
I'm really sorry for the delay. Partly because of work, partly because I
got lost between the messages and the suggested tests. Apparently
printers are still an unexplored world for me!

On lug 04 12:33, MLH wrote:
> https://support.brother.com/g/s/id/htmldoc/mfc/cv_mfc9120cn/us/html/sug/chapter1_5_3.html
> Printer Emulation Support
> 
> Setting the Emulation mode
> 
> 1 
> (MFC-9120CN and MFC-9320CW) Press Menu, 4, 1.
> (MFC-9010CN) Press Menu, 3, 1.

I noticed this, but it doesn't apply to my printer, which is:

Brother MFC L2710 DN

User Manual is available here:

 

> Maybe look at the CUPS source for a BR_Script driver? See what they
> are doing to set the emulation language?

Unfortunately this is beyond my capabilities. Also, I can't guess how to
detect the operations performed by the Windows driver to select the
emulation mode, as IIRC was suggested in a previous message. Maybe a
network sniffer would be needed?

A really, really huge thank you for all the effort for this issue.

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

Hi!
As in my reply a couple of minutes ago, sorry for the huge delay.

On lug 04 13:35, Greg A. Woods wrote:
> There's one other way to possibly find out at least what command
> languages your printer understands (besides obviously understanding
> Brother's variant of PJL):
> 
>snmpwalk -c public 10.0.1.19 .1.3.6.1.4.1.2699.1.2
> 
> (that tool is of course from pkgsrc/net/net-snmp

I tried to install and use the tool, but it required to create a
configuration file (can't recall the name), which was not trivial and
which required a lot of information I randomly chose.
After that, I run the command as you suggested, but I got this:

$ snmpwalk -c public 192.168.1.15 .1.3.6.1.4.1.2699.1.2 
snmpwalk: No securityName specified

I must have made some mistake in the configuration of `snmpwalk',
installed and used on NetBSD 8.1 amd64. I'm so sorry for this,
because it would have given a lot of information: exactly all the
information we are looking for!

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

(re-send attempt after network fail)

On lug 03 15:45, Johnny Billquist wrote:

> I think Brett was also hoping that the page that came out would have the
> text "test" written on it. Probably in the top left corner, with some
> reasonable size and font.
> 
> But it sounds as if you did not get such a text on the paper then...? Which
> would definitely confirm that it does not understand PCL.

It's exactly as you said: no `test' text on the paper. Just a blank page
on both sides.
At least, it's clear that PCL is not understood...
 
Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On lug 04 16:05, Martin Neitzel wrote:

> The character hex 0C (decimal 12) is ascii "formfeed".
> 
> It may well be that these control strings are simply added to the
> print jobs you supply rather than expected from it.  An added
> formfeed at the ensures that the last sheet gets ejected.

Oh, ok, I was naively thinking that it was a sort of end character. A
clarification. The line in my paste

Protocols   TCP/IPIPP 

is due to a bad text formatting, because of its HTML origin. It is
actually:

Protocols   [X] TCP/IP  [X] IPP

where the user has the chance to select one of them, or both. The
default was with both selected. I made an attempt with just TCP/IP
selected and

{ printf 'test \014'; sleep 5; } | telnet  9100

but again the famous blank page was generated.

> I just sent you a longer email regarding these printer Service Names
> and lpr/netcat/cups offlist.

It is enormously useful, because, yes, I have a big confusion about how
printing is.
Thank you!

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On giu 26  6:45, Robert Nestor wrote:
> It’s been a few years since I set up my HL-1270N, but as I recall I found the 
> information on the internal queues and the requirement for sending the EOT at 
> the end of a document in the User’s Manual.  And I think  they get listed 
> when I ask the printer to print out a configuration report which can be done 
> from the BRAdmin utility, logging into the printer via web browser or holding 
> down the test button in the back of the printer.

After (literally) days of attempts, I found something very similar,
also thanks to what you suggested here. In the web configuration
interface, I found (as mentioned in a previous message from today):

Enabled Services
BRN3C2AF4E874A2
BINARY_P1
TEXT_P1
POSTSCRIPT_P1
PCL_P1
BRN3C2AF4E874A2_AT

Each of them has a minimum possible configuration. For example,

TEXT_P1
Service NameTEXT_P1
Service PortP1
Protocols   TCP/IPIPP
Filter  Text Substitution
Control Strings Beginning of Job1)
End of Job  11)\0C
Raw TCP Port9100
Service Options Bi-Directional

`1)' and `11)' are just the lines of a list menu. It seems (but it's
just a guess from me) that a plain text stream must not be introduced by
any character and that must be concluded by `\0c'.

I tried

{ printf 'test \014%%'; sleep 5; } | telnet  9100 

with no success.

How did you append `\004' to the end of your files with your old Brother
HL-1270N?

> I scanned thru the manual on your printer and didn’t find any of the same 
> information, but it looks like that printer has more features than a moon 
> rocket, so I could have missed something though.

It seems so to me too :).

Bye!

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On lug 04  7:41, MLH wrote:
> From the multiple empty page output, it really looks to me like
> the default emulation is HP LaserJet. I just can't see a way from
> the manual to change to BR-Script3 yet.

Yes, exactly, me neither. It seems that some models (mentioned in a
previous message in this thread, CP-L2550DN/MFC-L2730DW/MFC-L2750DW) support
the emulation selection, but mine doesn't.

As regards the `TEXT_P1' of my previous message, I also tried something
extremely simple:

{ printf 'test \014%%'; sleep 5; } | telnet  9100 

Because it seemed that no introductory character was needed, and than
the ending character should be hex 0c, that is octal 014. But again
blank pages are generated.

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On lug 04  2:18, Greg A. Woods wrote:

> So, earlier in the thread you said:
> 
>   "Printer is in my case Brother MFC L2750DN."
> 
> However the link for a manual you pointed at was for the MFC-L2750DW.

I managed to do the same typo twice. Model is MFC L2710DN. So, the `DN'
part was always correct; I erroneously replaced `1' with `5'.
In fact, the formats found by Johnny Billquist, `PCL6, BR-Script3 and
PDF' are supported by models `DCP-L2550DN/MFC-L2730DW/MFC-L2750DW', but
not by my printer.

> I think the only difference between the DN and the DW is that the former
> may have the ability to use a larger toner cartridge?

It is no more important, but no, I think it's the WiFi interface
(present in DW, absent in DN).

> In any case the following command (where you replace my printer's
> address with yours) should tell you what language emulations your
> printer supports:
> 
>   { printf '\033%%-12345X@PJL\n'; printf "@PJL INFO ID\n@PJL INFO 
> CONFIG\n"; printf '\033%%-12345X'; sleep 5; } | telnet 10.0.1.19 9100

Thanks for this! I paste the relevant part of the output (WARNING: paste
is several lines long):

@PJL INFO ID
"Brother MFC-L2710DN series:8C5-K6G:Ver.Q"

@PJL INFO CONFIG
IN TRAYS [1 TABLES]
INTRAY2 PC
OUT TRAYS [1 ENUMERATED]
NORMAL FACEDOWN
PAPERS [21 ENUMERATED]
LETTER
LEGAL
A4
EXECUTIVE
COM10
DL
B5
A5
A6
MONARCH
C5
FOLIO
P3X5
POSTCARD
ENVCHOU3
A5L
SIXTEENK195X270
MEXICANLEGAL
A4SHORT
INDIALEGAL
USTATUS [4 ENUMERATED]
DEVICE
JOB
PAGE
TIMED
MEMORY=67108864
DISPLAY LINES=2
DISPLAY CHARACTER SIZE=16
LOCAL=ENGLISH

> lp|basic PJL printer using JetDirect protocol:\
> :sh:\
> :mx#0:\
> :lp=9100@10.0.1.19:\
> :sd=/var/spool/output/lpd-pjl:\
> :if=/usr/libexec/lpr/pjlfilter:\
> :lf=/var/log/lpd-errs:

I used this exact configuration (without modifying the
/usr/libexec/lpr/pjlfilter). Only blank pages were generated by the
printer.

> These printers also implement an LPD protocol spooler, though there may
> be an issue with knowing which remote queue name to use (in theory any
> random name will work, but some may have special meaning).

I really would not be able to know if my printer has a specific name for
this queue (unless another telnet "conversation" can reveal it).

> BTW, the term "BR-Script 3" refers to an emulated version of the Adobe
> PostScript 3 printer language from Brother.  It's just a trademark
> issue, just like GhostScript vs. PostScript.
> 
> I've got an HL-3045CN, and it's one of the "dumb" ones requiring a
> driver to send "binary" data to it, however it does claim to support the
> XL2HB emulation, which is apparently a modified version of HP PCL XL (an
> imaging "protocol" entirely unlike HP PCL).  For now since we've got
> several macOS machines here I just drive it from them using the easily
> installed CUPS driver for macOS.

In fact CUPS is the possible solution, if no other one is available, for
me too.
Thanks for your help!

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On lug 04  6:44, MLH wrote:

> The Letter vs A4 issue won't cause it to output multiple blank
> pages. That happened for me when the printer was emulating HPLaserjet.

Whoops, I really don't know the reason, then.
 
> Did you make sure the printer emulation was set for BRscript3?

I searched in the configuration panel, both physically in the printer,
and from the web interface. 
Till now I hadn't found nothing similar to emulation. In the TCP/IP
settings, however, I just found this:

Enabled Services
BRN3C2AF4E874A2
BINARY_P1
TEXT_P1
POSTSCRIPT_P1
PCL_P1
BRN3C2AF4E874A2_AT

Each of them has a minimum possible configuration. For example,

TEXT_P1
Service NameTEXT_P1
Service PortP1
Protocols   TCP/IPIPP
Filter  Text Substitution
Control Strings Beginning of Job1)
End of Job  11)\0C
Raw TCP Port9100
Service Options Bi-Directional

If you need the same details from any other line above (POSTSCRIPT_P1,
...), I will provide them.
I am very sorry for not noticing this before.
If you think I can made any attempt with `lpr' or even `netcat', I'm
ready to try. For example, maybe for the `Beginning of Job' no character
is needed (`1)' is just the first, blank line of a list), but the text
should be ended with the character `\0C'.

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On lug 03 19:49, MLH wrote:

> Yep... For me at least, adding
>  DefaultMedia: Letter
> to ~/.enscriptrc
> 
> got everything working again. So I will check into why /print/enscript
> is no longer adhering to PAPERSIZE at build time.

I tried with `DefaultMedia: Letter' as only line in `~/.enscriptrc', but
still the printer outputs blank pages. Thanks for investigating this,
but don't worry if it gets too much time or you can't. 

Rocky 

Re: Use network printer from NetBSD

[Rocky Hotas]

On lug 03 22:34, Rocky Hotas wrote:
> So, I did as follows:
> - installed just magicfilter and used the /etc/printcap file as you
>   suggested, in particular with
>   `if=/usr/pkg/libexec/magicfilter/psonly600-filter' (unmodified);

Actually, after this test, I installed enscript and tried with the same
/etc/printcap with no success. Then, I created...

> - ...a `psonly600alt-filter' where the
>   last line is exactly the one of your modification:
>   `default pipe /usr/pkg/bin/enscript -B -p-'

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On giu 26 18:37, MLH wrote:
> > > Then added the following to /etc/printcap :
> > > lp|Brother HL-5250DN:\
> > > :lp=:rp=brother5250:rm=brother5250:sd=/var/spool/output/lpd:\
> > > :if=/root/scripts/psonly600-filter:\
> > > :lf=/var/log/lpd-errs:mx#0:
> misfired
> 
>  :if=/usr/pkg/libexec/magicfilter/psonly600-filter:\

Sorry, I noticed your messages, but forgot to perform these tests.

The last line of the current psonly600-filter isn't (even now) the same
as your 2008 modification. It is (at least in my case):

`default pipe /usr/pkg/bin/a2ps -1 -B --borders=no -o -'

So, I did as follows:
- installed just magicfilter and used the /etc/printcap file as you
  suggested, in particular with
  `if=/usr/pkg/libexec/magicfilter/psonly600-filter' (unmodified);
- installed also enscript and created a `psonly600alt-filter' where the
  last line is exactly the one of your modification:
  `default pipe /usr/pkg/bin/enscript -B -p-'

In both cases, the printer generated only a long series of blank pages
(I stopped it manually). The /usr/pkg/libexec/magicfilter/ directory
contains a lot of filters; if you think that some other could be worth
testing, let me know.
In any case, thank you so much!

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On lug 03 15:45, Johnny Billquist wrote:

> I think Brett was also hoping that the page that came out would have the
> text "test" written on it. Probably in the top left corner, with some
> reasonable size and font.
> 
> But it sounds as if you did not get such a text on the paper then...? Which
> would definitely confirm that it does not understand PCL.

It's exactly as you said: no `test' text on the paper. Just a blank page
on both sides.
At least, it's clear that PCL is not understood...

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On giu 26  7:51, Greg Troxel wrote:

> Search the net for how people deal with this using Free Software under
> Other Operating Systems.

The information from Johnny Billquist and other was very useful to
understand what to expect or (at least) what to look for.

A massive thank you, to you and anyone who gave some advice and shared the
configuration files.

I think that in the end, if it's the only solution, I will try CUPS, as
you suggested since the beginning.

Bye!

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On giu 27  9:02, Brett Lymn wrote:

> what if you try:
> 
> telnet printerip 9100
> 
> then enter
> 
> test
> 
> followed by control-m control-j control-l
> 
> that should do a carriage return, newline, new page resulting in a page 
> coming out
> of the printer.

It does! A new blank page is "printed". Maybe it is not because the Ctrl-M
Ctrl+J Ctrl+L sequence is correctly parsed by the printer, but (just as
in previous tests) because a blank page is "printed" for any data I send
as plain text. If there's some more attempt I can make, let me know.

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On giu 26 15:35, Johnny Billquist wrote:

> By the way, the next step here is to check in the menu system of the printer
> if you can set the printer emulation. If you can, then set it to PCL then,
> and then do the printf | netcat thing. Hopefully that should then work...
> 
> If you don't have a setting for printer emulation, I would say that then you
> are stuck with the proprietary format, and your best bet is CUPS.

I'm afraid this is the case.

> If you can do PCL, then you just need something that can translate whatever
> documents you have to PCL, and you can just throw that at port 9100.

OK! This can be anyway useful, for any other situation.

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On giu 26 14:42, Johnny Billquist wrote:

[...]

> Essentially, the implications are that through port 9100 you just talk
> directly to the printer. But that still leaves figuring out what format the
> printer wants data in... (For HP printers, it was/is easy, it is PCL, and
> sometimes they also understand other things...)

Ok, this puts a big light on why it didn't work.

> Looking at the documentation, the following list of printers supports PCL6,
> BR-Script3 and PDF:
> 
> DCP-L2550DN/MFC-L2730DW/MFC-L2750DW.
> 
> I suspect BR-Script3 is Brothers own print
> handling/layout/formatting/whatever protocol
[...]
> It do.
> Page 615. "Emulation".

Thanks! I really would have never found the section where those 3 models
and formats are mentioned. Mine is not between them, but at least now it's
more clear what to search, and where.

(Not only I didn't know what kind of data my printer supports, but also I
didn't know exactly what kind of data to look for).

> Yes. The printer does not understand Postscript.

Ok! Apparently, only BR-Script3 as you were mentioning previously.

> Your problems are twofold, I'd say. First, trying to use various daemons and
> ports used by these daemons, but without the protocols, is not working.

Exactly.

> The closest is when you talk on port 9100. Then you have a clean connection
> to the actual printer without any daemons involved.
> But then you hit the problem that the printer expects a format for the
> things to be printed which you are not giving it. So you are feeding it
> things it don't understand. You're lucky you get nothing, You could be
> getting 200 pages of gibberish as well. :-)

In fact I was expecting this mess!! :) My printer seems to be gentle.
Thanks for all your help.

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On giu 25 14:18, Greg Troxel wrote:

> The first thing to understand is what format the printer wants.  Back in
> the old days, printers were postscript, and you basically had to send
> them postscript.  (Or HP's PCL.)

Yes, of couse, but how?
 
> I think 9100 is what HP called "jetdirect".  I somewhat fuzzily think
> that connecting to that is just like sending bytes over a serial port.

Yes, it is JetDirect, an HP technology, but also adopted by other
manufacturers.

> That doesn't really surprise me if the printer expects postscript.  You
> omitted the model from your email.

As in a previous answer to this thread, it is Brother MFC L2750DN.

> 0) read enough docs/etc. to find out what the printer wants

 

The user manual seems not to mention anything useful about queues of
accepted input formats.
How to correctly guess the requirements of a printer?

> 1) Try a postscript file.

Done with netcat. You can check out my reply to JingYuan. I also tried
it directly: `lpr testfile.ps'. The printer display lights up, shows
`Receiving data', and then nothing happens.

> 2) what you are basically trying to do is have  a printer configured,
> just like it was attached, except you it isn't physically attached.

Basically yes.

>   - put it on an Ethernet by itself, hooked up to an extra interface on
> a computer [...]
> 
>   - configure a print server that owns the printer [...]

In my case, security is not an issue and I would like not to install
something extra, even because I could move the NetBSD host to another
LAN, and I could need to use another printer.

Moreover, the way you suggest, each printer should be related to a
host with a CUPS server installed and configured. I imagine this is the
ideal scenario, but I'm afraid it could be too much, in my case.

>   - make all access go through the daemon.  Mutliple users accessing a
> network printer at once is a bit of a mess anyway.

This could be another issue, but I am sure that during my tests I was
the only one to access the printer.
Thanks for your suggestions,

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On giu 26 10:38, JingYuan Chen wrote:
> How about using pdf2ps and netcat ?
> 
> pdf2ps urpdf.pdf - | netcat -w 1 printer_IP printer_port

Thanks for this suggestion. I tried, but when using 515 as printer_port
nothing happens. When instead using 9100, in the printer display
`Receiving data' appears, but again nothing more happens.

The same if I use just a plain text string, terminated with `\004' as
suggested by Bob.

printf 'somestring\x04' | nc -w 1 printer_IP printer_port

Rocky

Re: Use network printer from NetBSD

[Rocky Hotas]

On giu 25 19:45, Robert Nestor wrote:
> On my old Brother HL-1270N I found I had to append a “\004” to the end of any 
> file sent to the printer to get it to print.

Ok! How did you figure it out? In my case, I tried to append a `\004'
(ASCII EOT) to my test plain-text file, but nothing happened.

> The printer has different internal “queues”

How did you manage to know this and how did you obtain their names,
like POSTSCRIPT_P1 and (in your printcap file) TEXT_P1?

> lp|patches:\
>  :sh:lp=:mx#0:rm=patches:rp=TEXT_P1:lf=/var/log/lpd-errs:\
>  :sd=/var/spool/output/lpd:

I tried to use this same file, but I am still not able to determine
the queues of my printer, neither if it requires a special end
character.
Printer is in my case Brother MFC L2750DN. I could not find anything
useful in the user's manual: print queues are not mentioned and
PostScript is only considered about an improvement quality Windows
driver. Manual is:



Thank you for sharing your printcap file and for all your advices.

Rocky

Use network printer from NetBSD

[Rocky Hotas]

Hello!
I would like to be able to print on NetBSD, but I don't want (and don't
need) to create a printer server. In fact, the printer is not directly
connected to my NetBSD host: it is a stand-alone network printer, with
its own LAN IP address.
I know the printer hostname and its IP address. How to send from NetBSD
a print job?

The system default /etc/printcap file is all commented, and it includes
some examples. Looking them, I tried to create my own entry:

remote|My network printer:\
   :sh:lp=9100@:rm=:rp=:\
   :sd=/var/spool/output/remote:\
   :lf=/var/log/lpd-errs:if=/usr/libexec/lpr/pjlfilter:

I tried with and without pjlfilter, and also with and without
/usr/libexec/lpr/lpf. It makes no difference. I also tried port 515
instead of 9100: they are surely open. Printer also supports `LDP (PC
Fax Send)', `Raw Port', `IPP', `Mopria', `AirPrint'.
For my attempts, after starting the lpd daemon, I simply use:

$ lpr -P remote a_test_file

The curious fact is that, in any of these attempts:
- when a_test_file is a plain-text file, the printer display just lights
up, but nothing else happens;
- when a_test_file is a non-blank PDF, the printer seems to print, but
the output is an endless series of white papers. I have to stop the job
from the printer stop button.

Am I doing something wrong? What is the correct way to configure
/etc/printcap in this case?
Bye,

Rocky

Re: NetBSD 9.0 not properly configuring NIC

[Rocky Hotas]

Hi!

> Sent: Friday, March 13, 2020 at 7:59 PM
> From: "SAITOH Masanobu" 
> To: "Rocky Hotas" 
> Cc: msai...@execsw.org, "NetBSD Users Mailing List" 
> Subject: Re: NetBSD 9.0 not properly configuring NIC
>
> Hi.
>
> Please try atphy.c rev. 1.28.

Reading the commit messages, maybe you are meaning rev. 1.29.
I built and used the -current kernel (NetBSD 9.99.49 GENERIC), which
includes rev. 1.29 for atphy(4). It works: NIC is ok.

dmesg: https://pastebin.com/3VtfbKsC
`ifconfig -a': https://pastebin.com/b68p3Z9c

If it is possible, can you make a pull-up to netbsd-9?
A huge thank you for your work.

Rocky