Re: Kernel panic probably linked to inteldrm
Ok, after further investigations, my problem does not seems to be linked with inteldrm. The kernel panic seems to happen randomly with or without inteldrm enabled. Nevertheless, if I disable inteldrm, I can access ddb when the panic happen (I don't know why, BTW) and I get this. kernel: double fault trap, code=0 Stopped at restore_saved+0x1e: xorq 0x30(%rsp),%r11 I have no stack trace with the ???trace??? command and the command ???show panic??? tells me that the kernel do not panic (which is... strange?). If I correctly look at the kernel sources, restore_saved seems to be called by cpu_switchto which is responsible to switch processes between the different processors. Any idea about this problem?
Re: Python 3.8 os.listdir EINVAL on large directories
I'm guessing that the usage of getdents was intentional: I've used this
system call in the same situation (thousands of entries in a directory,
inside a NFS exported directory) to avoid calling stat() in each entry
returned by getdents.
It is indead a "low level" system call, and usually readdir is the
preferred method, but might be too slow for your usage case (in my case
it was, it took hours to read the directory while with getdents took
minutes).
It doesn't seems a problem with OpenBSD, but with Python code. Looks
like some additional logic will be required to address the differences
betweens the UNIX-like OSs and their implementation of getdents.
On the other hand, by the error message the Python code is passing a
wrong argument, maybe it is a bug instead of a portability issue?
Em 25/07/2020 20:50, Aaron Miller escreveu:
> Hi all,
>
> I am getting a stacktrace from the borg command in the borgbackup
> package while checking a backup (see bottom of email for full
> output, since it's verbose). The relevant part is this:
>
> filenames = os.listdir(os.path.join(data_path, dir))
> OSError: [Errno 22] Invalid argument:
> '/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'
>
> This is same error is reproducible with a test Python 3.8 program:
>
> #!/usr/bin/env python
>
> import os
> os.listdir('/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/')
>
> Running ktrace & kdump reveals the error is from calling
> getdents(2):
>
> 76903 python3.8
> CALL open(0x1ec7f06de3b0,0x3)
> 76903 python3.8
> NAMI "/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/"
> 76903 python3.8 RET open 3
> [...]
> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
> 76903 python3.8 RET getdents 16384/0x4000
> [...]
> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
> 76903 python3.8 RET getdents 16384/0x4000
> [...]
> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
> 76903 python3.8 RET getdents 16384/0x4000
> [...]
> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
> 76903 python3.8 RET getdents -1 errno 22 Invalid argument
>
> Looking at the man page for getdents(2), I found it interesting
> that it says this call "is not a portable interface and should not
> be used directly by applications" and it recommends using
> readdir(3) instead.
>
> To give you a rough idea of the number of files and filename sizes
> in this directory:
>
> $ ls /mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/ | wc
> 15341534 10738
>
> Where does the problem lie -- the upstream Python code, the
> OpenBSD-specific patches in its port definition, or somewhere
> else? And in case it matters, this is a -current amd64 system,
> with "sysupgrade -s" executed on 7/15.
>
> Thank you,
> Aaron Miller
>
> --
> Exception ignored in: 0x1e17e13fd310>
> Traceback (most recent call last):
> File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 180, in __del__
> assert False, "cleanup happened in Repository.__del__"
> AssertionError: cleanup happened in Repository.__del__
> Local Exception
> Traceback (most recent call last):
> File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
> line 4565, in main
> exit_code = archiver.run(args)
> File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
> line 4497, in run
> return set_ec(func(args))
> File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
> line 161, in wrapper
> with repository:
> File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 190, in __enter__
> self.open(self.path, bool(self.exclusive),
> lock_wait=self.lock_wait, lock=self.do_lock)
> File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 450, in open
> segment = self.io.get_latest_segment()
> File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 1253, in get_latest_segment
> for segment, filename in self.segment_iterator(reverse=True):
> File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 1241, in segment_iterator
> filenames = os.listdir(os.path.join(data_path, dir))
> OSError: [Errno 22] Invalid argument:
> '/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'
>
> Platform: OpenBSD millipede.iforgotmy.name 6.7 GENERIC.MP#348
> amd64
> Borg: 1.1.13 Python: CPython 3.8.3 msgpack: 0.5.6
> PID: 31745 CWD: /mnt/thinkpad_void_obsd_borg
> sys.argv: ['/usr/local/bin/borg', 'check', 'thinkpad.borg']
> SSH_ORIGINAL_COMMAND: None
>
Re: Python 3.8 os.listdir EINVAL on large directories
Hi Alceu,
Note that there are additional messages about this in tech@.
--Aaron
On July 26, 2020 8:53:00 AM PDT, Alceu Rodrigues de Freitas Junior
wrote:
>I'm guessing that the usage of getdents was intentional: I've used this
>system call in the same situation (thousands of entries in a directory,
>inside a NFS exported directory) to avoid calling stat() in each entry
>returned by getdents.
>
>It is indead a "low level" system call, and usually readdir is the
>preferred method, but might be too slow for your usage case (in my case
>it was, it took hours to read the directory while with getdents took
>minutes).
>
>It doesn't seems a problem with OpenBSD, but with Python code. Looks
>like some additional logic will be required to address the differences
>betweens the UNIX-like OSs and their implementation of getdents.
>
>On the other hand, by the error message the Python code is passing a
>wrong argument, maybe it is a bug instead of a portability issue?
>
>Em 25/07/2020 20:50, Aaron Miller escreveu:
>> Hi all,
>>
>> I am getting a stacktrace from the borg command in the borgbackup
>> package while checking a backup (see bottom of email for full
>> output, since it's verbose). The relevant part is this:
>>
>> filenames = os.listdir(os.path.join(data_path, dir))
>> OSError: [Errno 22] Invalid argument:
>> '/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'
>>
>> This is same error is reproducible with a test Python 3.8 program:
>>
>> #!/usr/bin/env python
>>
>> import os
>> os.listdir('/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/')
>>
>> Running ktrace & kdump reveals the error is from calling
>> getdents(2):
>>
>> 76903 python3.8
>> CALL open(0x1ec7f06de3b0,0x3)
>> 76903 python3.8
>> NAMI "/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/"
>> 76903 python3.8 RET open 3
>> [...]
>> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
>> 76903 python3.8 RET getdents 16384/0x4000
>> [...]
>> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
>> 76903 python3.8 RET getdents 16384/0x4000
>> [...]
>> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
>> 76903 python3.8 RET getdents 16384/0x4000
>> [...]
>> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
>> 76903 python3.8 RET getdents -1 errno 22 Invalid argument
>>
>> Looking at the man page for getdents(2), I found it interesting
>> that it says this call "is not a portable interface and should not
>> be used directly by applications" and it recommends using
>> readdir(3) instead.
>>
>> To give you a rough idea of the number of files and filename sizes
>> in this directory:
>>
>> $ ls /mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/ | wc
>> 15341534 10738
>>
>> Where does the problem lie -- the upstream Python code, the
>> OpenBSD-specific patches in its port definition, or somewhere
>> else? And in case it matters, this is a -current amd64 system,
>> with "sysupgrade -s" executed on 7/15.
>>
>> Thank you,
>> Aaron Miller
>>
>> --
>> Exception ignored in: > 0x1e17e13fd310>
>> Traceback (most recent call last):
>> File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 180, in __del__
>> assert False, "cleanup happened in Repository.__del__"
>> AssertionError: cleanup happened in Repository.__del__
>> Local Exception
>> Traceback (most recent call last):
>> File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
>> line 4565, in main
>> exit_code = archiver.run(args)
>> File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
>> line 4497, in run
>> return set_ec(func(args))
>> File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
>> line 161, in wrapper
>> with repository:
>> File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 190, in __enter__
>> self.open(self.path, bool(self.exclusive),
>> lock_wait=self.lock_wait, lock=self.do_lock)
>> File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 450, in open
>> segment = self.io.get_latest_segment()
>> File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 1253, in get_latest_segment
>> for segment, filename in self.segment_iterator(reverse=True):
>> File "/usr/local/lib/python3.8/site-
>> packages/borg/repository.py", line 1241, in segment_iterator
>> filenames = os.listdir(os.path.join(data_path, dir))
>> OSError: [Errno 22] Invalid argument:
>> '/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'
>>
>> Platform: OpenBSD millipede.iforgotmy.name 6.7 GENERIC.MP#348
>> amd64
>> Borg: 1.1.13 Python: CPython 3.8.3 msgpack: 0.5.6
>> PID: 31745 CWD: /mnt/thinkpad_void_obsd_borg
>> sys.argv: ['/usr/local/bin/borg', 'check', 'thinkpad.borg']
>> SSH_ORIGINAL_COMMAND: None
>>
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
