Re: panic: invalid bcd xxx

[Adrian Chadd]

On 2 March 2017 at 01:31, Matthias Apitz  wrote:
> On Thursday, 2 March 2017 00:37:35 CET, Michael Gmelin 
> wrote:
>>
>>
>>
>>> On 2 Mar 2017, at 00:35, Adrian Chadd  wrote:
>>>
>>> This is an emulated BIOS though, right?
>>>
>>> I don't know if we're going to get the RTC 'bugfixed'...
>>>
>>
>>
>> It's SeaBIOS, yes. I feel like this might end up in another
>> quirk/workaround solution.
>>
>
> I'm one of the C720 owners and  apart of Michael, I only know two users more
> running FreeBSD. The SeaBIOS in our devices. is outdated, mine from 2013
> IIRC. I dont know if there is an easy way to update this.

We're not; we need to cope with crappy BIOS emulations and not crash :)

What's Linux doing instead? Ignoring the RTC?



-adrian
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CFLAGS for certain ports

[Shane Ambler]

On 04/03/2017 02:17, Julian Elischer wrote:

On 2/3/17 8:58 pm, Dimitry Andric wrote:

On 2 Mar 2017, at 12:02, Mingo Rrubioer  wrote:

I would like to see how well FreeBSD does as a workstation OS in the
HPC world due to its stability and reliability, as well as LLVM/clang.
I would like to know if FreeBSD has something similar to Gentoo's
/etc/portage/make.conf file and /etc/portage/package.use/* files in
order to compile certain ports with certain compiler flags.

It doesn't, though it would certainly be nice to have something like it
at some point.  The current idiom is to put something similar to the
following in your /etc/make.conf:

.if ${.CURDIR:M/usr/ports/foo/bar}
CFLAGS+= [... flags for the foo/bar port ...]
.endif

.if ${.CURDIR:M/usr/ports/what/ever}
CFLAGS+= [... flags for the what/ever port ...]
.endif



We can also put a Makefile.local in the port directory. There can also
be arch and system specific makefiles.

See /usr/ports/Mk/bsd.port.mk from about line 1211

https://svnweb.freebsd.org/ports/head/Mk/bsd.port.mk?view=markup#l1211


--
FreeBSD - the place to B...Software Developing

Shane Ambler

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r313268 - head/sys/kern [through -r313271 for atomic_fcmpset use and later: fails on PowerMac G5 "Quad Core"; -r313266 works]

[Mark Millard]

On 2017-Mar-3, at 6:17 AM, Rodney W. Grimes  wrote:

>> On 2017-Mar-2, at 7:19 AM, Steve Kargl  
>> wrote:
>> 
>> On Thu, Mar 02, 2017 at 01:10:21PM +0100, Mateusz Guzik wrote:
>>> On Wed, Mar 01, 2017 at 09:45:07AM -0800, Mark Millard wrote:
 
> Summary of the transition interval:
> 
> So for powerpc64 (and powerpc?) It is a good
> idea to avoid anything that is after -r313254
> and before -r314474 in head. (Would this be
> appropriate for a UPDATING notice given its
> span?)
> 
> There may be other architectures that might have
> a similar status(?): the last fixes involved were
> not in Machine Dependent code. (Some architectures
> are apparently insensitive to the errors, such as
> amd64).
> 
 
 When following current you are expected to be on the newest revision,
 so I don't think mentioning interim broken releases makes much sense.
 
>>> 
>>> Documenting the range may aid those bisecting src/ to find a bug. 
>>> How is one to know that anything in the range that Mark points
>>> out should be skipped on powerpc64?
>>> 
>>> -- 
>>> Steve
>> 
>> I have tested with a TARGET_ARCH=powerpc -r314473 build and
>> its kernel version has locking problems like
>> TARGET_ARCH=powerpc64 does for that version.
>> 
>> [Note: This was run on a PowerMac G5 so-called "Quad Core"
>> so most of the memory was ignored.]
>> 
>> Both TARGET_ARCH=powerpc64 and TARGET_ARCH=powerpc need -r314474
>> or later as of the new locking.
>> 
>> I've not explicitly tested other architectures. As I remember
>> armv6/v7 are classified as having some from of a weak memory
>> model compared to the likes of amd64. If so armv6/v7 might be
>> candidates for having problems. There might be other candidates.
> 
> I also had locking issues on amd64 around this build time that
> sent me down a week long rabbit hole chasing what I thought was
> a bug in the new AMD/IOMMU code.  IMHO if we can at least
> flag prior snapshot builds as "Broken for reason X" it might
> save someone some time and time is a one way depleting resource
> usually worth saving if possible.
> 
> If needed I can dig out the specifc build.  Oh, nvm, let me
> just do that, it was r309302.  This revision I beleive is
> a november snapshot.  It has kernel panics due to spinlock
> timeout and sparatic deadlock that is undetected.
> 
> 
> -- 
> Rod Grimes rgrimes at 
> freebsd.org

Sounds like that amd64 -r309302 problem might be another
good example.

Locking tends to be central and heavily used. When it
breaks many other things tend to also end up broken.
This is the sort of context I was thinking about if it
goes on very long.

I'm not sure that the -r309302 problem would reproduce
at -r313259 so -r309302 might be a separate issue. I've
no clue what -rdd range had the amd64 -r309302
problem.


Details that I'm aware of are something like:

-r309302 is dated 2016-Nov-30. (your reported amd64 locking problem's -rdd)

-r312973 is dated 2017-Jan-30. (example add of an atomic_fcmpset implementation)
   (getting ready for machine independent usage)

-r313259 is dated 2017-Feb-5.  (last before "machine independent use of 
atomic_fcmpset"?)
   (powerpc64 and powerpc working here)

-r313260 is dated 2017-Feb-5.  (first machine-independent usage of 
atomic_fcmpset?)

. . . (various machine-independent atomic_fcmpset usage check-ins) . . .

-r313271 is dated 2017-Feb-5.  (observed powerpc64 failures for this version)
   (powerpc would fail too)

. . . (various machine-independent atomic_fcmpset usage check-ins) . . .
. . . (powerpc64 [and powerpc] continuing to fail) . . .

-r314474 is dated 2017-Mar-1.  (powerpc64 and powerpc started working)



===
Mark Millard
markmi at dsl-only.net

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Problem with WITH_LLD_IS_LD

[Ed Maste]

On 3 March 2017 at 12:24, Jochen Neumeister  wrote:
> Hi current@
>
> I have a PR 217519 because of devel/pear: When WITH_LLD_IS_LD is set, the
> stage phase for devel/pear breaks.

Yes, there are issues building a number of ports with LLD at present.
There are some hacks / work in progress in both LLD and the Ports tree
to make things work.

The most common issue right now is that libtool looks at the output of
the linker's --version string and assumes that only linkers containing
the string "GNU" accept certain command-line options. Rafael
Espíndola's experimental Poudriere runs have a hack to have LLD
include something like "not GNU" as a workaround.

WIP changes on the ports side can be found in my ports tree clone on
GitHub: https://github.com/emaste/freebsd-ports/tree/ports-lld

At this point WITH_LLD_IS_LD works for the base system but is very
much experimental for building ports.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: effect of strip(1) on du(1)

[Alan Somers]

On Fri, Mar 3, 2017 at 10:25 AM, Allan Jude  wrote:
> On March 3, 2017 9:11:30 AM EST, "Rodney W. Grimes" 
>  wrote:
>>-- Start of PGP signed section.
>>[ Charset ISO-8859-1 unsupported, converting... ]
>>> On 2017-Mar-02 22:19:10 -0800, "Rodney W. Grimes"
>> wrote:
>>> >> du(1) is using fts_read(3), which is based on the stat(2)
>>information.
>>> >> The OpenGroup defines st_blocksize as "Number of blocks allocated
>>for
>>> >> this object."  In the case of ZFS, a write(2) may return before
>>any
>>> >> blocks are actually allocated.  And thanks to compression, gang
>>> ...
>>> >My gut tells me that this is gona cause problems, is it ONLY
>>> >the st_blocksize data that is incorrect then not such a big
>>> >problem, or are we returning other meta data that is wrong?
>>>
>>> Note that it's st_blocks, not st_blocksize.
>>Yes, I just ignore that digretion, as well as the digretion into
>>fts_read
>>being anything special about this, as it just ends up calling stat(2)
>>in
>>the end anyway.
>>
>>>
>>> I did an experiment, writing a (roughly) 113MB file (some data I had
>>> lying around), close()ing it and then stat()ing it in a loop.  This
>>is
>>> FreeBSD 10.3 with ZFS and lz4 compression.  Over the 26ms following
>>the
>>> close(), st_blocks gradually rose from 24169 to 51231.  It then
>>stayed
>>> stable until 4.968s after the close, when st_blocks again started
>>> increasing until it stabilized after a total of 5.031s at 87483.
>>Based
>>> on this, st_blocks reflects the actual number of blocks physically
>>> written to disk.  None of the other fields in the struct stat vary.
>> ^^^
>>Thank you for doing the proper regression test, that satisfies me that
>>we dont have a lattent bug sitting here and infact what we have is
>>exposure of the kernel caching, which I might be too thrilled about,
>>is just how its gona have to be.
>>
>>>
>>> The 5s delay is presumably the TXG delay (since this system is
>>basically
>>> unloaded).  I'm not sure why it writes roughly ? the data immediately
>>> and the rest as part of the next TXG write.
>>>
>>> >My expectactions of executing a stat(2) call on a file would
>>> >be that the data returned is valid and stable.  I think almost
>>> >any program would expect that.
>>>
>>> I think a case could be made that st_blocks is a valid representation
>>> of "the number of blocks allocated for this object" - with the number
>>> increasing as the data is physically written to disk.  As for it
>>being
>>> stable, consider a (hypothetical) filesystem that can transparently
>>> migrate data between different storage media, with different
>>compression
>>> algorithms etc (ZFS will be able to do this once the mythical block
>>> rewrite code is written).
>>
>>I could counter argue that st_blocks is:
>>st_blocks   The actual number of blocks allocated for the file in
>> 512-byte units.
>>
>>Nothing in that says anything about "on disk".  So while this thing
>>is sitting in memory on the TXG queue we should return the number of
>>512 byte blocks used by the memory holding the data.
>>I think that would be the more correct thing than exposing the
>>fact this thing is setting in a write back cache to userland.
>
> Can we compare the results of du with du -A?
>
> Du will show compression savings, and -A wont
>
> ZFS compresses between the write cache and the disk, so the final size may 
> not be know for 5+ seconds
> --
> Allan Jude

"du -A" does what you would expect.  It instantly reports the apparent
size of the file.  For incompressible files, this is actually less
than what "du" reports, because it doesn't take into account the znode
and indirect blocks.

-Alan
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: effect of strip(1) on du(1)

[Allan Jude]

On March 3, 2017 9:11:30 AM EST, "Rodney W. Grimes" 
 wrote:
>-- Start of PGP signed section.
>[ Charset ISO-8859-1 unsupported, converting... ]
>> On 2017-Mar-02 22:19:10 -0800, "Rodney W. Grimes"
> wrote:
>> >> du(1) is using fts_read(3), which is based on the stat(2)
>information.
>> >> The OpenGroup defines st_blocksize as "Number of blocks allocated
>for
>> >> this object."  In the case of ZFS, a write(2) may return before
>any
>> >> blocks are actually allocated.  And thanks to compression, gang
>> ...
>> >My gut tells me that this is gona cause problems, is it ONLY
>> >the st_blocksize data that is incorrect then not such a big
>> >problem, or are we returning other meta data that is wrong?
>> 
>> Note that it's st_blocks, not st_blocksize.
>Yes, I just ignore that digretion, as well as the digretion into
>fts_read
>being anything special about this, as it just ends up calling stat(2)
>in
>the end anyway.
>
>> 
>> I did an experiment, writing a (roughly) 113MB file (some data I had
>> lying around), close()ing it and then stat()ing it in a loop.  This
>is
>> FreeBSD 10.3 with ZFS and lz4 compression.  Over the 26ms following
>the
>> close(), st_blocks gradually rose from 24169 to 51231.  It then
>stayed
>> stable until 4.968s after the close, when st_blocks again started
>> increasing until it stabilized after a total of 5.031s at 87483. 
>Based
>> on this, st_blocks reflects the actual number of blocks physically
>> written to disk.  None of the other fields in the struct stat vary.
> ^^^
>Thank you for doing the proper regression test, that satisfies me that
>we dont have a lattent bug sitting here and infact what we have is
>exposure of the kernel caching, which I might be too thrilled about,
>is just how its gona have to be.
>
>> 
>> The 5s delay is presumably the TXG delay (since this system is
>basically
>> unloaded).  I'm not sure why it writes roughly ? the data immediately
>> and the rest as part of the next TXG write.
>> 
>> >My expectactions of executing a stat(2) call on a file would
>> >be that the data returned is valid and stable.  I think almost
>> >any program would expect that.
>> 
>> I think a case could be made that st_blocks is a valid representation
>> of "the number of blocks allocated for this object" - with the number
>> increasing as the data is physically written to disk.  As for it
>being
>> stable, consider a (hypothetical) filesystem that can transparently
>> migrate data between different storage media, with different
>compression
>> algorithms etc (ZFS will be able to do this once the mythical block
>> rewrite code is written).
>
>I could counter argue that st_blocks is:
>st_blocks   The actual number of blocks allocated for the file in
> 512-byte units.
>
>Nothing in that says anything about "on disk".  So while this thing
>is sitting in memory on the TXG queue we should return the number of
>512 byte blocks used by the memory holding the data.
>I think that would be the more correct thing than exposing the
>fact this thing is setting in a write back cache to userland.

Can we compare the results of du with du -A?

Du will show compression savings, and -A wont

ZFS compresses between the write cache and the disk, so the final size may not 
be know for 5+ seconds
-- 
Allan Jude
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Problem with WITH_LLD_IS_LD

[Jochen Neumeister]

Hi current@

I have a PR 217519 because of devel/pear: When WITH_LLD_IS_LD is set, 
the stage phase for devel/pear breaks.


I found this in UPDATING:

The WITH_LLD_AS_LD / WITHOUT_LLD_AS_LD build knobs have been 
renamed
WITH_LLD_IS_LD / WITHOUT_LLD_IS_LD, for consistency with 
CLANG_IS_CC.


What possibilities are there to solve that problem?
I'm not sure this is a port problem or a current problem?


Regards

Jochen

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: effect of strip(1) on du(1)

[Alan Somers]

On Fri, Mar 3, 2017 at 7:11 AM, Rodney W. Grimes
 wrote:
> -- Start of PGP signed section.
> [ Charset ISO-8859-1 unsupported, converting... ]
>> On 2017-Mar-02 22:19:10 -0800, "Rodney W. Grimes" 
>>  wrote:
>> >> du(1) is using fts_read(3), which is based on the stat(2) information.
>> >> The OpenGroup defines st_blocksize as "Number of blocks allocated for
>> >> this object."  In the case of ZFS, a write(2) may return before any
>> >> blocks are actually allocated.  And thanks to compression, gang
>> ...
>> >My gut tells me that this is gona cause problems, is it ONLY
>> >the st_blocksize data that is incorrect then not such a big
>> >problem, or are we returning other meta data that is wrong?
>>
>> Note that it's st_blocks, not st_blocksize.
> Yes, I just ignore that digretion, as well as the digretion into fts_read
> being anything special about this, as it just ends up calling stat(2) in
> the end anyway.
>
>>
>> I did an experiment, writing a (roughly) 113MB file (some data I had
>> lying around), close()ing it and then stat()ing it in a loop.  This is
>> FreeBSD 10.3 with ZFS and lz4 compression.  Over the 26ms following the
>> close(), st_blocks gradually rose from 24169 to 51231.  It then stayed
>> stable until 4.968s after the close, when st_blocks again started
>> increasing until it stabilized after a total of 5.031s at 87483.  Based
>> on this, st_blocks reflects the actual number of blocks physically
>> written to disk.  None of the other fields in the struct stat vary.
>  ^^^
> Thank you for doing the proper regression test, that satisfies me that
> we dont have a lattent bug sitting here and infact what we have is
> exposure of the kernel caching, which I might be too thrilled about,
> is just how its gona have to be.
>
>>
>> The 5s delay is presumably the TXG delay (since this system is basically
>> unloaded).  I'm not sure why it writes roughly ? the data immediately
>> and the rest as part of the next TXG write.
>>
>> >My expectactions of executing a stat(2) call on a file would
>> >be that the data returned is valid and stable.  I think almost
>> >any program would expect that.
>>
>> I think a case could be made that st_blocks is a valid representation
>> of "the number of blocks allocated for this object" - with the number
>> increasing as the data is physically written to disk.  As for it being
>> stable, consider a (hypothetical) filesystem that can transparently
>> migrate data between different storage media, with different compression
>> algorithms etc (ZFS will be able to do this once the mythical block
>> rewrite code is written).
>
> I could counter argue that st_blocks is:
> st_blocks   The actual number of blocks allocated for the file in
>  512-byte units.
>
> Nothing in that says anything about "on disk".  So while this thing
> is sitting in memory on the TXG queue we should return the number of
> 512 byte blocks used by the memory holding the data.
> I think that would be the more correct thing than exposing the
> fact this thing is setting in a write back cache to userland.
>
> --
> Rod Grimes rgri...@freebsd.org

"Transparent" does not mean "undetectable".  For example, ZFS's
transparent compression will affect the st_blocks reported for a file.
I think the only sane use of st_blocks is to treat it as advisory.
I've seen a lot of bugs caused by programmers assuming a certain
mathematical relationship between the numbers presented by "df", "zfs
list", etc.

BTW, I've confirmed that ZFS on Illumos has the same behavior.  A
file's st_blocks doesn't stabilize until a few seconds after you write
it.

And it turns out that the fsync(1) doesn't work.  This suggests that
ZFS doesn't consider blocks in the ZIL when it reports st_blocks.

-Alan
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Boot failure - svn up from this morning

[Alex Deiter]

Hello Dexuan,

Thank you for the patch!
I’ll test it and let you know the result.

Thank you!

Alex Deiter
alex.dei...@gmail.com



> On 3 Mar 2017, at 14:49, Dexuan Cui  wrote:
> 
>> From: Alex Deiter
>> Sent: Friday, March 3, 2017 17:22
>> 
>> Hello,
>> The same issue with FreeBSD 12.0-CURRENT-r314563:
>> elf64_loadimage: could not read symbols - skipped!
>> 
>> I suspect regression after:
>> 
>> Revision 314547 - Directory Listing
>> Modified Thu Mar 2 07:25:50 2017 UTC (25 hours, 54 minutes ago) by dexuan
>> loader.efi: reduce the size of the staging area if necessary
> 
> Yes, I believe the issue is caused by the patch, which is supposed to PR 
> 211746:
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211746
> 
> Sorry for causing the issue to you, but I suspect the patch reveals a bug in 
> your
> host's firmware: the memory map reported by the host's firmware may be
> incorrect.
> 
> Can you please try the patch to dump the memory map?
> https://github.com/dcui/freebsd/commit/6094aac8ac9bddb24e3ac45493ac020c94029ce8.patch
> 
> And the patch will allow your host to boot.
> Note: there is a 10-second pause every time 5 lines are printed. This is to 
> make
> sure we have enough time to take a screenshot or photo. :-)
> 
> About how to apply the patch and build/install it:
> 'wget' the above patch, and 'cd' into your FREEBSD_SOURCE_ROOT/sys/boot/ and 
> run
> "patch -p3 < the_patch_name". If you have run 'make buildworld", just run 
> 'make' in the
> sys/boot/ directory and copy the new loader.efi into the boot folder, e.g. in 
> my side, I
> use
> cp -iv  /usr/obj/root/bsd.git/sys/boot/efi/loader/loader.efi /boot/loader.efi
> 
> Thanks,
> -- Dexuan

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CFLAGS for certain ports

[Julian Elischer]

On 2/3/17 8:58 pm, Dimitry Andric wrote:

On 2 Mar 2017, at 12:02, Mingo Rrubioer  wrote:

I would like to see how well FreeBSD does as a workstation OS in the
HPC world due to its stability and reliability, as well as LLVM/clang.
I would like to know if FreeBSD has something similar to Gentoo's
/etc/portage/make.conf file and /etc/portage/package.use/* files in
order to compile certain ports with certain compiler flags.

It doesn't, though it would certainly be nice to have something like it
at some point.  The current idiom is to put something similar to the
following in your /etc/make.conf:

.if ${.CURDIR:M/usr/ports/foo/bar}
CFLAGS+= [... flags for the foo/bar port ...]
.endif

.if ${.CURDIR:M/usr/ports/what/ever}
CFLAGS+= [... flags for the what/ever port ...]
.endif



Regarding LLVM/clang, I've been reading the documentation and found
these flags: -arch=, -march=, -mcpu=,
--target=, target-cpu . I'm not quite sure which
one would be the one to use. In case someone wants to know, my initial
play/test machine has this processor:

CPU: Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz (3600.11-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206d7  Family=0x6  Model=0x2d  Stepping=7

And I'm currently running: 11.0-RELEASE-p8.

So I imagine I should use something like CFLAGS+= -march=corei7-avx
-march=sandybridge -target-cpu. Is that correct?

Don't specify -march or -mcpu directly, but add the following line to
/etc/make.conf:

CPUTYPE?= native

This will take care of everything automatically.  See also make.conf(5).

-Dimitry

Many many 3rd party packages use a consistent set of variables fed 
into the automumble tools.
Only today I was pleasantly surprised to see the lftp port pick up my 
CFLAGS and LDFLAGS changes without me having to change anything in the 
ports themselves..


we should investigate what these defacto stndards are and document them


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Buildworld fails if WITHOUT_INET6=YES defined

[Oleg V. Nauman]

On Thursday 02 March 2017 22:40:05 Alex Deiter wrote:
> Hello,

 Hello,

> 
> Please apply patch from upstream:
> 
> https://github.com/the-tcpdump-group/libpcap/pull/541
> 
> Fix compilation if INET6 isn't defined.
> Addresses GitHub issue #541, but differently from the pull request (it
> defines gen_gateway() with a function prototype rather than using a
> pre-prototype-style definition).
> 
> https://github.com/the-tcpdump-group/libpcap/commit/470df104c6f55f6d6f390df7
> 448d8eb65c7642b9#diff-021c0dd9e9ed7100b9e31d8d95c930f2

 I can confirm that it fixes the issue.

> 
> Thank you!
> 
> Alex Deiter
> alex.dei...@gmail.com
> 
> > On 18 Feb 2017, at 00:09, Bryan Drewery  wrote:
> > 
> > On 2/17/2017 1:03 PM, Bryan Drewery wrote:
> >> On 2/16/2017 10:07 AM, Ngie Cooper (yaneurabeya) wrote:
>  On Feb 16, 2017, at 07:30, Oleg V. Nauman 
>  wrote:
>  
>  cc -target i386-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -
>  B/usr/obj/usr/src/tmp/usr/bin  -O2 -pipe -march=core2  -DHAVE_CONFIG_H
>  -
>  I/usr/src/lib/libpcap -I/usr/obj/usr/src/lib/libpcap -
>  D_U_="__attribute__((unused))" -DHAVE_SNPRINTF -DHAVE_VSNPRINTF -
>  DBUILDING_PCAP -DHAVE_NET_PFVAR_H -I/usr/src/contrib/libpcap -MD  -
>  MF.depend.fad-getad.o -MTfad-getad.o -std=gnu99
>  -fstack-protector-strong -Wno- pointer-sign -Wno-empty-body
>  -Wno-string-plus-int -Wno-unused-const-variable -
>  Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality
>  -Wno- unused-function -Wno-enum-conversion -Wno-unused-local-typedef
>  -Wno-switch - Wno-switch-enum -Wno-knr-promoted-parameter
>  -Wno-parentheses  -Qunused- arguments  -c
>  /usr/src/contrib/libpcap/fad-getad.c -o fad-getad.o
>  cc -target i386-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -
>  B/usr/obj/usr/src/tmp/usr/bin  -O2 -pipe -march=core2  -DHAVE_CONFIG_H
>  -
>  I/usr/src/lib/libpcap -I/usr/obj/usr/src/lib/libpcap -
>  D_U_="__attribute__((unused))" -DHAVE_SNPRINTF -DHAVE_VSNPRINTF -
>  DBUILDING_PCAP -DHAVE_NET_PFVAR_H -I/usr/src/contrib/libpcap -MD  -
>  MF.depend.gencode.o -MTgencode.o -std=gnu99 -fstack-protector-strong
>  -Wno-
>  pointer-sign -Wno-empty-body -Wno-string-plus-int
>  -Wno-unused-const-variable - Wno-tautological-compare
>  -Wno-unused-value -Wno-parentheses-equality -Wno- unused-function
>  -Wno-enum-conversion -Wno-unused-local-typedef -Wno-switch -
>  Wno-switch-enum -Wno-knr-promoted-parameter -Wno-parentheses 
>  -Qunused- arguments  -c /usr/src/contrib/libpcap/gencode.c -o
>  gencode.o
>  /usr/src/contrib/libpcap/gencode.c:695:9: error: no member named 'ai'
>  in
>  'struct _compiler_state'
>  
>    cstate.ai = NULL;
>    ~~ ^
>  
>  /usr/src/contrib/libpcap/gencode.c:4916:13: error: use of undeclared
>  identifier 'cstate'
>  
>    bpf_error(cstate, "direction applied to 'gateway'");
>    
>  ^
>  
>  /usr/src/contrib/libpcap/gencode.c:4923:11: error: use of undeclared
>  identifier 'cstate'
>  
>    switch (cstate->linktype) {
>    
>    ^
>  
>  /usr/src/contrib/libpcap/gencode.c:4961:17: error: use of undeclared
>  identifier 'cstate'
>  
>    b1 = gen_host(cstate, **alist++, 0x, proto, Q_OR,
>  
>  Q_HOST);
>  
>  ^
>  
>  /usr/src/contrib/libpcap/gencode.c:4963:19: error: use of undeclared
>  identifier 'cstate'
>  
>    tmp = gen_host(cstate, **alist++, 0x,
>    proto,
>  
>  Q_OR,
>  
>   ^
>  
>  /usr/src/contrib/libpcap/gencode.c:4972:12: error: use of undeclared
>  identifier 'cstate'
>  
>    bpf_error(cstate, "illegal modifier of 'gateway'");
>    
>  ^
>  
>  6 errors generated.
>  *** Error code 1
>  
>  Stop.
>  make[5]: stopped in /usr/src/lib/libpcap
>  *** Error code 1
> >>> 
> >>> CCing Xin, who did the libpcap upgrade.
> >>> -Ngie
> >> 
> >> I get an error even with INET6 building grammar.o:
> >>> /root/git/freebsd/contrib/libpcap/grammar.y:348:15: error: use of
> >>> undeclared identifier 'cstate'; did you mean 'cstat'?>>> 
> >>>finish_parse(cstate, yystack.l_mark[0].blk.b);
> >>>
> >>> ^~
> >>> cstat
> >>> 
> >>> grammar.c:1291:1: note: 'cstat' declared here
> >>> YYPARSE_DECL()
> >>> ^
> > 
> > Ignore me, my yacc is just outdated.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 

Re: svn commit: r313268 - head/sys/kern [through -r313271 for atomic_fcmpset use and later: fails on PowerMac G5 "Quad Core"; -r313266 works]

[Rodney W. Grimes]

> On 2017-Mar-2, at 7:19 AM, Steve Kargl  
> wrote:
> 
> On Thu, Mar 02, 2017 at 01:10:21PM +0100, Mateusz Guzik wrote:
> > On Wed, Mar 01, 2017 at 09:45:07AM -0800, Mark Millard wrote:
> >> 
> >>> Summary of the transition interval:
> >>> 
> >>> So for powerpc64 (and powerpc?) It is a good
> >>> idea to avoid anything that is after -r313254
> >>> and before -r314474 in head. (Would this be
> >>> appropriate for a UPDATING notice given its
> >>> span?)
> >>> 
> >>> There may be other architectures that might have
> >>> a similar status(?): the last fixes involved were
> >>> not in Machine Dependent code. (Some architectures
> >>> are apparently insensitive to the errors, such as
> >>> amd64).
> >>> 
> >> 
> >> When following current you are expected to be on the newest revision,
> >> so I don't think mentioning interim broken releases makes much sense.
> >> 
> > 
> > Documenting the range may aid those bisecting src/ to find a bug. 
> > How is one to know that anything in the range that Mark points
> > out should be skipped on powerpc64?
> > 
> > -- 
> > Steve
> 
> I have tested with a TARGET_ARCH=powerpc -r314473 build and
> its kernel version has locking problems like
> TARGET_ARCH=powerpc64 does for that version.
> 
> [Note: This was run on a PowerMac G5 so-called "Quad Core"
> so most of the memory was ignored.]
> 
> Both TARGET_ARCH=powerpc64 and TARGET_ARCH=powerpc need -r314474
> or later as of the new locking.
> 
> I've not explicitly tested other architectures. As I remember
> armv6/v7 are classified as having some from of a weak memory
> model compared to the likes of amd64. If so armv6/v7 might be
> candidates for having problems. There might be other candidates.

I also had locking issues on amd64 around this build time that
sent me down a week long rabbit hole chasing what I thought was
a bug in the new AMD/IOMMU code.  IMHO if we can at least
flag prior snapshot builds as "Broken for reason X" it might
save someone some time and time is a one way depleting resource
usually worth saving if possible.

If needed I can dig out the specifc build.  Oh, nvm, let me
just do that, it was r309302.  This revision I beleive is
a november snapshot.  It has kernel panics due to spinlock
timeout and sparatic deadlock that is undetected.


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: effect of strip(1) on du(1)

[Rodney W. Grimes]

-- Start of PGP signed section.
[ Charset ISO-8859-1 unsupported, converting... ]
> On 2017-Mar-02 22:19:10 -0800, "Rodney W. Grimes" 
>  wrote:
> >> du(1) is using fts_read(3), which is based on the stat(2) information.
> >> The OpenGroup defines st_blocksize as "Number of blocks allocated for
> >> this object."  In the case of ZFS, a write(2) may return before any
> >> blocks are actually allocated.  And thanks to compression, gang
> ...
> >My gut tells me that this is gona cause problems, is it ONLY
> >the st_blocksize data that is incorrect then not such a big
> >problem, or are we returning other meta data that is wrong?
> 
> Note that it's st_blocks, not st_blocksize.
Yes, I just ignore that digretion, as well as the digretion into fts_read
being anything special about this, as it just ends up calling stat(2) in
the end anyway.

> 
> I did an experiment, writing a (roughly) 113MB file (some data I had
> lying around), close()ing it and then stat()ing it in a loop.  This is
> FreeBSD 10.3 with ZFS and lz4 compression.  Over the 26ms following the
> close(), st_blocks gradually rose from 24169 to 51231.  It then stayed
> stable until 4.968s after the close, when st_blocks again started
> increasing until it stabilized after a total of 5.031s at 87483.  Based
> on this, st_blocks reflects the actual number of blocks physically
> written to disk.  None of the other fields in the struct stat vary.
 ^^^
Thank you for doing the proper regression test, that satisfies me that
we dont have a lattent bug sitting here and infact what we have is
exposure of the kernel caching, which I might be too thrilled about,
is just how its gona have to be.

> 
> The 5s delay is presumably the TXG delay (since this system is basically
> unloaded).  I'm not sure why it writes roughly ? the data immediately
> and the rest as part of the next TXG write.
> 
> >My expectactions of executing a stat(2) call on a file would
> >be that the data returned is valid and stable.  I think almost
> >any program would expect that.
> 
> I think a case could be made that st_blocks is a valid representation
> of "the number of blocks allocated for this object" - with the number
> increasing as the data is physically written to disk.  As for it being
> stable, consider a (hypothetical) filesystem that can transparently
> migrate data between different storage media, with different compression
> algorithms etc (ZFS will be able to do this once the mythical block
> rewrite code is written).

I could counter argue that st_blocks is:
st_blocks   The actual number of blocks allocated for the file in
 512-byte units.

Nothing in that says anything about "on disk".  So while this thing
is sitting in memory on the TXG queue we should return the number of
512 byte blocks used by the memory holding the data.
I think that would be the more correct thing than exposing the
fact this thing is setting in a write back cache to userland.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Boot failure - svn up from this morning

[Michael Tuexen]

> On 3 Mar 2017, at 12:59, Dexuan Cui  wrote:
> 
>> From: Dexuan Cui
>> Sent: Friday, March 3, 2017 19:50
>>> From: Alex Deiter
>>> Sent: Friday, March 3, 2017 17:22
>>> Hello,
>>> The same issue with FreeBSD 12.0-CURRENT-r314563:
>>> elf64_loadimage: could not read symbols - skipped!
>>> 
>>> I suspect regression after:
>>> 
>>> Revision 314547 - Directory Listing
>>> Modified Thu Mar 2 07:25:50 2017 UTC (25 hours, 54 minutes ago) by dexuan
>>> loader.efi: reduce the size of the staging area if necessary
>> 
>> Yes, I believe the issue is caused by the patch, which is supposed to PR 
>> 211746:
>> ...
>> Can you please try the patch to dump the memory map?
>> ...
> 
> BTW, I understand it's really annoying to boot the host first...
> I'm really sorry for this.
> 
> I suppose you're able to build or find a good 'loader.efi' binary on another 
> host,
> and  then manage to replace the bad 'loader.efi' on the host broken by me. :-)
This problem also occurred on a Dell R430...

Best regards
Michael
> 
> Thanks,
> -- Dexuan
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



smime.p7s
Description: S/MIME cryptographic signature

RE: Boot failure - svn up from this morning

[Dexuan Cui]

> From: Michael Tuexen
> Sent: Friday, March 3, 2017 21:30
> > BTW, I understand it's really annoying to boot the host first...
> > I'm really sorry for this.
> >
> > I suppose you're able to build or find a good 'loader.efi' binary on 
> > another host,
> > and  then manage to replace the bad 'loader.efi' on the host broken by me. 
> > :-)
> This problem also occurred on a Dell R430...
> 
> Best regards
> Michael

Can you please use the patch to dump the EFI memory map:
https://lists.freebsd.org/pipermail/freebsd-current/2017-March/064979.html

Let's get this fixed ASAP.

Thanks,
-- Dexuan
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: effect of strip(1) on du(1)

[Rodney W. Grimes]

> On Thu, Mar 2, 2017 at 6:12 PM, Ngie Cooper  wrote:
> > On Thu, Mar 2, 2017 at 4:31 PM, Rodney W. Grimes
> >  wrote:
> > ...
> >> Even if that is the case file system cache effects should NOT be
> >> visible to a userland process.   This is NOT as if your running
> >> 2 different processing beating on a file.  Your test cases are
> >> serialially syncronous shell invoked commands seperated with
> >> && the results should be exact and predictable.
> >>
> >> When strip returns the operation from the userland perspecive
> >> is completed and any and all processeses started after that
> >> should have the view of the completed strip command.
> >>
> >> This IS a bug.
> >
> > Would the same statement necessarily apply if the filesystem was
> > writing things asynchronously to the backing storage?
> > Thanks,
> > -Ngie
> 
> du(1) is using fts_read(3), which is based on the stat(2) information.
> The OpenGroup defines st_blocksize as "Number of blocks allocated for
> this object."  In the case of ZFS, a write(2) may return before any
> blocks are actually allocated.  And thanks to compression, gang
> blocks, and deduplication, at this point it's not even possible for
> ZFS to know how many blocks it will need to allocate.  I think
> st_blocksize should be interpreted as a "best effort" output.  Just
> like df(1), you can't rely on du's output to be mathematically precise
> in any way.  I certainly don't see any way to fix it besides doing
> something like an fsync(2) before getting stat information, and we
> certainly don't want to do that.

My gut tells me that this is gona cause problems, is it ONLY
the st_blocksize data that is incorrect then not such a big
problem, or are we returning other meta data that is wrong?

Waving hands over this report as async write behind meta
data issues is not making sure we dont have a more serious
problem.

My expectactions of executing a stat(2) call on a file would
be that the data returned is valid and stable.  I think almost
any program would expect that.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: effect of strip(1) on du(1)

[Rodney W. Grimes]

[ Charset UTF-8 unsupported, converting... ]
> On Thu, Mar 2, 2017 at 4:31 PM, Rodney W. Grimes
>  wrote:
> ...
> > Even if that is the case file system cache effects should NOT be
> > visible to a userland process.   This is NOT as if your running
> > 2 different processing beating on a file.  Your test cases are
> > serialially syncronous shell invoked commands seperated with
> > && the results should be exact and predictable.
> >
> > When strip returns the operation from the userland perspecive
> > is completed and any and all processeses started after that
> > should have the view of the completed strip command.
> >
> > This IS a bug.
> 
> Would the same statement necessarily apply if the filesystem was
> writing things asynchronously to the backing storage?

Caching should^h^h^h^hshall be transparent to a userland process.
Are you actually advocating that a userland process should
be able to see that zfs is write caching meta data?

The strip(1) command has completed and exited, pola dictates
that anything I asked strip(1) to do be reflected in all commands
or system calls executed after it.

Anything else would be a bug.

> Thanks,
> -Ngie

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

RE: Boot failure - svn up from this morning

[Dexuan Cui]

> From: Dexuan Cui
> Sent: Friday, March 3, 2017 19:50
> > From: Alex Deiter
> > Sent: Friday, March 3, 2017 17:22
> > Hello,
> > The same issue with FreeBSD 12.0-CURRENT-r314563:
> > elf64_loadimage: could not read symbols - skipped!
> >
> > I suspect regression after:
> >
> > Revision 314547 - Directory Listing
> > Modified Thu Mar 2 07:25:50 2017 UTC (25 hours, 54 minutes ago) by dexuan
> > loader.efi: reduce the size of the staging area if necessary
> 
> Yes, I believe the issue is caused by the patch, which is supposed to PR 
> 211746:
> ...
> Can you please try the patch to dump the memory map?
> ...

BTW, I understand it's really annoying to boot the host first...
I'm really sorry for this.

I suppose you're able to build or find a good 'loader.efi' binary on another 
host,
and  then manage to replace the bad 'loader.efi' on the host broken by me. :-)

Thanks,
-- Dexuan
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

RE: Boot failure - svn up from this morning

[Dexuan Cui]

> From: Alex Deiter
> Sent: Friday, March 3, 2017 17:22
>
> Hello,
> The same issue with FreeBSD 12.0-CURRENT-r314563:
> elf64_loadimage: could not read symbols - skipped!
>
> I suspect regression after:
>
> Revision 314547 - Directory Listing
> Modified Thu Mar 2 07:25:50 2017 UTC (25 hours, 54 minutes ago) by dexuan
> loader.efi: reduce the size of the staging area if necessary

Yes, I believe the issue is caused by the patch, which is supposed to PR 211746:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211746

Sorry for causing the issue to you, but I suspect the patch reveals a bug in 
your
host's firmware: the memory map reported by the host's firmware may be
incorrect.

Can you please try the patch to dump the memory map?
https://github.com/dcui/freebsd/commit/6094aac8ac9bddb24e3ac45493ac020c94029ce8.patch

And the patch will allow your host to boot.
Note: there is a 10-second pause every time 5 lines are printed. This is to make
sure we have enough time to take a screenshot or photo. :-)

About how to apply the patch and build/install it:
'wget' the above patch, and 'cd' into your FREEBSD_SOURCE_ROOT/sys/boot/ and run
"patch -p3 < the_patch_name". If you have run 'make buildworld", just run 
'make' in the
sys/boot/ directory and copy the new loader.efi into the boot folder, e.g. in 
my side, I
use
cp -iv  /usr/obj/root/bsd.git/sys/boot/efi/loader/loader.efi /boot/loader.efi

Thanks,
-- Dexuan
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r313268 - head/sys/kern [through -r313271 for atomic_fcmpset use and later: fails on PowerMac G5 "Quad Core"; -r313266 works]

[Mark Millard]

On 2017-Mar-2, at 7:19 AM, Steve Kargl  
wrote:

On Thu, Mar 02, 2017 at 01:10:21PM +0100, Mateusz Guzik wrote:
> On Wed, Mar 01, 2017 at 09:45:07AM -0800, Mark Millard wrote:
>> 
>>> Summary of the transition interval:
>>> 
>>> So for powerpc64 (and powerpc?) It is a good
>>> idea to avoid anything that is after -r313254
>>> and before -r314474 in head. (Would this be
>>> appropriate for a UPDATING notice given its
>>> span?)
>>> 
>>> There may be other architectures that might have
>>> a similar status(?): the last fixes involved were
>>> not in Machine Dependent code. (Some architectures
>>> are apparently insensitive to the errors, such as
>>> amd64).
>>> 
>> 
>> When following current you are expected to be on the newest revision,
>> so I don't think mentioning interim broken releases makes much sense.
>> 
> 
> Documenting the range may aid those bisecting src/ to find a bug. 
> How is one to know that anything in the range that Mark points
> out should be skipped on powerpc64?
> 
> -- 
> Steve

I have tested with a TARGET_ARCH=powerpc -r314473 build and
its kernel version has locking problems like
TARGET_ARCH=powerpc64 does for that version.

[Note: This was run on a PowerMac G5 so-called "Quad Core"
so most of the memory was ignored.]

Both TARGET_ARCH=powerpc64 and TARGET_ARCH=powerpc need -r314474
or later as of the new locking.

I've not explicitly tested other architectures. As I remember
armv6/v7 are classified as having some from of a weak memory
model compared to the likes of amd64. If so armv6/v7 might be
candidates for having problems. There might be other candidates.


===
Mark Millard
markmi at dsl-only.net

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Boot failure - svn up from this morning

[Alex Deiter]

Hello,

The same issue with FreeBSD 12.0-CURRENT-r314563:

elf64_loadimage: could not read symbols - skipped!

http://picpaste.com/IMG_1764-vfZl1l5o.JPG

I suspect regression after: 

Revision 314547 - Directory Listing 
Modified Thu Mar 2 07:25:50 2017 UTC (25 hours, 54 minutes ago) by dexuan
loader.efi: reduce the size of the staging area if necessary

The loader assumes physical memory in [2MB, 2MB + EFI_STAGING_SIZE)
is Conventional Memory, but actually it may not, e.g. in the case
of Hyper-V Generation-2 VM (i.e. UEFI VM) running on Windows
Server 2012 R2 host, there is a BootServiceData memory block at
the address 47.449MB and the memory is not writable.

Without the patch, the loader will crash in efi_copy_finish():
see PR 211746.

The patch verifies the end of the staging area, and reduces its
size if necessary. This way, the loader will not try to write into
the BootServiceData memory any longer.

Thank Marcel Moolenaar for helping me on this issue!

The patch also allocates the staging area in the first 1GB memory.
See the comment in the patch for this.

PR: 211746
Reviewed by:marcel, kib, sephe
Approved by:sephe (mentor)
MFC after:  2 weeks
Sponsored by:   Microsoft
Differential Revision:  
https://reviews.freebsd.org/D9686

Alex Deiter
alex.dei...@gmail.com



> On 3 Mar 2017, at 09:20, Chris H  wrote:
> 
> On Thu, 2 Mar 2017 21:32:39 -0800 "Ngie Cooper (yaneurabeya)"
>  wrote
> 
>>> On Mar 2, 2017, at 21:16, Chris H  wrote:
>>> 
>>> On Thu, 2 Mar 2017 21:56:38 -0500 (EST) AN  wrote
>>> 
 Hi:
 
 I'm having a major problem after updating a 12-current machine today.
 After buildworld/kernel/install cycle on reboot I'm getting the following
 failure:
 
 "/boot/kernel/kernel text=0xb7716f data=0x100548+0x398358
 elf64_loadimage: read failed
 can't load file '/boot/kernel/kernel': input/output error
 can't load file '/boot/kernel/kernel': input/output error
 
 OK boot kernel.old
 elf64_loadimage: read failed
 can't load file '/boot/kernel.old/kernel': input/output error
 can't load file '/boot/kernel.old/kernel': input/output error"
 
 I have never experienced this failure before, and don't know how to
 proceed.  Any help recovering from this would be greatly appreciated.
 Thanks in advance.
>>> 
>>> I don't suppose you could post the output of
>>> uname -a
>>> 
>>> and maybe a copy of dmesg(8) could you?
>> 
>> That would be good, but I don’t know if it’s possible (the OP is noting
>> that the boot is broken when executing loader(8)..). -Ngie
> 
> Indeed. But that doesn't stop the OP from booting from the
> install media. ;-) :-)
> 
> --Chris
> 
> 
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: effect of strip(1) on du(1)

[Peter Jeremy]

On 2017-Mar-02 22:19:10 -0800, "Rodney W. Grimes" 
 wrote:
>> du(1) is using fts_read(3), which is based on the stat(2) information.
>> The OpenGroup defines st_blocksize as "Number of blocks allocated for
>> this object."  In the case of ZFS, a write(2) may return before any
>> blocks are actually allocated.  And thanks to compression, gang
...
>My gut tells me that this is gona cause problems, is it ONLY
>the st_blocksize data that is incorrect then not such a big
>problem, or are we returning other meta data that is wrong?

Note that it's st_blocks, not st_blocksize.

I did an experiment, writing a (roughly) 113MB file (some data I had
lying around), close()ing it and then stat()ing it in a loop.  This is
FreeBSD 10.3 with ZFS and lz4 compression.  Over the 26ms following the
close(), st_blocks gradually rose from 24169 to 51231.  It then stayed
stable until 4.968s after the close, when st_blocks again started
increasing until it stabilized after a total of 5.031s at 87483.  Based
on this, st_blocks reflects the actual number of blocks physically
written to disk.  None of the other fields in the struct stat vary.

The 5s delay is presumably the TXG delay (since this system is basically
unloaded).  I'm not sure why it writes roughly ½ the data immediately
and the rest as part of the next TXG write.

>My expectactions of executing a stat(2) call on a file would
>be that the data returned is valid and stable.  I think almost
>any program would expect that.

I think a case could be made that st_blocks is a valid representation
of "the number of blocks allocated for this object" - with the number
increasing as the data is physically written to disk.  As for it being
stable, consider a (hypothetical) filesystem that can transparently
migrate data between different storage media, with different compression
algorithms etc (ZFS will be able to do this once the mythical block
rewrite code is written).

-- 
Peter Jeremy


signature.asc
Description: PGP signature

Re: effect of strip(1) on du(1)

[Alfred Perlstein]

On 3/2/17 5:30 PM, Alan Somers wrote:

On Thu, Mar 2, 2017 at 6:12 PM, Ngie Cooper  wrote:

On Thu, Mar 2, 2017 at 4:31 PM, Rodney W. Grimes
 wrote:
...

Even if that is the case file system cache effects should NOT be
visible to a userland process.   This is NOT as if your running
2 different processing beating on a file.  Your test cases are
serialially syncronous shell invoked commands seperated with
&& the results should be exact and predictable.

When strip returns the operation from the userland perspecive
is completed and any and all processeses started after that
should have the view of the completed strip command.

This IS a bug.

Would the same statement necessarily apply if the filesystem was
writing things asynchronously to the backing storage?
Thanks,
-Ngie

du(1) is using fts_read(3), which is based on the stat(2) information.
The OpenGroup defines st_blocksize as "Number of blocks allocated for
this object."  In the case of ZFS, a write(2) may return before any
blocks are actually allocated.  And thanks to compression, gang
blocks, and deduplication, at this point it's not even possible for
ZFS to know how many blocks it will need to allocate.  I think
st_blocksize should be interpreted as a "best effort" output.  Just
like df(1), you can't rely on du's output to be mathematically precise
in any way.  I certainly don't see any way to fix it besides doing
something like an fsync(2) before getting stat information, and we
certainly don't want to do that.



Try adding an fsync(1) to the file before running du(1) on it.

-Alfred
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"