On May 2, 2024, at 13:42, Mark Millard wrote:
> On Apr 30, 2024, at 19:08, Mark Millard wrote:
>
>> On Apr 28, 2024, at 18:48, Mark Millard wrote:
>>
>>> I've modified my local poudriere-devel to have Success and Failure lines
>>> also
>>> report the tmpfs size at that point. Using, say, script to log the output to
>>> a file allows later sorting and listing of the TMPFS usage filgures. (The
>>> context is an amd64 one that has the resources to do a full bulk -a with
>>> USE_TMPFS=all invovled.) An example (from an in-progress bulk -a that has a
>>> long way to go):
>>>
>>> # grep TMPFS: ~/mmjnk-bulk-a-output.txt | sort -n -r -k11
>>>
>>> [00:36:37] [18] [00:26:53] Finished print/texlive-docs |
>>> texlive-docs-20230313: Success ending TMPFS: 10.67 GiB
>>> [01:04:41] [07] [00:06:58] Finished net-mgmt/telegraf | telegraf-1.30.1_1:
>>> Success ending TMPFS: 10.52 GiB
>>> [01:03:32] [25] [00:06:09] Finished security/trivy | trivy-0.50.1_1:
>>> Success ending TMPFS: 10.10 GiB
>>> . . .
>>> [01:15:56] [20] [00:00:54] Finished databases/pg_tileserv |
>>> pg_tileserv-1.0.9_12: Failed: build TMPFS: 2.61 GiB
>>> . . .
>>>
>>> Note that the design is for sort with -k11 to work for Success and
>>> for Failure. (This is why "ending" is in place for Success.) I
>>> choose to use poudriere -N (no coloring) for this kind of activity.
>>>
>>> This helps for figuring out what all is appropriate for listing in
>>> TMPFS_BLACKLIST for a poudriere-devel configuration to avoid tmpfs
>>> competing too much for RAM+SWAP. (But approraite free file system
>>> space is needed.)
>>>
>>>
>>> "<" below is what is new, ">" is what was original, in
>>> /usr/local/share/poudriere/common.sh :
>>>
>>> 5928,5934d5927
>>> < tmpfs_at_end="$(env BLOCKSIZE=512 df -t tmpfs \
>>> < ${MASTERMNTROOT}/${MY_JOBID}/ \
>>> < ${MASTERMNTROOT}/${MY_JOBID}/.p/ \
>>> < ${MASTERMNTROOT}/${MY_JOBID}/usr/local/ \
>>> < 2>/dev/null | tail -3 \
>>> < | awk '{ tmpfs_use += $3; } END { printf "%s %.2f %s", "TMPFS:",
>>> tmpfs_use*512/(1024**3), "GiB" }')"
>>> <
>>> 5942c5935
>>> <"Success${COLOR_RESET} ending ${tmpfs_at_end}"
>>> ---
"Success"
>>> 5968c5961
>>> <"Failed: ${COLOR_PHASE}${failed_phase}${COLOR_RESET} ${tmpfs_at_end}"
>>> ---
"Failed: ${COLOR_PHASE}${failed_phase}"
>>>
>>>
>>> The form of use that I've done also involves (over?) use of
>>> MUTUALLY_EXCLUSIVE_BUILD_PACKAGES . It is not as good of data
>>> for this other use, but the same .txt file can be processed
>>> with:
>>>
>>> # grep TMPFS: ~/mmjnk-bulk-a-output.txt | sort -r -k3 | more
>>> [01:42:09] [04] [00:48:16] Finished lang/erlang-runtime21 |
>>> erlang-runtime21-21.3.8.24_3: Success ending TMPFS: 1.92 GiB
>>> [01:38:39] [28] [00:44:41] Finished lang/erlang-runtime22 |
>>> erlang-runtime22-22.3.4.27: Success ending TMPFS: 1.92 GiB
>>> [01:05:41] [02] [00:34:54] Finished lang/erlang-runtime26 |
>>> erlang-runtime26-26.2.4: Success ending TMPFS: 2.02 GiB
>>> . . .
>>>
>>> to find longer running package builds. This is subject to
>>> significant variation based on what other builders are
>>> running in parallel at the time and what sort of load
>>> averages are involved over period in question. The
>>> MUTUALLY_EXCLUSIVE_BUILD_PACKAGES that I've used will
>>> limit that to some extent. But the result is comparisons
>>> of some builds that have no activity in parallel by other
>>> builders vs. builds that have extensive parallel activity
>>> by other builders (of not-huge packages).
>>>
>>> Note: In modern times the [1D: notation and the like for
>>> what the -k3 compares are not well placed in the overall
>>> list compared to the likes of, say, [20: . The day vs.
>>> hour comparison is not a straight forward thing to sort
>>> on.
>>>
>>> Hopefully in a few days I'll be able to list off example
>>> top tmpfs usage for USE_TMPFS=all and top build times as
>>> well (such as they are).
>>>
>>> Note:
>>> This is from my personal environment. I've not tried to
>>> simulate how FreeBSD's official package builders are set
>>> up to operate.
>>
>> I've decided that the "build times" list is not reasonable
>> for use: too much variability of context.
>>
>> But here is a list of 148 or so of the bigger USE_TMPFS=all
>> tmpfs-usage builds from a poudriere-devel "-N bulk -c -a"
>> run. It covers the > 7.66 GiByte tmpfs examples for how I
>> build.
>>
>> # grep TMPFS: mmjnk-bulk-a-output.txt | sort -n -r -k11 | head -84
>> [1D:10:04:32] [25] [02:14:20] Finished www/chromium |
>> chromium-124.0.6367.60: Success ending TMPFS: 31.76 GiB
> . . .
>> [1D:11:05:43] [04] [00:16:26] Finished astro/kstars | kstars-3.6.6_1,1:
>> Success ending TMPFS: 7.69 GiB
>> . . .
>
> Note: I had not put ttk into the TMPFS_BLACKLIST as I had intended.
>
>> For reference: Queued: 34535 Built: 33990 Failed: 159 Skipped: 100
>> Ignored: 286 Fetched: 0
>>
>> After rebooting, I'm going to re-run a "-N bulk -c -a" based on
>> TMPFS_BLACKLIST having a hopefully
