Re: unlock mmap(2) for anonymous mappings
Klemens Nanni: > > > Now this is clearly a "slow" path. I don't think there is any reason > > > not to put all the code in that if (uvw_wxabort) block under the > > > kernel lock. For now I think making access to ps_wxcounter atomic is > > > just too fine grained. > > > > Right. Lock the whole block. > > Thanks everyone, here's the combined diff for that. -snip- FWIW, I ran an amd64 package bulk build with this. No problems. -- Christian "naddy" Weisgerber na...@mips.inka.de
Re: rpki-client introduce validated cache
On Thu, Jan 13, 2022 at 05:05:57PM +0100, Claudio Jeker wrote:
> This diff adds a new cache subdir called "valid". This is the place where
> all verified and good files are stored after a run. It makes -n work a lot
> better since -n will now only look at what's inside "valid" and ignore
> "rsync" and "rrdp".
>
> The trust anchors are still stored in "ta" even if valid.
> The rsync repo will only hold temporary files and be empty otherwise.
> The rrdp repo still may contain some superfluous files that people
> included in their snapshots. Not keeping them could result in extra
> snapshot downloads since delta updates could refer to these files.
> Since there is a valid repo, rrdp no longer needs an additional temp
> directory for its sync.
>
> With this rsync will use the --compare-dest feature and use the valid
> cache as a base.
>
> The file cleanup at the end got more complex. There is now
> repo_move_valid() and repo_cleanup_rrdp() to do two initial steps of
> cleanup before the tree traversal starts. The fts function now uses
> some fts features to find a) empty directories and b) superfluous files.
>
> I think the code changes outside of repo.c should be straight forward.
>
> To the rpki connoisseur, this currently only implements a simple newest
> file wins startegy. The handling of .mft and their serial number will
> follow once this is in.
I have made a first pass over this. I will need to sleep on this and
think through the repo.c changes in more detail with a fresh head.
Couple leaks and a few stylistic remarks noted inline.
> --
> :wq Claudio
>
> ? obj
> Index: extern.h
> ===
> RCS file: /cvs/src/usr.sbin/rpki-client/extern.h,v
> retrieving revision 1.103
> diff -u -p -r1.103 extern.h
> --- extern.h 13 Jan 2022 13:46:03 - 1.103
> +++ extern.h 13 Jan 2022 15:50:40 -
> @@ -341,7 +341,7 @@ enum publish_type {
> struct entity {
> TAILQ_ENTRY(entity) entries;
> char*path; /* path relative to repository */
> - char*file; /* filename */
> + char*file; /* filename or valid repo path */
> unsigned char *data; /* optional data blob */
> size_t datasz;/* length of optional data blob */
> unsigned int repoid;/* repository identifier */
> @@ -380,6 +380,7 @@ struct stats {
> size_t vrps; /* total number of vrps */
> size_t uniqs; /* number of unique vrps */
> size_t del_files; /* number of files removed in cleanup */
> + size_t extra_files; /* number of superfluous files */
> size_t del_dirs; /* number of directories removed in cleanup */
> size_t brks; /* number of BGPsec Router Key (BRK) certificates */
> struct timeval elapsed_time;
> @@ -506,7 +507,7 @@ void rrdp_clear(unsigned int);
> void rrdp_save_state(unsigned int, struct rrdp_session *);
> int rrdp_handle_file(unsigned int, enum publish_type, char *,
> char *, size_t, char *, size_t);
> -char *repo_basedir(const struct repo *);
> +char *repo_basedir(const struct repo *, int);
> unsigned int repo_id(const struct repo *);
> const char *repo_uri(const struct repo *);
> struct repo *ta_lookup(int, struct tal *);
> @@ -520,7 +521,8 @@ void rsync_finish(unsigned int, int);
> void http_finish(unsigned int, enum http_result, const char *);
> void rrdp_finish(unsigned int, int);
>
> -void rsync_fetch(unsigned int, const char *, const char *);
> +void rsync_fetch(unsigned int, const char *, const char *,
> + const char *);
> void http_fetch(unsigned int, const char *, const char *, int);
> void rrdp_fetch(unsigned int, const char *, const char *,
> struct rrdp_session *);
> Index: main.c
> ===
> RCS file: /cvs/src/usr.sbin/rpki-client/main.c,v
> retrieving revision 1.175
> diff -u -p -r1.175 main.c
> --- main.c13 Jan 2022 13:18:41 - 1.175
> +++ main.c13 Jan 2022 15:50:40 -
> @@ -151,17 +151,18 @@ entity_write_repo(struct repo *rp)
> struct ibuf *b;
> enum rtype type = RTYPE_REPO;
> unsigned int repoid;
> - char *path;
> + char *path, *altpath;
> int talid = 0;
>
> repoid = repo_id(rp);
> - path = repo_basedir(rp);
> + path = repo_basedir(rp, 0);
> + altpath = repo_basedir(rp, 1);
> b = io_new_buffer();
> io_simple_buffer(b, &type, sizeof(type));
> io_simple_buffer(b, &repoid, sizeof(repoid));
> io_simple_buffer(b, &talid, sizeof(talid));
> io_str_buffer(b, path);
> - io_str_buffer(b, NULL);
> + io_str_buffer(b, altpath);
> io_buf_buffer(b, NULL, 0);
> io_close_buffer(&procq, b);
> free(path);
Don't leak alt
Re: ix(4): enable TCPv6/UDPv6 cksum offloading
On Thu, Jan 13, 2022 at 12:43:57PM -0700, Theo de Raadt wrote:
> > - m_getptr() returns the correct mbuf and offset to the header. I
> > think we can assume that a single IPv6 header, that our stack has
> > created, is in contiguous memory. The IPv4 case just above makes
> > the same assumption.
>
> And if the assumption is not met due to interations between various
> tunnel or encapsulation layers... then panic the kernel? That is OK?
It was OK for IPv4 during more than 6 years.
1.118(mikeb20-Mar-15): m = m_getptr(mp, ehdrlen,
&ipoff);
1.118(mikeb20-Mar-15): KASSERT(m != NULL && m->m_len -
ipoff >= sizeof(*ip));
1.118(mikeb20-Mar-15): ip = (struct ip *)(m->m_data +
ipoff);
But to be safe we can use a combination of simple access in the hot
path and m_copydata() in rare cases.
Untested and IPv4-only. If you like it, Jan can do it with IPv6
tomorrow.
bluhm
Index: dev/pci/if_ix.c
===
RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_ix.c,v
retrieving revision 1.180
diff -u -p -r1.180 if_ix.c
--- dev/pci/if_ix.c 27 Jul 2021 01:44:55 - 1.180
+++ dev/pci/if_ix.c 13 Jan 2022 20:14:33 -
@@ -2437,12 +2437,9 @@ ixgbe_tx_ctx_setup(struct tx_ring *txr,
#else
struct ether_header *eh;
#endif
- struct ip *ip;
#ifdef notyet
struct ip6_hdr *ip6;
#endif
- struct mbuf *m;
- int ipoff;
uint32_t vlan_macip_lens = 0, type_tucmd_mlhl = 0;
int ehdrlen, ip_hlen = 0;
uint16_t etype;
@@ -2511,16 +2508,23 @@ ixgbe_tx_ctx_setup(struct tx_ring *txr,
vlan_macip_lens |= ehdrlen << IXGBE_ADVTXD_MACLEN_SHIFT;
switch (etype) {
- case ETHERTYPE_IP:
+ case ETHERTYPE_IP: {
+ struct ip *ip, ipdata;
+
if (mp->m_pkthdr.len < ehdrlen + sizeof(*ip))
return (-1);
- m = m_getptr(mp, ehdrlen, &ipoff);
- KASSERT(m != NULL && m->m_len - ipoff >= sizeof(*ip));
- ip = (struct ip *)(m->m_data + ipoff);
+ if ((mtod(mp, unsigned long) & ALIGNBYTES) == 0 &&
+ mp->m_len >= ehdrlen + sizeof(*ip)) {
+ ip = (struct ip *)(mp->m_data + ehdrlen);
+ } else {
+ m_copydata(mp, ehdrlen, sizeof(ipdata), &ipdata);
+ ip = &ipdata;
+ }
ip_hlen = ip->ip_hl << 2;
ipproto = ip->ip_p;
type_tucmd_mlhl |= IXGBE_ADVTXD_TUCMD_IPV4;
break;
+ }
#ifdef notyet
case ETHERTYPE_IPV6:
if (mp->m_pkthdr.len < ehdrlen + sizeof(*ip6))
Re: ix(4): enable TCPv6/UDPv6 cksum offloading
> Date: Thu, 13 Jan 2022 20:34:36 +0100 > From: Alexander Bluhm > > On Wed, Jan 12, 2022 at 05:36:01PM +0100, Mark Kettenis wrote: > > > Date: Wed, 12 Jan 2022 17:02:03 +0100 > > > From: Jan Klemkow > > > > > > Hi, > > > > > > This diff enables TCP and UDP checksum offloading in ix(4) for IPv6. > > > > > > IPv6 extension headers aren't a problem in this case. > > > in6_proto_cksum_out() in netinet6/ip6_output.c disables checksum > > > offloading if ip6_nxt is not TCP or UDP. Thus, we can just use this > > > field. > > > > > > Tested with: > > > ix0 at pci5 dev 0 function 0 "Intel 82599" rev 0x01, msix, 8 queues, > > > address 00:1b:21:94:4c:48 > > > > > > OK? > > > > Isn't this the same disaster as the ixl(4) diff you sent earlier? We > > have sparc64 machines with onboard ix(4)... > > No, it is another case. The disaster was with TCP bit fields, here > we are talking about IPv6 header. > > - A few lines above ip->ip_p is used the same way, and that works. > - I have checked sparc64 assembler, it uses byte instruction. > 1bd0: c4 08 60 06 ldub [ %g1 + 6 ], %g2 > - struct ip6_hdr uses __packed. I think this is wrong, but that > is a different story. > - Even if I remove this __packed, gcc uses a byte instruction for > reading ip6_nxt. > - m_getptr() returns the correct mbuf and offset to the header. I > think we can assume that a single IPv6 header, that our stack has > created, is in contiguous memory. The IPv4 case just above makes > the same assumption. So the crucial thing here is that this only accesses a single byte in the header. We did indeed establish that that is safe provided the header itself is indeed not fragmented. > OK bluhm@ > > > > Index: dev/pci/if_ix.c > > > === > > > RCS file: /mount/openbsd/cvs/src/sys/dev/pci/if_ix.c,v > > > retrieving revision 1.180 > > > diff -u -p -r1.180 if_ix.c > > > --- dev/pci/if_ix.c 27 Jul 2021 01:44:55 - 1.180 > > > +++ dev/pci/if_ix.c 12 Jan 2022 14:53:14 - > > > @@ -1879,7 +1879,8 @@ ixgbe_setup_interface(struct ix_softc *s > > > ifp->if_capabilities |= IFCAP_VLAN_HWTAGGING; > > > #endif > > > > > > - ifp->if_capabilities |= IFCAP_CSUM_TCPv4 | IFCAP_CSUM_UDPv4; > > > + ifp->if_capabilities |= IFCAP_CSUM_TCPv4 | IFCAP_CSUM_UDPv4 > > > + | IFCAP_CSUM_TCPv6 | IFCAP_CSUM_UDPv6; > > > > > > /* > > >* Specify the media types supported by this sc and register > > > @@ -2438,9 +2439,7 @@ ixgbe_tx_ctx_setup(struct tx_ring *txr, > > > struct ether_header *eh; > > > #endif > > > struct ip *ip; > > > -#ifdef notyet > > > struct ip6_hdr *ip6; > > > -#endif > > > struct mbuf *m; > > > int ipoff; > > > uint32_t vlan_macip_lens = 0, type_tucmd_mlhl = 0; > > > @@ -2521,19 +2520,16 @@ ixgbe_tx_ctx_setup(struct tx_ring *txr, > > > ipproto = ip->ip_p; > > > type_tucmd_mlhl |= IXGBE_ADVTXD_TUCMD_IPV4; > > > break; > > > -#ifdef notyet > > > case ETHERTYPE_IPV6: > > > if (mp->m_pkthdr.len < ehdrlen + sizeof(*ip6)) > > > return (-1); > > > m = m_getptr(mp, ehdrlen, &ipoff); > > > KASSERT(m != NULL && m->m_len - ipoff >= sizeof(*ip6)); > > > - ip6 = (struct ip6 *)(m->m_data + ipoff); > > > + ip6 = (struct ip6_hdr *)(m->m_data + ipoff); > > > ip_hlen = sizeof(*ip6); > > > - /* XXX-BZ this will go badly in case of ext hdrs. */ > > > ipproto = ip6->ip6_nxt; > > > type_tucmd_mlhl |= IXGBE_ADVTXD_TUCMD_IPV6; > > > break; > > > -#endif > > > default: > > > offload = FALSE; > > > break; > > > Index: dev/pci/ixgbe.h > > > === > > > RCS file: /mount/openbsd/cvs/src/sys/dev/pci/ixgbe.h,v > > > retrieving revision 1.32 > > > diff -u -p -r1.32 ixgbe.h > > > --- dev/pci/ixgbe.h 18 Jul 2020 07:18:22 - 1.32 > > > +++ dev/pci/ixgbe.h 12 Jan 2022 14:57:13 - > > > @@ -65,6 +65,7 @@ > > > #include > > > #include > > > #include > > > +#include > > > > > > #if NBPFILTER > 0 > > > #include > > > > > > >
Re: request for testing: malloc and large allocations
Hey, On 1/9/22 14:54, Otto Moerbeek wrote: currently malloc does cache a number of free'ed regions up to 128k in size. This cache is indexed by size (in # of pages), so it is very quick to check. Some programs allocate and deallocate larger allocations in a frantic way. Accodomate those programs by also keeping a cache of regions betwen 128k and 2M, in a cache of variable sized regions. My test case speeds up about twice. A make build gets a small speedup. This has been tested by myself on amd64 quite intensively. I am asking for more tests, especialy on more "exotic" platforms. I wil do arm64 myself soon. Test can be running your favorite programs, doing make builds or running tests in regress/lib/libc/malloc. Thanks in advance! i did some workloads with your patch applied. I did not notice any speedups (but i also did not time it, just a suggestive feeling while waiting for cargo build to finish) To me everything works fine. I am on a normal amd64, sadly i dont have other hardware available. I hope this helps Greetings Leo
Re: ix(4): enable TCPv6/UDPv6 cksum offloading
> - m_getptr() returns the correct mbuf and offset to the header. I > think we can assume that a single IPv6 header, that our stack has > created, is in contiguous memory. The IPv4 case just above makes > the same assumption. And if the assumption is not met due to interations between various tunnel or encapsulation layers... then panic the kernel? That is OK?
Re: ix(4): enable TCPv6/UDPv6 cksum offloading
On Wed, Jan 12, 2022 at 05:36:01PM +0100, Mark Kettenis wrote: > > Date: Wed, 12 Jan 2022 17:02:03 +0100 > > From: Jan Klemkow > > > > Hi, > > > > This diff enables TCP and UDP checksum offloading in ix(4) for IPv6. > > > > IPv6 extension headers aren't a problem in this case. > > in6_proto_cksum_out() in netinet6/ip6_output.c disables checksum > > offloading if ip6_nxt is not TCP or UDP. Thus, we can just use this > > field. > > > > Tested with: > > ix0 at pci5 dev 0 function 0 "Intel 82599" rev 0x01, msix, 8 queues, > > address 00:1b:21:94:4c:48 > > > > OK? > > Isn't this the same disaster as the ixl(4) diff you sent earlier? We > have sparc64 machines with onboard ix(4)... No, it is another case. The disaster was with TCP bit fields, here we are talking about IPv6 header. - A few lines above ip->ip_p is used the same way, and that works. - I have checked sparc64 assembler, it uses byte instruction. 1bd0: c4 08 60 06 ldub [ %g1 + 6 ], %g2 - struct ip6_hdr uses __packed. I think this is wrong, but that is a different story. - Even if I remove this __packed, gcc uses a byte instruction for reading ip6_nxt. - m_getptr() returns the correct mbuf and offset to the header. I think we can assume that a single IPv6 header, that our stack has created, is in contiguous memory. The IPv4 case just above makes the same assumption. OK bluhm@ > > Index: dev/pci/if_ix.c > > === > > RCS file: /mount/openbsd/cvs/src/sys/dev/pci/if_ix.c,v > > retrieving revision 1.180 > > diff -u -p -r1.180 if_ix.c > > --- dev/pci/if_ix.c 27 Jul 2021 01:44:55 - 1.180 > > +++ dev/pci/if_ix.c 12 Jan 2022 14:53:14 - > > @@ -1879,7 +1879,8 @@ ixgbe_setup_interface(struct ix_softc *s > > ifp->if_capabilities |= IFCAP_VLAN_HWTAGGING; > > #endif > > > > - ifp->if_capabilities |= IFCAP_CSUM_TCPv4 | IFCAP_CSUM_UDPv4; > > + ifp->if_capabilities |= IFCAP_CSUM_TCPv4 | IFCAP_CSUM_UDPv4 > > + | IFCAP_CSUM_TCPv6 | IFCAP_CSUM_UDPv6; > > > > /* > > * Specify the media types supported by this sc and register > > @@ -2438,9 +2439,7 @@ ixgbe_tx_ctx_setup(struct tx_ring *txr, > > struct ether_header *eh; > > #endif > > struct ip *ip; > > -#ifdef notyet > > struct ip6_hdr *ip6; > > -#endif > > struct mbuf *m; > > int ipoff; > > uint32_t vlan_macip_lens = 0, type_tucmd_mlhl = 0; > > @@ -2521,19 +2520,16 @@ ixgbe_tx_ctx_setup(struct tx_ring *txr, > > ipproto = ip->ip_p; > > type_tucmd_mlhl |= IXGBE_ADVTXD_TUCMD_IPV4; > > break; > > -#ifdef notyet > > case ETHERTYPE_IPV6: > > if (mp->m_pkthdr.len < ehdrlen + sizeof(*ip6)) > > return (-1); > > m = m_getptr(mp, ehdrlen, &ipoff); > > KASSERT(m != NULL && m->m_len - ipoff >= sizeof(*ip6)); > > - ip6 = (struct ip6 *)(m->m_data + ipoff); > > + ip6 = (struct ip6_hdr *)(m->m_data + ipoff); > > ip_hlen = sizeof(*ip6); > > - /* XXX-BZ this will go badly in case of ext hdrs. */ > > ipproto = ip6->ip6_nxt; > > type_tucmd_mlhl |= IXGBE_ADVTXD_TUCMD_IPV6; > > break; > > -#endif > > default: > > offload = FALSE; > > break; > > Index: dev/pci/ixgbe.h > > === > > RCS file: /mount/openbsd/cvs/src/sys/dev/pci/ixgbe.h,v > > retrieving revision 1.32 > > diff -u -p -r1.32 ixgbe.h > > --- dev/pci/ixgbe.h 18 Jul 2020 07:18:22 - 1.32 > > +++ dev/pci/ixgbe.h 12 Jan 2022 14:57:13 - > > @@ -65,6 +65,7 @@ > > #include > > #include > > #include > > +#include > > > > #if NBPFILTER > 0 > > #include > > > >
Re: request for testing: malloc and large allocations
Hi Otto, * Otto Moerbeek wrote: > Hi, > > currently malloc does cache a number of free'ed regions up to 128k in > size. This cache is indexed by size (in # of pages), so it is very > quick to check. > > Some programs allocate and deallocate larger allocations in a frantic > way. Accodomate those programs by also keeping a cache of regions > betwen 128k and 2M, in a cache of variable sized regions. > > My test case speeds up about twice. A make build gets a small speedup. > > This has been tested by myself on amd64 quite intensively. I am asking > for more tests, especialy on more "exotic" platforms. I wil do arm64 > myself soon. Test can be running your favorite programs, doing make > builds or running tests in regress/lib/libc/malloc. I have your patch running on an amd64 Thinkpad T450s with usual desktop usage and noticed no regression so far. Cheers Matthias
fix active scan on iwm and iwx
At present active scans (which send probe requests, as opposed to
just listening for beacons) are disabled on iwm 9k and iwx. This
was done because firmware misbehaved after association.
zxystd from the OpenIntelWireless project has debugged the issue
and has sent me a patch against OpenBSD which fixes this problem.
The patch is below, with some small tweaks by me which have already
been reviewed by zxystd.
It seems that firmware misbehaves if the driver sets the DTIM period
to zero. This value is read from TIM information elements (IE) in beacons.
Passive scans worked because we picked up the DTIM period from a beacon,
while probe responses received during active scans lack the TIM IE, which
resulted in a zero DTIM period being configured in firmware. We then never
updated TIM information when a beacon was recieved, letting firmware run
with a zero DTIM period until it eventually stopped working.
I have tested this patch on iwm 8265 and iwx ax200.
And jmc@ has been testing this on ax200 for some time.
More tests are welcome, on any supported device.
diff refs/heads/remove-find-node-for-beacon refs/heads/iwm-iwx-active-scan
blob - 607a45b71f7ea71773809136cadf122c72375558
blob + 937f2cc28f6c85502031e4c9efa0a02c75fd1a6d
--- sys/dev/pci/if_iwm.c
+++ sys/dev/pci/if_iwm.c
@@ -340,6 +340,7 @@ voidiwm_updateprot(struct ieee80211com *);
void iwm_updateslot(struct ieee80211com *);
void iwm_updateedca(struct ieee80211com *);
void iwm_updatechan(struct ieee80211com *);
+void iwm_updatedtim(struct ieee80211com *);
void iwm_init_reorder_buffer(struct iwm_reorder_buffer *, uint16_t,
uint16_t);
void iwm_clear_reorder_buffer(struct iwm_softc *, struct iwm_rxba_data *);
@@ -3374,6 +3375,8 @@ iwm_mac_ctxt_task(void *arg)
err = iwm_mac_ctxt_cmd(sc, in, IWM_FW_CTXT_ACTION_MODIFY, 1);
if (err)
printf("%s: failed to update MAC\n", DEVNAME(sc));
+
+ iwm_unprotect_session(sc, in);
refcnt_rele_wake(&sc->task_refs);
splx(s);
@@ -3454,6 +3457,16 @@ iwm_updatechan(struct ieee80211com *ic)
iwm_add_task(sc, systq, &sc->phy_ctxt_task);
}
+void
+iwm_updatedtim(struct ieee80211com *ic)
+{
+ struct iwm_softc *sc = ic->ic_softc;
+
+ if (ic->ic_state == IEEE80211_S_RUN &&
+ !task_pending(&sc->newstate_task))
+ iwm_add_task(sc, systq, &sc->mac_ctxt_task);
+}
+
int
iwm_sta_tx_agg(struct iwm_softc *sc, struct ieee80211_node *ni, uint8_t tid,
uint16_t ssn, uint16_t winsize, int start)
@@ -7275,12 +7288,7 @@ iwm_lmac_scan_fill_channels(struct iwm_softc *sc,
chan->iter_count = htole16(1);
chan->iter_interval = 0;
chan->flags = htole32(IWM_UNIFIED_SCAN_CHANNEL_PARTIAL);
- /*
-* Firmware may become unresponsive when asked to send
-* a directed probe request on a passive channel.
-*/
- if (n_ssids != 0 && !bgscan &&
- (c->ic_flags & IEEE80211_CHAN_PASSIVE) == 0)
+ if (n_ssids != 0 && !bgscan)
chan->flags |= htole32(1 << 1); /* select SSID 0 */
chan++;
nchan++;
@@ -7307,12 +7315,7 @@ iwm_umac_scan_fill_channels(struct iwm_softc *sc,
chan->channel_num = ieee80211_mhz2ieee(c->ic_freq, 0);
chan->iter_count = 1;
chan->iter_interval = htole16(0);
- /*
-* Firmware may become unresponsive when asked to send
-* a directed probe request on a passive channel.
-*/
- if (n_ssids != 0 && !bgscan &&
- (c->ic_flags & IEEE80211_CHAN_PASSIVE) == 0)
+ if (n_ssids != 0 && !bgscan)
chan->flags = htole32(1 << 0); /* select SSID 0 */
chan++;
nchan++;
@@ -7782,13 +7785,7 @@ iwm_umac_scan(struct iwm_softc *sc, int bgscan)
IWM_UMAC_SCAN_GEN_FLAGS2_ALLOW_CHNL_REORDER;
}
- /*
-* Check if we're doing an active directed scan.
-* 9k devices may randomly lock up (no interrupts) after association
-* following active scans. Use passive scan only for now on 9k.
-*/
- if (sc->sc_device_family != IWM_DEVICE_FAMILY_9000 &&
- ic->ic_des_esslen != 0) {
+ if (ic->ic_des_esslen != 0) {
if (isset(sc->sc_ucode_api,
IWM_UCODE_TLV_API_SCAN_EXT_CHAN_VER)) {
tail->direct_scan[0].id = IEEE80211_ELEMID_SSID;
@@ -11620,6 +11617,7 @@ iwm_attach(struct device *parent, struct device *self,
ic->ic_updateprot = iwm_updateprot;
ic->ic_updateslot = iwm_updateslot;
ic->ic_updateedca = iwm_updateedca;
+ ic->ic_updatedtim = iwm_updatedtim;
ic->ic_ampdu_rx_start = iwm_ampdu_rx_start;
ic->ic_ampdu_rx_stop = iwm_ampdu_rx_stop;
ic
rpki-client introduce validated cache
This diff adds a new cache subdir called "valid". This is the place where
all verified and good files are stored after a run. It makes -n work a lot
better since -n will now only look at what's inside "valid" and ignore
"rsync" and "rrdp".
The trust anchors are still stored in "ta" even if valid.
The rsync repo will only hold temporary files and be empty otherwise.
The rrdp repo still may contain some superfluous files that people
included in their snapshots. Not keeping them could result in extra
snapshot downloads since delta updates could refer to these files.
Since there is a valid repo, rrdp no longer needs an additional temp
directory for its sync.
With this rsync will use the --compare-dest feature and use the valid
cache as a base.
The file cleanup at the end got more complex. There is now
repo_move_valid() and repo_cleanup_rrdp() to do two initial steps of
cleanup before the tree traversal starts. The fts function now uses
some fts features to find a) empty directories and b) superfluous files.
I think the code changes outside of repo.c should be straight forward.
To the rpki connoisseur, this currently only implements a simple newest
file wins startegy. The handling of .mft and their serial number will
follow once this is in.
--
:wq Claudio
? obj
Index: extern.h
===
RCS file: /cvs/src/usr.sbin/rpki-client/extern.h,v
retrieving revision 1.103
diff -u -p -r1.103 extern.h
--- extern.h13 Jan 2022 13:46:03 - 1.103
+++ extern.h13 Jan 2022 15:50:40 -
@@ -341,7 +341,7 @@ enum publish_type {
struct entity {
TAILQ_ENTRY(entity) entries;
char*path; /* path relative to repository */
- char*file; /* filename */
+ char*file; /* filename or valid repo path */
unsigned char *data; /* optional data blob */
size_t datasz;/* length of optional data blob */
unsigned int repoid;/* repository identifier */
@@ -380,6 +380,7 @@ struct stats {
size_t vrps; /* total number of vrps */
size_t uniqs; /* number of unique vrps */
size_t del_files; /* number of files removed in cleanup */
+ size_t extra_files; /* number of superfluous files */
size_t del_dirs; /* number of directories removed in cleanup */
size_t brks; /* number of BGPsec Router Key (BRK) certificates */
struct timeval elapsed_time;
@@ -506,7 +507,7 @@ void rrdp_clear(unsigned int);
voidrrdp_save_state(unsigned int, struct rrdp_session *);
int rrdp_handle_file(unsigned int, enum publish_type, char *,
char *, size_t, char *, size_t);
-char *repo_basedir(const struct repo *);
+char *repo_basedir(const struct repo *, int);
unsigned intrepo_id(const struct repo *);
const char *repo_uri(const struct repo *);
struct repo*ta_lookup(int, struct tal *);
@@ -520,7 +521,8 @@ void rsync_finish(unsigned int, int);
voidhttp_finish(unsigned int, enum http_result, const char *);
voidrrdp_finish(unsigned int, int);
-voidrsync_fetch(unsigned int, const char *, const char *);
+voidrsync_fetch(unsigned int, const char *, const char *,
+ const char *);
voidhttp_fetch(unsigned int, const char *, const char *, int);
voidrrdp_fetch(unsigned int, const char *, const char *,
struct rrdp_session *);
Index: main.c
===
RCS file: /cvs/src/usr.sbin/rpki-client/main.c,v
retrieving revision 1.175
diff -u -p -r1.175 main.c
--- main.c 13 Jan 2022 13:18:41 - 1.175
+++ main.c 13 Jan 2022 15:50:40 -
@@ -151,17 +151,18 @@ entity_write_repo(struct repo *rp)
struct ibuf *b;
enum rtype type = RTYPE_REPO;
unsigned int repoid;
- char *path;
+ char *path, *altpath;
int talid = 0;
repoid = repo_id(rp);
- path = repo_basedir(rp);
+ path = repo_basedir(rp, 0);
+ altpath = repo_basedir(rp, 1);
b = io_new_buffer();
io_simple_buffer(b, &type, sizeof(type));
io_simple_buffer(b, &repoid, sizeof(repoid));
io_simple_buffer(b, &talid, sizeof(talid));
io_str_buffer(b, path);
- io_str_buffer(b, NULL);
+ io_str_buffer(b, altpath);
io_buf_buffer(b, NULL, 0);
io_close_buffer(&procq, b);
free(path);
@@ -254,14 +255,15 @@ rrdp_fetch(unsigned int id, const char *
* Request a repository sync via rsync URI to directory local.
*/
void
-rsync_fetch(unsigned int id, const char *uri, const char *local)
+rsync_fetch(unsigned int id, const char *uri, const char *local,
+const char *base)
{
struct ibuf *b;
b = io_new_buffer();
i
Re: rpki-client reshuffle deck chairs
On Thu, Jan 13, 2022 at 02:52:22PM +0100, Claudio Jeker wrote: > This diff just shuffles some functions around to reduce the size of the > validate cache diff. ok
Re: rpki-client stop checking stale manifests
On Thu, Jan 13, 2022 at 02:50:59PM +0100, Claudio Jeker wrote:
> Noticed the other day, a stale manifest tries to check the fileandhash
> data. But when running with -n none of this data will be around since it
> was most probably removed on the previous run. The result is a lot of
> warnings on top of the warning about the mft being stale. It is better to
> skip mft_check() if the mft is stale.
>
> Also reshuffle the order of setting repoid and path to be before the
> mft_check() call. Upcoming code will require this information in
> mft_check().
ok
> --
> :wq Claudio
>
> Index: parser.c
> ===
> RCS file: /cvs/src/usr.sbin/rpki-client/parser.c,v
> retrieving revision 1.35
> diff -u -p -r1.35 parser.c
> --- parser.c 13 Jan 2022 13:46:03 - 1.35
> +++ parser.c 13 Jan 2022 13:47:04 -
> @@ -310,15 +310,17 @@ proc_parser_mft(char *file, const unsign
> sk_X509_free(chain);
> X509_free(x509);
>
> - if (!mft_check(file, mft)) {
> - mft_free(mft);
> - return NULL;
> - }
> -
> + mft->repoid = repoid;
> if (path != NULL)
> if ((mft->path = strdup(path)) == NULL)
> err(1, NULL);
> - mft->repoid = repoid;
> +
> + if (!mft->stale)
> + if (!mft_check(file, mft)) {
> + mft_free(mft);
> + return NULL;
> + }
> +
> return mft;
> }
>
>
rpki-client reshuffle deck chairs
This diff just shuffles some functions around to reduce the size of the
validate cache diff.
--
:wq Claudio
Index: repo.c
===
RCS file: /cvs/src/usr.sbin/rpki-client/repo.c,v
retrieving revision 1.23
diff -u -p -r1.23 repo.c
--- repo.c 13 Jan 2022 13:46:03 - 1.23
+++ repo.c 13 Jan 2022 13:47:22 -
@@ -336,47 +336,6 @@ ta_filename(const struct tarepo *tr, int
return nfile;
}
-/*
- * Build local file name base on the URI and the rrdprepo info.
- */
-static char *
-rrdp_filename(const struct rrdprepo *rr, const char *uri, int temp)
-{
- char *nfile;
- char *dir = rr->basedir;
-
- if (temp)
- dir = rr->temp;
-
- if (!valid_uri(uri, strlen(uri), "rsync://")) {
- warnx("%s: bad URI %s", rr->basedir, uri);
- return NULL;
- }
-
- uri += strlen("rsync://"); /* skip proto */
- if (asprintf(&nfile, "%s/%s", dir, uri) == -1)
- err(1, NULL);
- return nfile;
-}
-
-/*
- * Build RRDP state file name based on the repo info.
- * If temp is set add Xs for mkostemp.
- */
-static char *
-rrdp_state_filename(const struct rrdprepo *rr, int temp)
-{
- char *nfile;
-
- if (asprintf(&nfile, "%s/.state%s", rr->basedir,
- temp ? ".": "") == -1)
- err(1, NULL);
-
- return nfile;
-}
-
-
-
static void
ta_fetch(struct tarepo *tr)
{
@@ -556,51 +515,43 @@ rsync_free(void)
static int rrdprepo_fetch(struct rrdprepo *);
-static struct rrdprepo *
-rrdp_get(const char *uri, int nofetch)
+/*
+ * Build local file name base on the URI and the rrdprepo info.
+ */
+static char *
+rrdp_filename(const struct rrdprepo *rr, const char *uri, int temp)
{
- struct rrdprepo *rr;
-
- SLIST_FOREACH(rr, &rrdprepos, entry)
- if (strcmp(rr->notifyuri, uri) == 0) {
- if (rr->state == REPO_FAILED)
- return NULL;
- return rr;
- }
+ char *nfile;
+ char *dir = rr->basedir;
- if ((rr = calloc(1, sizeof(*rr))) == NULL)
- err(1, NULL);
+ if (temp)
+ dir = rr->temp;
- rr->id = ++repoid;
- SLIST_INSERT_HEAD(&rrdprepos, rr, entry);
+ if (!valid_uri(uri, strlen(uri), "rsync://")) {
+ warnx("%s: bad URI %s", rr->basedir, uri);
+ return NULL;
+ }
- if ((rr->notifyuri = strdup(uri)) == NULL)
+ uri += strlen("rsync://"); /* skip proto */
+ if (asprintf(&nfile, "%s/%s", dir, uri) == -1)
err(1, NULL);
- rr->basedir = repo_dir(uri, "rrdp", 1);
-
- RB_INIT(&rr->added);
- RB_INIT(&rr->deleted);
+ return nfile;
+}
- if (noop || nofetch) {
- rr->state = REPO_DONE;
- logx("%s: using cache", rr->notifyuri);
- repo_done(rr, 0);
- } else {
- /* create base directory */
- if (mkpath(rr->basedir) == -1) {
- warn("mkpath %s", rr->basedir);
- rrdp_finish(rr->id, 0);
- return rr;
- }
- if (rrdprepo_fetch(rr) == -1) {
- rrdp_finish(rr->id, 0);
- return rr;
- }
+/*
+ * Build RRDP state file name based on the repo info.
+ * If temp is set add Xs for mkostemp.
+ */
+static char *
+rrdp_state_filename(const struct rrdprepo *rr, int temp)
+{
+ char *nfile;
- logx("%s: pulling from %s", rr->notifyuri, "network");
- }
+ if (asprintf(&nfile, "%s/.state%s", rr->basedir,
+ temp ? ".": "") == -1)
+ err(1, NULL);
- return rr;
+ return nfile;
}
static struct rrdprepo *
@@ -786,6 +737,53 @@ fail:
unlink(temp);
free(temp);
free(file);
+}
+
+static struct rrdprepo *
+rrdp_get(const char *uri, int nofetch)
+{
+ struct rrdprepo *rr;
+
+ SLIST_FOREACH(rr, &rrdprepos, entry)
+ if (strcmp(rr->notifyuri, uri) == 0) {
+ if (rr->state == REPO_FAILED)
+ return NULL;
+ return rr;
+ }
+
+ if ((rr = calloc(1, sizeof(*rr))) == NULL)
+ err(1, NULL);
+
+ rr->id = ++repoid;
+ SLIST_INSERT_HEAD(&rrdprepos, rr, entry);
+
+ if ((rr->notifyuri = strdup(uri)) == NULL)
+ err(1, NULL);
+ rr->basedir = repo_dir(uri, "rrdp", 1);
+
+ RB_INIT(&rr->added);
+ RB_INIT(&rr->deleted);
+
+ if (noop || nofetch) {
+ rr->state = REPO_DONE;
+ logx("%s: using cache", rr->notifyuri);
+ repo_done(rr, 0);
+ } else {
+ /* create base directory */
+ if (mkpath(rr->basedir) == -1) {
+ warn
rpki-client stop checking stale manifests
Noticed the other day, a stale manifest tries to check the fileandhash
data. But when running with -n none of this data will be around since it
was most probably removed on the previous run. The result is a lot of
warnings on top of the warning about the mft being stale. It is better to
skip mft_check() if the mft is stale.
Also reshuffle the order of setting repoid and path to be before the
mft_check() call. Upcoming code will require this information in
mft_check().
--
:wq Claudio
Index: parser.c
===
RCS file: /cvs/src/usr.sbin/rpki-client/parser.c,v
retrieving revision 1.35
diff -u -p -r1.35 parser.c
--- parser.c13 Jan 2022 13:46:03 - 1.35
+++ parser.c13 Jan 2022 13:47:04 -
@@ -310,15 +310,17 @@ proc_parser_mft(char *file, const unsign
sk_X509_free(chain);
X509_free(x509);
- if (!mft_check(file, mft)) {
- mft_free(mft);
- return NULL;
- }
-
+ mft->repoid = repoid;
if (path != NULL)
if ((mft->path = strdup(path)) == NULL)
err(1, NULL);
- mft->repoid = repoid;
+
+ if (!mft->stale)
+ if (!mft_check(file, mft)) {
+ mft_free(mft);
+ return NULL;
+ }
+
return mft;
}
Re: rpki-client, adjust valid_filehash and callers for repo split
On Thu, Jan 13, 2022 at 02:24:59PM +0100, Theo Buehler wrote: > On Thu, Jan 13, 2022 at 02:16:02PM +0100, Claudio Jeker wrote: > > Right now a file can only exist in one place in the rpki-client cache. > > This will change when we split valid data to its own repo. > > > > One step to get closer to that is to alter valid_filehash() to take an > > open filedescriptor instead of using open(2) itself. This allows the > > callers to decide which file to pass. valid_filehash() handles fd == -1 > > and so the open call does not need to be checked for error. > > > > Additionally move mft_check() from mft.c to parser.c. It simplifies later > > work. > > > > ok > > > Index: validate.c > > === > > RCS file: /cvs/src/usr.sbin/rpki-client/validate.c,v > > retrieving revision 1.23 > > diff -u -p -r1.23 validate.c > > --- validate.c 26 Dec 2021 12:32:28 - 1.23 > > +++ validate.c 13 Jan 2022 12:07:54 - > > @@ -269,29 +269,29 @@ valid_filename(const char *fn) > > > > /* > > * Validate a file by verifying the SHA256 hash of that file. > > - * Returns 1 if valid, 0 otherwise. > > + * The file is passed a an open filedescriptor fd which can be -1. > > missing s: passed as > and "file descriptor" should be two words > (also: -1 can't be an open fd, but I guess it's clear what is meant) > I simplified the comment a bit. The function is easy enough to understand. /* * Validate a file by verifying the SHA256 hash of that file. * The file to check is passed as a filedescriptor. * Returns 1 if hash matched, 0 otherwise. Closes fd when done. */ -- :wq Claudio
Re: rpki-client, adjust valid_filehash and callers for repo split
On Thu, Jan 13, 2022 at 02:16:02PM +0100, Claudio Jeker wrote:
> Right now a file can only exist in one place in the rpki-client cache.
> This will change when we split valid data to its own repo.
>
> One step to get closer to that is to alter valid_filehash() to take an
> open filedescriptor instead of using open(2) itself. This allows the
> callers to decide which file to pass. valid_filehash() handles fd == -1
> and so the open call does not need to be checked for error.
>
> Additionally move mft_check() from mft.c to parser.c. It simplifies later
> work.
>
ok
> Index: validate.c
> ===
> RCS file: /cvs/src/usr.sbin/rpki-client/validate.c,v
> retrieving revision 1.23
> diff -u -p -r1.23 validate.c
> --- validate.c26 Dec 2021 12:32:28 - 1.23
> +++ validate.c13 Jan 2022 12:07:54 -
> @@ -269,29 +269,29 @@ valid_filename(const char *fn)
>
> /*
> * Validate a file by verifying the SHA256 hash of that file.
> - * Returns 1 if valid, 0 otherwise.
> + * The file is passed a an open filedescriptor fd which can be -1.
missing s: passed as
and "file descriptor" should be two words
(also: -1 can't be an open fd, but I guess it's clear what is meant)
> + * Returns 1 if valid, 0 otherwise. Closes fd when done.
> */
> int
> -valid_filehash(const char *fn, const char *hash, size_t hlen)
> +valid_filehash(int fd, const char *hash, size_t hlen)
> {
> SHA256_CTX ctx;
> charfilehash[SHA256_DIGEST_LENGTH];
> charbuffer[8192];
> ssize_t nr;
> - int fd;
>
> if (hlen != sizeof(filehash))
> errx(1, "bad hash size");
>
> - if ((fd = open(fn, O_RDONLY)) == -1)
> + if (fd == -1)
> return 0;
>
> SHA256_Init(&ctx);
> while ((nr = read(fd, buffer, sizeof(buffer))) > 0)
> SHA256_Update(&ctx, buffer, nr);
> close(fd);
> -
> SHA256_Final(filehash, &ctx);
> +
> if (memcmp(hash, filehash, sizeof(filehash)) != 0)
> return 0;
>
>
rpki-client, adjust valid_filehash and callers for repo split
Right now a file can only exist in one place in the rpki-client cache.
This will change when we split valid data to its own repo.
One step to get closer to that is to alter valid_filehash() to take an
open filedescriptor instead of using open(2) itself. This allows the
callers to decide which file to pass. valid_filehash() handles fd == -1
and so the open call does not need to be checked for error.
Additionally move mft_check() from mft.c to parser.c. It simplifies later
work.
--
:wq Claudio
Index: extern.h
===
RCS file: /cvs/src/usr.sbin/rpki-client/extern.h,v
retrieving revision 1.101
diff -u -p -r1.101 extern.h
--- extern.h11 Jan 2022 13:06:07 - 1.101
+++ extern.h13 Jan 2022 12:56:56 -
@@ -445,7 +445,7 @@ int valid_cert(const char *, struct au
const struct cert *);
int valid_roa(const char *, struct auth_tree *, struct roa *);
int valid_filename(const char *);
-int valid_filehash(const char *, const char *, size_t);
+int valid_filehash(int, const char *, size_t);
int valid_uri(const char *, size_t, const char *);
int valid_origin(const char *, const char *);
Index: mft.c
===
RCS file: /cvs/src/usr.sbin/rpki-client/mft.c,v
retrieving revision 1.44
diff -u -p -r1.44 mft.c
--- mft.c 11 Jan 2022 13:06:07 - 1.44
+++ mft.c 13 Jan 2022 12:56:38 -
@@ -20,7 +20,6 @@
#include
#include
#include
-#include
#include
#include
#include
@@ -456,44 +455,6 @@ out:
}
free(cms);
return p.res;
-}
-
-/*
- * Check all files and their hashes in a MFT structure.
- * Return zero on failure, non-zero on success.
- */
-int
-mft_check(const char *fn, struct mft *p)
-{
- size_t i;
- int rc = 1;
- char*cp, *h, *path = NULL;
-
- /* Check hash of file now, but first build path for it */
- cp = strrchr(fn, '/');
- assert(cp != NULL);
- assert(cp - fn < INT_MAX);
-
- for (i = 0; i < p->filesz; i++) {
- const struct mftfile *m = &p->files[i];
- if (!valid_filename(m->file)) {
- if (base64_encode(m->hash, sizeof(m->hash), &h) == -1)
- errx(1, "base64_encode failed in %s", __func__);
- warnx("%s: unsupported filename for %s", fn, h);
- free(h);
- continue;
- }
- if (asprintf(&path, "%.*s/%s", (int)(cp - fn), fn,
- m->file) == -1)
- err(1, NULL);
- if (!valid_filehash(path, m->hash, sizeof(m->hash))) {
- warnx("%s: bad message digest for %s", fn, m->file);
- rc = 0;
- }
- free(path);
- }
-
- return rc;
}
/*
Index: parser.c
===
RCS file: /cvs/src/usr.sbin/rpki-client/parser.c,v
retrieving revision 1.34
diff -u -p -r1.34 parser.c
--- parser.c11 Jan 2022 13:06:07 - 1.34
+++ parser.c13 Jan 2022 13:00:27 -
@@ -22,6 +22,7 @@
#include
#include
+#include
#include
#include
#include
@@ -218,6 +219,45 @@ proc_parser_roa(char *file, const unsign
X509_free(x509);
return roa;
+}
+
+/*
+ * Check all files and their hashes in a MFT structure.
+ * Return zero on failure, non-zero on success.
+ */
+int
+mft_check(const char *fn, struct mft *p)
+{
+ size_t i;
+ int fd, rc = 1;
+ char*cp, *h, *path = NULL;
+
+ /* Check hash of file now, but first build path for it */
+ cp = strrchr(fn, '/');
+ assert(cp != NULL);
+ assert(cp - fn < INT_MAX);
+
+ for (i = 0; i < p->filesz; i++) {
+ const struct mftfile *m = &p->files[i];
+ if (!valid_filename(m->file)) {
+ if (base64_encode(m->hash, sizeof(m->hash), &h) == -1)
+ errx(1, "base64_encode failed in %s", __func__);
+ warnx("%s: unsupported filename for %s", fn, h);
+ free(h);
+ continue;
+ }
+ if (asprintf(&path, "%.*s/%s", (int)(cp - fn), fn,
+ m->file) == -1)
+ err(1, NULL);
+ fd = open(path, O_RDONLY);
+ if (!valid_filehash(fd, m->hash, sizeof(m->hash))) {
+ warnx("%s: bad message digest for %s", fn, m->file);
+ rc = 0;
+ }
+ free(path);
+ }
+
+ return rc;
}
/*
Index: repo.c
===
RCS file: /cvs/src/usr.sbin/rpki-client/repo.c,v
retrieving revision 1.21
diff -u -p -r1.21 repo.c
Re: request for testing: malloc and large allocations
On 2022/01/09 14:54, Otto Moerbeek wrote:
> currently malloc does cache a number of free'ed regions up to 128k in
> size. This cache is indexed by size (in # of pages), so it is very
> quick to check.
>
> Some programs allocate and deallocate larger allocations in a frantic
> way. Accodomate those programs by also keeping a cache of regions
> betwen 128k and 2M, in a cache of variable sized regions.
>
> My test case speeds up about twice. A make build gets a small speedup.
>
> This has been tested by myself on amd64 quite intensively. I am asking
> for more tests, especialy on more "exotic" platforms. I wil do arm64
> myself soon. Test can be running your favorite programs, doing make
> builds or running tests in regress/lib/libc/malloc.
This has been through mkr and ports bulk build on i386.
Ports build times vary too much to say if it's faster there but
no issues have been seen.
> Thanks in advance!
>
> -Otto
>
> Index: stdlib/malloc.c
> ===
> RCS file: /cvs/src/lib/libc/stdlib/malloc.c,v
> retrieving revision 1.272
> diff -u -p -r1.272 malloc.c
> --- stdlib/malloc.c 19 Sep 2021 09:15:22 - 1.272
> +++ stdlib/malloc.c 9 Jan 2022 13:10:35 -
> @@ -113,13 +113,28 @@ struct region_info {
>
> LIST_HEAD(chunk_head, chunk_info);
>
> -#define MAX_CACHEABLE_SIZE 32
> -struct cache {
> - void *pages[MALLOC_MAXCACHE];
> +/*
> + * Two caches, one for "small" regions, one for "big".
> + * Small cacche is an array per size, big cache is one array with different
> + * sizes regions
> + */
> +#define MAX_SMALLCACHEABLE_SIZE 32
> +#define MAX_BIGCACHEABLE_SIZE512
> +#define BIGCACHE_SIZEMALLOC_MAXCACHE
> +/* If the total # of pages is larger than this, evict before inserting */
> +#define BIGCACHE_FILL(sz)(MAX_BIGCACHEABLE_SIZE * (sz) / 4)
> +
> +struct smallcache {
> + void **pages;
> ushort length;
> ushort max;
> };
>
> +struct bigcache {
> + void *page;
> + size_t psize;
> +};
> +
> struct dir_info {
> u_int32_t canary1;
> int active; /* status of malloc */
> @@ -139,7 +154,10 @@ struct dir_info {
> void *delayed_chunks[MALLOC_DELAYED_CHUNK_MASK + 1];
> u_char rbytes[32]; /* random bytes */
> /* free pages cache */
> - struct cache cache[MAX_CACHEABLE_SIZE];
> + struct smallcache smallcache[MAX_SMALLCACHEABLE_SIZE];
> + ushort bigcache_size;
> + size_t bigcache_used;
> + struct bigcache bigcache[BIGCACHE_SIZE];
> #ifdef MALLOC_STATS
> size_t inserts;
> size_t insert_collisions;
> @@ -714,18 +732,61 @@ unmap(struct dir_info *d, void *p, size_
> size_t psz = sz >> MALLOC_PAGESHIFT;
> void *r;
> u_short i;
> - struct cache *cache;
> + struct smallcache *cache;
>
> if (sz != PAGEROUND(sz) || psz == 0)
> wrterror(d, "munmap round");
>
> - if (psz > MAX_CACHEABLE_SIZE || d->cache[psz - 1].max == 0) {
> + if (d->bigcache_size > 0 && psz > MAX_SMALLCACHEABLE_SIZE &&
> + psz <= MAX_BIGCACHEABLE_SIZE) {
> + u_short base = getrbyte(d);
> + u_short j;
> +
> + /* don't look through all slots */
> + for (j = 0; j < d->bigcache_size / 4; j++) {
> + i = (base + j) % d->bigcache_size;
> + if (d->bigcache_used <
> + BIGCACHE_FILL(d->bigcache_size)) {
> + if (d->bigcache[i].psize == 0)
> + break;
> + } else {
> + if (d->bigcache[i].psize != 0)
> + break;
> + }
> + }
> + /* if we didn't find a preferred slot, use random one */
> + if (d->bigcache[i].psize != 0) {
> + size_t tmp;
> +
> + r = d->bigcache[i].page;
> + d->bigcache_used -= d->bigcache[i].psize;
> + tmp = d->bigcache[i].psize << MALLOC_PAGESHIFT;
> + if (!mopts.malloc_freeunmap)
> + validate_junk(d, r, tmp);
> + if (munmap(r, tmp))
> + wrterror(d, "munmap %p", r);
> + STATS_SUB(d->malloc_used, tmp);
> + }
> +
> + if (clear > 0)
> + explicit_bzero(p, clear);
> + if (mopts.malloc_freeunmap) {
> + if (mprotect(p, sz, PROT_NONE))
> + wrterror(d, "mprotect %p", r);
> + } else
> + junk_free(d->malloc_junk, p, sz);
> + d->bigcache[i].page = p;
> + d->bigcache[i].psize = psz;
> + d->bigcache_used += psz;
> + return;
> + }
> +
Re: rpki-client real cleanup before snapshot
On Thu, Jan 13, 2022 at 12:24:55PM +0100, Claudio Jeker wrote: > This introduces a function remove_contents() which is implementing a basic > rm -r and uses it to clean the RRDP repository when downloading a > snapshot (especially after a delta failure). It also cleans out the temp > directory after a failure to fetch. > > With the introduction of a validated cache this will become a bit simpler > since the temp dir will disapear. Still this is a simple step and reduces > the size of the validated cache patch. ok tb
Re: rpki-client prepare to use rsync --compare-dest
On Thu, Jan 13, 2022 at 11:42:24AM +0100, Claudio Jeker wrote:
> This diff adds the code to pass --compare-dest to rsync. This will be
> used once there is a valid cache and then the rsync repo will just act as
> a delta on top.
>
> Now --compare-dest is a bit strange as in the directory passed is relative
> to the destination directory (last argument of rsync command). Because of
> this the right amount of ../../../ need to be added to the compare-dest
> path. rsync_fixup_dest() is taking care of that.
Wow
>
> While there upgrade an assert to a proper error condition since it would
> lead to an access beyond the ids array.
ok
one small comment inline
> --
> :wq Claudio
>
> Index: main.c
> ===
> RCS file: /cvs/src/usr.sbin/rpki-client/main.c,v
> retrieving revision 1.173
> diff -u -p -r1.173 main.c
> --- main.c11 Jan 2022 13:06:07 - 1.173
> +++ main.c13 Jan 2022 10:06:56 -
> @@ -261,6 +261,7 @@ rsync_fetch(unsigned int id, const char
> b = io_new_buffer();
> io_simple_buffer(b, &id, sizeof(id));
> io_str_buffer(b, local);
> + io_str_buffer(b, NULL);
> io_str_buffer(b, uri);
> io_close_buffer(&rsyncq, b);
> }
> Index: rsync.c
> ===
> RCS file: /cvs/src/usr.sbin/rpki-client/rsync.c,v
> retrieving revision 1.31
> diff -u -p -r1.31 rsync.c
> --- rsync.c 22 Dec 2021 09:35:14 - 1.31
> +++ rsync.c 13 Jan 2022 10:08:46 -
> @@ -99,6 +99,31 @@ rsync_base_uri(const char *uri)
> return base_uri;
> }
>
> +/*
> + * The directory passed as --compare-dest needs to be relative to
> + * the destination directory. This function takes care of that.
> + */
> +static char *
> +rsync_fixup_dest(char *destdir, char *compdir)
> +{
> + const char *dotdot = "../../../../../../"; /* should be enough */
> + int dirs = 1;
> + char *fn;
> + char c;
> +
> + while ((c = *destdir++) != '\0')
> + if (c == '/')
> + dirs++;
> +
> + if (dirs > 6)
> + /* too deep for us */
> + return NULL;
> +
> + if ((asprintf(&fn, "%.*s%s", dirs * 3, dotdot, compdir)) == -1)
> + err(1, NULL);
> + return fn;
> +}
> +
> static void
> proc_child(int signal)
> {
> @@ -181,7 +206,7 @@ proc_rsync(char *prog, char *bind_addr,
> err(1, NULL);
>
> for (;;) {
> - char *uri = NULL, *dst = NULL;
> + char *uri, *dst, *compdst;
> unsigned int id;
> pid_t pid;
> int st;
> @@ -209,7 +234,8 @@ proc_rsync(char *prog, char *bind_addr,
> for (i = 0; i < idsz; i++)
> if (ids[i].pid == pid)
> break;
> - assert(i < idsz);
> + if (!(i < idsz))
I would use
if (i >= idsz)
> + errx(1, "waitpid: %d unexpected", pid);
>
> if (!WIFEXITED(st)) {
> warnx("rsync %s terminated abnormally",
> @@ -261,6 +287,7 @@ proc_rsync(char *prog, char *bind_addr,
> /* Read host and module. */
> io_read_buf(b, &id, sizeof(id));
> io_read_str(b, &dst);
> + io_read_str(b, &compdst);
> io_read_str(b, &uri);
>
> ibuf_free(b);
> @@ -275,6 +302,7 @@ proc_rsync(char *prog, char *bind_addr,
>
> if (pid == 0) {
> char *args[32];
> + char *reldst;
>
> if (pledge("stdio exec", NULL) == -1)
> err(1, "pledge");
> @@ -295,6 +323,11 @@ proc_rsync(char *prog, char *bind_addr,
> args[i++] = "--address";
> args[i++] = (char *)bind_addr;
> }
> + if (compdst != NULL &&
> + (reldst = rsync_fixup_dest(dst, compdst)) != NULL) {
> + args[i++] = "--compare-dest";
> + args[i++] = reldst;
> + }
> args[i++] = uri;
> args[i++] = dst;
> args[i] = NULL;
> @@ -323,6 +356,7 @@ proc_rsync(char *prog, char *bind_addr,
> /* Clean up temporary values. */
>
> free(dst);
> + free(compdst);
> }
>
> /* No need for these to be hanging around. */
>
Re: rpki-client fix -n mode
On Thu, Jan 13, 2022 at 11:37:32AM +0100, Claudio Jeker wrote: > Since we push repository information over to the parser -n mode was broken > because in that case the TA repositories did not get sent. > This little diff fixes the problem. ok tb > > -- > :wq Claudio > > Index: repo.c > === > RCS file: /cvs/src/usr.sbin/rpki-client/repo.c,v > retrieving revision 1.20 > diff -u -p -r1.20 repo.c > --- repo.c11 Jan 2022 13:06:07 - 1.20 > +++ repo.c13 Jan 2022 10:28:24 - > @@ -1095,6 +1095,9 @@ ta_lookup(int id, struct tal *tal) > > rp->ta = ta_get(tal); > > + if (repo_state(rp) != REPO_LOADING) > + entityq_flush(&rp->queue, rp); > + > return rp; > } > >
rpki-client real cleanup before snapshot
This introduces a function remove_contents() which is implementing a basic
rm -r and uses it to clean the RRDP repository when downloading a
snapshot (especially after a delta failure). It also cleans out the temp
directory after a failure to fetch.
With the introduction of a validated cache this will become a bit simpler
since the temp dir will disapear. Still this is a simple step and reduces
the size of the validated cache patch.
--
:wq Claudio
Index: extern.h
===
RCS file: /cvs/src/usr.sbin/rpki-client/extern.h,v
retrieving revision 1.101
diff -u -p -r1.101 extern.h
--- extern.h11 Jan 2022 13:06:07 - 1.101
+++ extern.h13 Jan 2022 10:20:53 -
@@ -309,10 +309,11 @@ enum rrdp_msg {
RRDP_START,
RRDP_SESSION,
RRDP_FILE,
+ RRDP_CLEAR,
RRDP_END,
RRDP_HTTP_REQ,
RRDP_HTTP_INI,
- RRDP_HTTP_FIN
+ RRDP_HTTP_FIN,
};
/*
@@ -501,6 +502,7 @@ void proc_rrdp(int);
/* Repository handling */
int filepath_add(struct filepath_tree *, char *);
+voidrrdp_clear(unsigned int);
voidrrdp_save_state(unsigned int, struct rrdp_session *);
int rrdp_handle_file(unsigned int, enum publish_type, char *,
char *, size_t, char *, size_t);
Index: main.c
===
RCS file: /cvs/src/usr.sbin/rpki-client/main.c,v
retrieving revision 1.173
diff -u -p -r1.173 main.c
--- main.c 11 Jan 2022 13:06:07 - 1.173
+++ main.c 13 Jan 2022 10:20:04 -
@@ -631,6 +631,9 @@ rrdp_process(struct ibuf *b)
free(uri);
free(data);
break;
+ case RRDP_CLEAR:
+ rrdp_clear(id);
+ break;
default:
errx(1, "unexpected rrdp response");
}
Index: repo.c
===
RCS file: /cvs/src/usr.sbin/rpki-client/repo.c,v
retrieving revision 1.20
diff -u -p -r1.20 repo.c
--- repo.c 11 Jan 2022 13:06:07 - 1.20
+++ repo.c 13 Jan 2022 11:07:49 -
@@ -106,6 +106,7 @@ static SLIST_HEAD(, repo) repos = SLIST_
unsigned int repoid;
static struct rsyncrepo*rsync_get(const char *, int);
+static void remove_contents(char *);
/*
* Database of all file path accessed during a run.
@@ -788,6 +789,23 @@ fail:
}
/*
+ * Remove RRDP repo and start over.
+ */
+void
+rrdp_clear(unsigned int id)
+{
+ struct rrdprepo *rr;
+
+ rr = rrdp_find(id);
+ if (rr == NULL)
+ errx(1, "non-existant rrdp repo %u", id);
+
+ /* remove rrdp repository contents */
+ remove_contents(rr->basedir);
+ remove_contents(rr->temp);
+}
+
+/*
* Write a file into the temporary RRDP dir but only after checking
* its hash (if required). The function also makes sure that the file
* tracking is properly adjusted.
@@ -932,24 +950,6 @@ fail:
return 0;
}
-static void
-rrdp_clean_temp(struct rrdprepo *rr)
-{
- struct filepath *fp, *nfp;
- char *fn;
-
- filepath_free(&rr->deleted);
-
- RB_FOREACH_SAFE(fp, filepath_tree, &rr->added, nfp) {
- if ((fn = rrdp_filename(rr, fp->file, 1)) != NULL) {
- if (unlink(fn) == -1)
- warn("unlink %s", fn);
- free(fn);
- }
- filepath_put(&rr->added, fp);
- }
-}
-
/*
* RSYNC sync finished, either with or without success.
*/
@@ -981,10 +981,10 @@ rsync_finish(unsigned int id, int ok)
rr = rsync_find(id);
if (rr == NULL)
errx(1, "unknown rsync repo %u", id);
-
/* repository changed state already, ignore request */
if (rr->state != REPO_LOADING)
return;
+
if (ok) {
logx("%s: loaded from network", rr->basedir);
stats.rsync_repos++;
@@ -1016,15 +1016,15 @@ rrdp_finish(unsigned int id, int ok)
if (ok && rrdp_merge_repo(rr)) {
logx("%s: loaded from network", rr->notifyuri);
- rr->state = REPO_DONE;
stats.rrdp_repos++;
+ rr->state = REPO_DONE;
repo_done(rr, ok);
} else {
- rrdp_clean_temp(rr);
- stats.rrdp_fails++;
- rr->state = REPO_FAILED;
logx("%s: load from network failed, fallback to rsync",
rr->notifyuri);
+ stats.rrdp_fails++;
+ rr->state = REPO_FAILED;
+ remove_contents(rr->temp);
repo_done(rr, 0);
}
}
@@ -1414,4 +1414,49 @@ repo_free(void)
ta_free();
rrdp_free();
rsync_free();
+}
+
+static void
+remove_contents(char *base)
+{
+ char *argv[2] = { base, NULL };
+ FTS *fts;
rpki-client prepare to use rsync --compare-dest
This diff adds the code to pass --compare-dest to rsync. This will be
used once there is a valid cache and then the rsync repo will just act as
a delta on top.
Now --compare-dest is a bit strange as in the directory passed is relative
to the destination directory (last argument of rsync command). Because of
this the right amount of ../../../ need to be added to the compare-dest
path. rsync_fixup_dest() is taking care of that.
While there upgrade an assert to a proper error condition since it would
lead to an access beyond the ids array.
--
:wq Claudio
Index: main.c
===
RCS file: /cvs/src/usr.sbin/rpki-client/main.c,v
retrieving revision 1.173
diff -u -p -r1.173 main.c
--- main.c 11 Jan 2022 13:06:07 - 1.173
+++ main.c 13 Jan 2022 10:06:56 -
@@ -261,6 +261,7 @@ rsync_fetch(unsigned int id, const char
b = io_new_buffer();
io_simple_buffer(b, &id, sizeof(id));
io_str_buffer(b, local);
+ io_str_buffer(b, NULL);
io_str_buffer(b, uri);
io_close_buffer(&rsyncq, b);
}
Index: rsync.c
===
RCS file: /cvs/src/usr.sbin/rpki-client/rsync.c,v
retrieving revision 1.31
diff -u -p -r1.31 rsync.c
--- rsync.c 22 Dec 2021 09:35:14 - 1.31
+++ rsync.c 13 Jan 2022 10:08:46 -
@@ -99,6 +99,31 @@ rsync_base_uri(const char *uri)
return base_uri;
}
+/*
+ * The directory passed as --compare-dest needs to be relative to
+ * the destination directory. This function takes care of that.
+ */
+static char *
+rsync_fixup_dest(char *destdir, char *compdir)
+{
+ const char *dotdot = "../../../../../../"; /* should be enough */
+ int dirs = 1;
+ char *fn;
+ char c;
+
+ while ((c = *destdir++) != '\0')
+ if (c == '/')
+ dirs++;
+
+ if (dirs > 6)
+ /* too deep for us */
+ return NULL;
+
+ if ((asprintf(&fn, "%.*s%s", dirs * 3, dotdot, compdir)) == -1)
+ err(1, NULL);
+ return fn;
+}
+
static void
proc_child(int signal)
{
@@ -181,7 +206,7 @@ proc_rsync(char *prog, char *bind_addr,
err(1, NULL);
for (;;) {
- char *uri = NULL, *dst = NULL;
+ char *uri, *dst, *compdst;
unsigned int id;
pid_t pid;
int st;
@@ -209,7 +234,8 @@ proc_rsync(char *prog, char *bind_addr,
for (i = 0; i < idsz; i++)
if (ids[i].pid == pid)
break;
- assert(i < idsz);
+ if (!(i < idsz))
+ errx(1, "waitpid: %d unexpected", pid);
if (!WIFEXITED(st)) {
warnx("rsync %s terminated abnormally",
@@ -261,6 +287,7 @@ proc_rsync(char *prog, char *bind_addr,
/* Read host and module. */
io_read_buf(b, &id, sizeof(id));
io_read_str(b, &dst);
+ io_read_str(b, &compdst);
io_read_str(b, &uri);
ibuf_free(b);
@@ -275,6 +302,7 @@ proc_rsync(char *prog, char *bind_addr,
if (pid == 0) {
char *args[32];
+ char *reldst;
if (pledge("stdio exec", NULL) == -1)
err(1, "pledge");
@@ -295,6 +323,11 @@ proc_rsync(char *prog, char *bind_addr,
args[i++] = "--address";
args[i++] = (char *)bind_addr;
}
+ if (compdst != NULL &&
+ (reldst = rsync_fixup_dest(dst, compdst)) != NULL) {
+ args[i++] = "--compare-dest";
+ args[i++] = reldst;
+ }
args[i++] = uri;
args[i++] = dst;
args[i] = NULL;
@@ -323,6 +356,7 @@ proc_rsync(char *prog, char *bind_addr,
/* Clean up temporary values. */
free(dst);
+ free(compdst);
}
/* No need for these to be hanging around. */
rpki-client fix -n mode
Since we push repository information over to the parser -n mode was broken because in that case the TA repositories did not get sent. This little diff fixes the problem. -- :wq Claudio Index: repo.c === RCS file: /cvs/src/usr.sbin/rpki-client/repo.c,v retrieving revision 1.20 diff -u -p -r1.20 repo.c --- repo.c 11 Jan 2022 13:06:07 - 1.20 +++ repo.c 13 Jan 2022 10:28:24 - @@ -1095,6 +1095,9 @@ ta_lookup(int id, struct tal *tal) rp->ta = ta_get(tal); + if (repo_state(rp) != REPO_LOADING) + entityq_flush(&rp->queue, rp); + return rp; }
