Jarrod,
Thank you for the analysis and suggestions, one of us on the IBM xCAT team can attempt to rebuild xNBA with your suggestions to see if it performs better. Thomas, This effort may take some time, so I would suggest that you continue using xNBA 1.0.3 for now. I'll let you know when we have a candidate rpm available to test with. Thanks, Nate From: Jarrod Johnson <jjohns...@lenovo.com> To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net> Date: 06/03/2021 07:16 AM Subject: Re: [xcat-user] [External] Re: xCAT 2.16.2 new xNBA issue Yes, my hope is that someone will post and offer up an example new rpm to reflect the changes I suggest, and that future xCAT upgrades would be consistent with that. Unless I'm totally wrong and it is something fundamental to newer iPXE codebase, and that would be a different matter, but I'd give it about 90% chance that snponly.efi would be fine. -----Original Message----- From: Thomas HUMMEL <thomas.hum...@pasteur.fr> Sent: Thursday, June 3, 2021 7:12 AM To: xcat-user@lists.sourceforge.net Subject: Re: [xcat-user] [External] Re: xCAT 2.16.2 new xNBA issue Thanks a lot Jarrod. Well I guess indeed, for my use case the old xNBA seesm the way to go. I should have done it (and still can) more cleanly using the RPM. Problem is though that any xCAT upgrade would pull the new rpm. -- Thomas HUMMEL On 03/06/2021 12:57, Jarrod Johnson wrote: > The slowness may relate to the fact it would be initializing *every* nic in the system, regardless of the one that booted, it could also relate to features, e.g. HTTPS can in some case cause init to be painfully slow depending on what entropy source it latches onto, and will do this initialization even if never used. I haven't actually booted the xnba to see the behavior personally. > > My info about snponly.efi and ipxe.efi is intended mostly the benefit of whoever is building the xnba rpms. I'm a bit rusty and don't quite see how the build is currently managed though on that front. I suspect the only difference between iPXE and xNBA is the user class identifier. I've been mostly building for confluent, where we just use plain iPXE and if we do care about firmware ipxe missing features at some point, will use the feature block of iPXE requests to detect HTTP presence or absence instead of relying upon the user class identifier. > > Xnba.kpxe is for 'legacy' bios boot rather than uefi, ipxe would call it 'undionly.kpxe', and is not offered to UEFI boots. > > One theory I have on what could go wrong is that when you chainboot, then the UEFI network stack is concurrently active with whatever iPXE stack is doing. The iPXE codebase may be changing things at the same time the firmware's code is also changing things . So an update to the UEFI generic network stack may produce better compatibility with whatever the ipxe codebase is doing, even if ipxe is to some extent trying to directly use the device. Snponly.efi may be easier to live with, since all access to the hardware is arbitrated by a single set of code and there would be reduced potential for surprises. Perhaps steps are taken that *should* suppress the uefi network stack, but somehow don't work consistently (maybe because the older firmware, didn't implement them correctly, or an unexpected gap in operation, hard to speculate). Or it could simply be that non-boot portion of the intel firmware that the new driver interacts with had bugs. > > Again, I have only really seen this list activity to go on, so this is speculation without actually seeing the situation first hand. I think a rebuild of xnba-undi rpm with snponly.efi is in order, and if HTTPS is enabled, probably remove it for transfer size and initialization time benefits (the binary is tftp transferred, so security is not in the cards as a would-be http attack could simply freely modify the tftp data and change the cert validation freely). If I were targeting security, UEFI enrolled certificates with UEFI native HTTPS boot would be a better cross-vendor approach. > > -----Original Message----- > From: Thomas HUMMEL <thomas.hum...@pasteur.fr> > Sent: Thursday, June 3, 2021 5:24 AM > To: xcat-user@lists.sourceforge.net > Subject: Re: [xcat-user] [External] Re: xCAT 2.16.2 new xNBA issue > > Hello, > > got it. This could explain (ipxe.efi case) the long initializing time as well, right ? > > But in > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furld > efense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook.c > om%2F%3Furl%3Dhttp*3A*2F*2Fxcat.org*2Ffiles*2Fxcat*2Frepos*2Fyum*2Fxca > t-dep*2Frh8*2Fx86_64*2F%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.co > m*7C5111016cfe4c415c61bc08d926718d4f*7C5c7d0b28bdf8410caa934df372b1620 > 3*7C1*7C0*7C637583091425208346*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjA > wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsda > ta%3DofxojrtbeVSzj6OxsDgkPeoqf2uZvngki1dBe8b*2B2Ok*3D%26amp%3Breserved > %3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJQ!!JFdNOqOXpB6UZW0!6XUUnBKqB > TubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwTBpFCwcg%24&d > ata=04%7C01%7Cjjohnson2%40lenovo.com%7C2d457570211b4410a74408d92680a1b > 9%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C637583156188968584%7CUn > known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haW > wiLCJXVCI6Mn0%3D%7C1000&sdata=ezZQkiVhg6IDUEz%2F1%2BGzDxUpTdtVnmvm > ctE50BMN20c%3D&reserved=0 I only see > > xnba-undi-1.0.3-131028.noarch.rpm and xnba-undi-1.20.1-0.noarch.rpm > > So I guess you imply that if one wants eht snp version it has to built it himself, correct ? > > Also, what is xnba.kpxe ? > > What's strange though is that, for me it did or did not work across exact same nics (Intel X550 chipset) until I flashed the PXE and EFI Option ROM boot on the ones not working. That would rather happen with the snponly.efi case, wouldn't it ? > > Thanks for your help > > -- > Thomas HUMMEL > > > On 02/06/2021 20:14, Jarrod Johnson wrote: >> The iPXE codebase has code for things like UNDI ('legacy' bios >> generic network stack) and SNP/NII for UEFI. As far as code running >> targeting these, the nic vendor shouldn't matter, because they >> abstract away their differences in their uefi driver. >> >> The ipxe code also strives to port in specific drivers, that bypass >> the generic and target the network hardware more like linux would. >> iPXE does this so it could boot from usb key or actually get flashed >> and *be* the firmware driver of an adapter card. >> >> When building ipxe from source, you specify your target name: >> make bin-x864_64-efi/ipxe.efi >> That builds an ipxe with every driver bundled in and tries to use >> specific driver for each possible device. >> >> make bin-x86_64-efi/snponly.efi >> Builds a binary suitable only for chainloader, delegating the network >> hardware operation to the implementation provided by the firmware >> rather than trying to do direct driver binding. >> >> With default config options: >> jjohnson2@wirbelwind:~/devel/ipxe/src$ ls -lh bin-x86_64-efi/*.efi >> -rw-rw-r-- 1 jjohnson2 jjohnson2 1.1M Jun 2 14:13 >> bin-x86_64-efi/ipxe.efi >> -rw-rw-r-- 1 jjohnson2 jjohnson2 234K Jun 2 14:13 >> bin-x86_64-efi/snponly.efi The snponly.efi is much smaller (my >> production build is smaller still, I think my config drops some >> features that I don't use) >> >> --------------------------------------------------------------------- >> - >> -- >> *From:* Thomas HUMMEL <thomas.hum...@pasteur.fr> >> *Sent:* Wednesday, June 2, 2021 1:14 PM >> *To:* xcat-user@lists.sourceforge.net >> <xcat-user@lists.sourceforge.net> >> *Subject:* Re: [xcat-user] [External] Re: xCAT 2.16.2 new xNBA issue >> Hello, >> >> thanks for your answer. >> >> Let me get this straight: basically you're saying that xnba can >> either use ipxe.efi which would somehow user its own nic driver vs >> snponly.efi which would rely on the built-in nic driver or something like this ? >> >> I did indeed substitute xnba.efi with the one coming from the >> xnba-undi-1.0.3 package (just the file, which is a dirty fix I know). >> >> Thanks for your help >> >> -- >> TH >> >> On 02/06/2021 18:36, Jarrod Johnson wrote: >>> I decided to take a peek and note that the efi binary is pretty >>> large, so I guess it built ipxe.efi instead of 'snponly.efi', which >>> may complicate uefi boots (it at least means having to tftp down 10x >>> as much content before http can begin, which somewhat reduces the >>> point of ipxe chain booting). ipxe.efi will try to use the 'good' >>> drivers for every network device while snponly.efi will try to hook >>> the generic uefi protocols provided by the devices. >>> >>> When the 'good' drivers work, it generally does have somewhat better >>> performance, but I have generally better luck with the snponly.efi builds. >>> >>> Might want to roll back to >>> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fur >>> ldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlo >>> ok.com%2F%3Furl%3Dhttps*3A*2F*2Fxca__%3BJSUl!!JFdNOqOXpB6UZW0!6XUUnB >>> KqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRmMKionQ%24& >>> amp;data=04%7C01%7Cjjohnson2%40lenovo.com%7C2d457570211b4410a74408d9 >>> 2680a1b9%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C63758315618897 >>> 8592%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ >>> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GJAR4lkW0Co3%2Bn53lSV56 >>> HHiG%2Bcv070kmBYHPvRoHtc%3D&reserved=0 >>> t.org%2Ffiles%2Fxcat%2Frepos%2Fyum%2F2.16%2Fxcat-dep%2Fxnba-undi-1.0. >>> 3-131028.noarch.rpm&data=04%7C01%7Cjjohnson2%40lenovo.com%7C5111 >>> 0 >>> 16cfe4c415c61bc08d926718d4f%7C5c7d0b28bdf8410caa934df372b16203%7C1%7 >>> C >>> 0%7C637583091425208346%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL >>> C >>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QY%2BB >>> b >>> K3K6DyVA8%2B2xZxj9frwSgfT7ZupT%2BInIozW2%2FU%3D&reserved=0 >> < https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fur >> ldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outloo >> k.com%2F%3Furl%3Dhttps*3A*2F*2Furl__%3BJSUl!!JFdNOqOXpB6UZW0!6XUUnBKq >> BTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ%24& >> ;data=04%7C01%7Cjjohnson2%40lenovo.com%7C2d457570211b4410a74408d92680 >> a1b9%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C637583156188978592% >> 7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I >> k1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CDArldBoVPg6tn%2FZYT5S%2Fl3Ldw >> My5QGmnE%2F0qVkEXrg%3D&reserved=0 >> defense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook. >> com%2F%3Furl%3Dhttps*3A*2F*2Fxcat.org*2Ffiles*2Fxcat*2Frepos*2Fyum*2F >> 2 >> .16*2Fxcat-dep*2Fxnba-undi-1.0.3-131028.noarch.rpm%26amp%3Bdata%3D04* >> 7 >> C01*7Cjjohnson2*40lenovo.com*7C670c545b49d9431173a608d925ea22b8*7C5c7 >> d >> 0b28bdf8410caa934df372b16203*7C1*7C0*7C637582509812053894*7CUnknown*7 >> C >> TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXV >> C >> I6Mn0*3D*7C1000%26amp%3Bsdata%3DNMbStO8pBhneWbPy091jtS5LW7qtrax4kaDin >> c >> BYtuw*3D%26amp%3Breserved%3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU!!JF >> d >> NOqOXpB6UZW0!-Y0C_83SB3EOTw_sJofG7wQUpTDFxPGI_aaz3lCWw4AmC2ZGd1KGIY3H >> s >> iUf_MIOvG6ZCQ%24&data=04%7C01%7Cjjohnson2%40lenovo.com%7C5111016c >> f >> e4c415c61bc08d926718d4f%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C >> 6 >> 37583091425208346%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjo >> i >> V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hapVVnmQNalrU >> q 7iNu%2FKTwZquLTE6jn1TCDbA7UCTv0%3D&reserved=0> >> >>> < https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fu >>> rldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outl >>> ook.com%2F%3Furl%3Dhttps*3A*2F*2Fur__%3BJSUl!!JFdNOqOXpB6UZW0!6XUUnB >>> KqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRjRqQRmQ%24& >>> amp;data=04%7C01%7Cjjohnson2%40lenovo.com%7C2d457570211b4410a74408d9 >>> 2680a1b9%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C63758315618897 >>> 8592%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ >>> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BP4cw8jzjerBgu6wM2xQ5 >>> tdhCV%2BbN%2BK3moVS9vZ8Bg4%3D&reserved=0 >>> ldefense.com%2Fv3%2F__https%3A%2F%2Fxcat.org%2Ffiles%2Fxcat%2Frepos% >>> 2 >>> Fyum%2F2.16%2Fxcat-dep%2Fxnba-undi-1.0.3-131028.noarch.rpm__%3B!!JFd >>> N >>> OqOXpB6UZW0!6qpRG5NAI9z1J0vKAjYRLInh8G_Ce4PMXR9wecE-uPtkM38P9Arf76U- >>> o >>> 7jLMfjioNLUlQ%24&data=04%7C01%7Cjjohnson2%40lenovo.com%7C5111016 >>> c >>> fe4c415c61bc08d926718d4f%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0% >>> 7 >>> C637583091425208346%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ >>> I >>> joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0YzQGXG1r >>> w >>> dHeAqmiSUKqc%2BK600PqZwxkVl8yv5pZVY%3D&reserved=0 >> < https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fur >> ldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outloo >> k.com%2F%3Furl%3Dhttps*3A*2F*2Furl__%3BJSUl!!JFdNOqOXpB6UZW0!6XUUnBKq >> BTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ%24& >> ;data=04%7C01%7Cjjohnson2%40lenovo.com%7C2d457570211b4410a74408d92680 >> a1b9%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C637583156188978592% >> 7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I >> k1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CDArldBoVPg6tn%2FZYT5S%2Fl3Ldw >> My5QGmnE%2F0qVkEXrg%3D&reserved=0 >> defense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook. >> com%2F%3Furl%3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fxc >> a >> t.org*2Ffiles*2Fxcat*2Frepos*2Fyum*2F2.16*2Fxcat-dep*2Fxnba-undi-1.0. >> 3 >> -131028.noarch.rpm__*3B!!JFdNOqOXpB6UZW0!6qpRG5NAI9z1J0vKAjYRLInh8G_C >> e >> 4PMXR9wecE-uPtkM38P9Arf76U-o7jLMfjioNLUlQ*24%26amp%3Bdata%3D04*7C01*7 >> C >> jjohnson2*40lenovo.com*7C670c545b49d9431173a608d925ea22b8*7C5c7d0b28b >> d >> f8410caa934df372b16203*7C1*7C0*7C637582509812053894*7CUnknown*7CTWFpb >> G >> Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0 >> * >> 3D*7C1000%26amp%3Bsdata%3DDASb7ZFfbKRmlH*2BcrXAet806ZvuLhuVrGuPxiw9KS >> g >> A*3D%26amp%3Breserved%3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU >> l >> JQ!!JFdNOqOXpB6UZW0!-Y0C_83SB3EOTw_sJofG7wQUpTDFxPGI_aaz3lCWw4AmC2ZGd >> 1 >> KGIY3HsiUf_MIlFhVk5g%24&data=04%7C01%7Cjjohnson2%40lenovo.com%7C5 >> 1 >> 11016cfe4c415c61bc08d926718d4f%7C5c7d0b28bdf8410caa934df372b16203%7C1 >> % >> 7C0%7C637583091425218340%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi >> L >> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=S39Fzy >> D JRlsMPFe7MM15zHMjSS7vOLUrZ%2BidW5HooJo%3D&reserved=0>> for >> >>> now, might be also interesting to build an snponly.efi instead of >>> ipxe.efi for a test.. >>> >>> -------------------------------------------------------------------- >>> - >>> --- >>> *From:* Thomas HUMMEL <thomas.hum...@pasteur.fr> >>> *Sent:* Wednesday, June 2, 2021 10:52 AM >>> *To:* xcat-user@lists.sourceforge.net >>> <xcat-user@lists.sourceforge.net> >>> *Subject:* [External] Re: [xcat-user] xCAT 2.16.2 new xNBA issue So, >>> basically, for me: >>> >>> Supermicro H12DST-B with Intel 10G X550T (rev 01) and BIOS >>> configured in full UEFI mode with new xNBA: >>> >>> a) does not boot if FW 1.93 and PXE version 2.4.16 >>> >>> b) does boot if PXE upgraded to 2.4.44 but after what seems an >>> anormaly long time spent in that stage: >>> >>> >>> downloading NBP file : 10s >>> xNBA initializing devices : 15s >>> start of osimage fetch : 15s later >>> >>> whereas regardless of PXE version, with old xNBA systems boot and >>> spend max 1s in xNBA stage. >>> >>> What do you think about it ? >>> >>> Thanks for your help >>> >>> -- >>> Thomas HUMMEL >>> >>> On 02/06/2021 14:34, Thomas HUMMEL wrote: >>>> Hello, thanks for your answer. >>>> >>>> On 01/06/2021 22:42, Nathan A Besaw wrote: >>>> >>>>> xNBA 1.20.1 is almost identical to iPXE 1.20.1 with minimal >>>>> customization. >>>> >>>> I'm by no means a PXE expert : am I right assuming iPXE and/or xNBA >>>> use some PXE firmware built in into some EEPROM on the nic or board ? >>>> >>>> >>>>> Are there any differences in the BIOS boot mode settings between >>>>> the servers that are working and the servers that are failing? >>>>> Legacy mode vs Legacy + UEFI mode vs UEFI only mode? >>>> >>>> No. They both boot in full UEFI mode (no legacy, no dual). Some may >>>> have a different version but the issue occurs across the same BIOS version too. >>>> >>>> >>>>> Do you have better results if you downgrade xNBA to the previous version? >>>> >>>> I indeed manage to workaround the issue by savagely replacing >>>> xnba.efi by the one I extracted from the 2.15 branch, which I >>>> believe was already xnba-undi-1.0.3-131028.noarch.rpm >>>> >>>> Supermicro provided me as well with: >>>> >>>> - nic EEPROM new firmare (I did not use nor flash) >>>> - new PXE firmware which I flashed: this made the new xNBA works as >>>> expected. >>>> >>>> They stated that: >>>> >>>> Batch 1 was shipped with a different EEPROM and PXE option ROIM >>>> version then Batch 2 and 3. >>>> Batch one has FW 1.93 and PXE version 2.4.16 while batch 2 and 3 >>>> will have FW 2.2 and PXE version 2.4.44 >>>> >>>> I still have to test flashing this new PXE fw to a batch2 node >>>> (which worked with the new xNBA but spent >30s in it) and see if it >>>> fixes the slow initialization. >>>> >>>> Thanks >>>> >>>> -- >>>> Thomas HUMMEL >>>> >>>> >>>> _______________________________________________ >>>> xCAT-user mailing list >>>> xCAT-user@lists.sourceforge.net >>>> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fu >>>> rldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.out >>>> look.com%2F%3Furl%3Dhttps*3A*2F*2Fur__%3BJSUl!!JFdNOqOXpB6UZW0!6XUU >>>> nBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRjRqQRmQ% >>>> 24&data=04%7C01%7Cjjohnson2%40lenovo.com%7C2d457570211b4410a744 >>>> 08d92680a1b9%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C637583156 >>>> 188978592%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM >>>> zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BP4cw8jzjerBgu >>>> 6wM2xQ5tdhCV%2BbN%2BK3moVS9vZ8Bg4%3D&reserved=0 >>>> ldefense.com%2Fv3%2F__https%3A%2F%2Flists.sourceforge.net%2Flists%2 >>>> F >>>> listinfo%2Fxcat-user__%3B!!JFdNOqOXpB6UZW0!7CP7jiJu_SapsF7A5b-99TpS >>>> 0 >>>> lU01tKQJu5BHi7YpIvbH3ekgOyxSZXEftL6i0z5gH4nOA%24&data=04%7C01%7 >>>> C >>>> jjohnson2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5c7d0b2 >>>> 8 >>>> bdf8410caa934df372b16203%7C1%7C0%7C637583091425218340%7CUnknown%7CT >>>> W >>>> FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXV >>>> C >>>> I6Mn0%3D%7C1000&sdata=Hx%2FepCgCuSbKBQfiJi1BlEhgY9KQJZjR7drqFNA >>>> s >>>> zNc%3D&reserved=0 >> < https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fur >> ldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outloo >> k.com%2F%3Furl%3Dhttps*3A*2F*2Furl__%3BJSUl!!JFdNOqOXpB6UZW0!6XUUnBKq >> BTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ%24& >> ;data=04%7C01%7Cjjohnson2%40lenovo.com%7C2d457570211b4410a74408d92680 >> a1b9%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C637583156188978592% >> 7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I >> k1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CDArldBoVPg6tn%2FZYT5S%2Fl3Ldw >> My5QGmnE%2F0qVkEXrg%3D&reserved=0 >> defense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook. >> com%2F%3Furl%3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fli >> s >> ts.sourceforge.net*2Flists*2Flistinfo*2Fxcat-user__*3B!!JFdNOqOXpB6UZ >> W >> 0!7CP7jiJu_SapsF7A5b-99TpS0lU01tKQJu5BHi7YpIvbH3ekgOyxSZXEftL6i0z5gH4 >> n >> OA*24%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.com*7C670c545b49d94 >> 3 >> 1173a608d925ea22b8*7C5c7d0b28bdf8410caa934df372b16203*7C1*7C0*7C63758 >> 2 >> 509812063820*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2lu >> M >> zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3DMIQ*2BZeb31p >> Z >> 1X*2Ba*2FarIbGWyh8*2FoxUMyj3f7H9RLu7rE*3D%26amp%3Breserved%3D0__%3BJS >> U >> lJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!JFdNOqOXpB6UZW0!-Y0C_83SB3EOTw >> _ >> sJofG7wQUpTDFxPGI_aaz3lCWw4AmC2ZGd1KGIY3HsiUf_MIVV0TrcQ%24&data=0 >> 4 >> %7C01%7Cjjohnson2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5 >> c >> 7d0b28bdf8410caa934df372b16203%7C1%7C0%7C637583091425218340%7CUnknown >> % >> 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ >> X >> VCI6Mn0%3D%7C1000&sdata=LiTRzrueLyI%2FYZU3qWTIcRM3opt5bWj2oTAgYjU >> q >> 2hA%3D&reserved=0> >> >>> < https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fu >>> rldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outl >>> ook.com%2F%3Furl%3Dhttps*3A*2F*2Fur__%3BJSUl!!JFdNOqOXpB6UZW0!6XUUnB >>> KqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRjRqQRmQ%24& >>> amp;data=04%7C01%7Cjjohnson2%40lenovo.com%7C2d457570211b4410a74408d9 >>> 2680a1b9%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C63758315618897 >>> 8592%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ >>> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BP4cw8jzjerBgu6wM2xQ5 >>> tdhCV%2BbN%2BK3moVS9vZ8Bg4%3D&reserved=0 >>> ldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlo >>> o >>> k.com%2F%3Furl%3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2 >>> F >>> lists.sourceforge.net*2Flists*2Flistinfo*2Fxcat-user__*3B!!JFdNOqOXp >>> B >>> 6UZW0!7CP7jiJu_SapsF7A5b-99TpS0lU01tKQJu5BHi7YpIvbH3ekgOyxSZXEftL6i0 >>> z >>> 5gH4nOA*24%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.com*7Cf83e67c >>> 8 >>> 7894408a53e708d925d63f3e*7C5c7d0b28bdf8410caa934df372b16203*7C1*7C0* >>> 7 >>> C637582424394899812*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ >>> I >>> joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3DMv2 >>> 6 >>> iG1vAmV6VkL9T2y9y7mCw2HynRccrgLEmkI7qPE*3D%26amp%3Breserved%3D0__%3B >>> J >>> SUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU!!JFdNOqOXpB6UZW0!6qpRG5NAI9z1J0vK >>> A >>> jYRLInh8G_Ce4PMXR9wecE-uPtkM38P9Arf76U-o7jLMfhpT0Lceg%24&data=04 >>> % >>> 7C01%7Cjjohnson2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5 >>> c >>> 7d0b28bdf8410caa934df372b16203%7C1%7C0%7C637583091425218340%7CUnknow >>> n >>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL >>> C >>> JXVCI6Mn0%3D%7C1000&sdata=lWRVCh0Tnan8ZQyNcC4vp%2BahO6nkEl51pOO6 >>> t >>> 1Z1w50%3D&reserved=0 >> < https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fur >> ldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outloo >> k.com%2F%3Furl%3Dhttps*3A*2F*2Furl__%3BJSUl!!JFdNOqOXpB6UZW0!6XUUnBKq >> BTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ%24& >> ;data=04%7C01%7Cjjohnson2%40lenovo.com%7C2d457570211b4410a74408d92680 >> a1b9%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C637583156188978592% >> 7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I >> k1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CDArldBoVPg6tn%2FZYT5S%2Fl3Ldw >> My5QGmnE%2F0qVkEXrg%3D&reserved=0 >> defense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook. >> com%2F%3Furl%3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fap >> c >> 01.safelinks.protection.outlook.com*2F*3Furl*3Dhttps*3A*2F*2Furldefen >> s >> e.com*2Fv3*2F__https*3A*2F*2Flists.sourceforge.net*2Flists*2Flistinfo >> * >> 2Fxcat-user__*3B!!JFdNOqOXpB6UZW0!7CP7jiJu_SapsF7A5b-99TpS0lU01tKQJu5 >> B >> Hi7YpIvbH3ekgOyxSZXEftL6i0z5gH4nOA*24*26amp*3Bdata*3D04*7C01*7Cjjohns >> o >> n2*40lenovo.com*7Cf83e67c87894408a53e708d925d63f3e*7C5c7d0b28bdf8410c >> a >> a934df372b16203*7C1*7C0*7C637582424394899812*7CUnknown*7CTWFpbGZsb3d8 >> e >> yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1 >> 0 >> 00*26amp*3Bsdata*3DMv26iG1vAmV6VkL9T2y9y7mCw2HynRccrgLEmkI7qPE*3D*26a >> m >> p*3Breserved*3D0__*3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU!!JFdNOqOXpB6 >> U >> ZW0!6qpRG5NAI9z1J0vKAjYRLInh8G_Ce4PMXR9wecE-uPtkM38P9Arf76U-o7jLMfhpT >> 0 >> Lceg*24%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.com*7C670c545b49d >> 9 >> 431173a608d925ea22b8*7C5c7d0b28bdf8410caa934df372b16203*7C1*7C0*7C637 >> 5 >> 82509812063820*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2 >> l >> uMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3DW2RnPg6*2F >> g >> LMHOeUd*2BIWpAOcpR8L611KCet3RMfR9QSc*3D%26amp%3Breserved%3D0__%3BJSUl >> J >> SUlJSUlJSUqKioqKioqKioqKioqJSUlKioqKioqKioqKioqJSUlKiUlJSUlJSUlJSUlJS >> U >> lJSUlJSUl!!JFdNOqOXpB6UZW0!-Y0C_83SB3EOTw_sJofG7wQUpTDFxPGI_aaz3lCWw4 >> A >> mC2ZGd1KGIY3HsiUf_MIx-aW5dA%24&data=04%7C01%7Cjjohnson2%40lenovo. >> c >> om%7C5111016cfe4c415c61bc08d926718d4f%7C5c7d0b28bdf8410caa934df372b16 >> 2 >> 03%7C1%7C0%7C637583091425228336%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wL >> j >> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata >> = ta6B5GOnTfihdGlDxWtSRSEXO93v5VltKvkqjexE0XE%3D&reserved=0>> >> >>> >>>> >>> >>> >>> _______________________________________________ >>> xCAT-user mailing list >>> xCAT-user@lists.sourceforge.net >>> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fur >>> ldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlo >>> ok.com%2F%3Furl%3Dhttps*3A*2F*2Flis__%3BJSUl!!JFdNOqOXpB6UZW0!6XUUnB >>> KqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwT28Xg0FQ%24& >>> amp;data=04%7C01%7Cjjohnson2%40lenovo.com%7C2d457570211b4410a74408d9 >>> 2680a1b9%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C63758315618897 >>> 8592%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ >>> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8VzpvrqSkjkrwGK6%2FeI7V >>> Sss1hqk7w2s7vbVAat%2FGMk%3D&reserved=0 >>> ts.sourceforge.net%2Flists%2Flistinfo%2Fxcat-user&data=04%7C01%7 >>> C >>> jjohnson2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5c7d0b28 >>> b >>> df8410caa934df372b16203%7C1%7C0%7C637583091425228336%7CUnknown%7CTWF >>> p >>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 >>> M >>> n0%3D%7C1000&sdata=XUCmt%2F4MAP6cVf5TVLjDWbCRsEyB%2FJ%2FVsjfv4XV >>> v >>> X%2FE%3D&reserved=0 >> < https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fur >> ldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outloo >> k.com%2F%3Furl%3Dhttps*3A*2F*2Furl__%3BJSUl!!JFdNOqOXpB6UZW0!6XUUnBKq >> BTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ%24& >> ;data=04%7C01%7Cjjohnson2%40lenovo.com%7C2d457570211b4410a74408d92680 >> a1b9%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C637583156188978592% >> 7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I >> k1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CDArldBoVPg6tn%2FZYT5S%2Fl3Ldw >> My5QGmnE%2F0qVkEXrg%3D&reserved=0 >> defense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook. >> com%2F%3Furl%3Dhttps*3A*2F*2Flists.sourceforge.net*2Flists*2Flistinfo >> * >> 2Fxcat-user%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.com*7C670c545 >> b >> 49d9431173a608d925ea22b8*7C5c7d0b28bdf8410caa934df372b16203*7C1*7C0*7 >> C >> 637582509812063820*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIj >> o >> iV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3D0CSRpb >> q >> *2By49yGyAuyvlRxvYaKdzwoSa0YD69vJAQNIk*3D%26amp%3Breserved%3D0__%3BJS >> U >> lJSUlJSUlJSUlJSUlJSUlJSU!!JFdNOqOXpB6UZW0!-Y0C_83SB3EOTw_sJofG7wQUpTD >> F >> xPGI_aaz3lCWw4AmC2ZGd1KGIY3HsiUf_MLo7YZbdg%24&data=04%7C01%7Cjjoh >> n >> son2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5c7d0b28bdf841 >> 0 >> caa934df372b16203%7C1%7C0%7C637583091425228336%7CUnknown%7CTWFpbGZsb3 >> d >> 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7 >> C >> 1000&sdata=3nup2jzmVhmFM5%2F7tCL26rN%2FWmZTjPqDhuypLjlydxA%3D& >> ; >> reserved=0> >> >>> < https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fu >>> rldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outl >>> ook.com%2F%3Furl%3Dhttps*3A*2F*2Fur__%3BJSUl!!JFdNOqOXpB6UZW0!6XUUnB >>> KqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRjRqQRmQ%24& >>> amp;data=04%7C01%7Cjjohnson2%40lenovo.com%7C2d457570211b4410a74408d9 >>> 2680a1b9%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C63758315618898 >>> 8579%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ >>> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qkfAdnDy6bZygxx1RxHpLeh >>> zOJrxmNYDM3CbFDPOkl8%3D&reserved=0 >>> ldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlo >>> o >>> k.com%2F%3Furl%3Dhttps*3A*2F*2Flists.sourceforge.net*2Flists*2Flisti >>> n >>> fo*2Fxcat-user%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.com*7Cf83 >>> e >>> 67c87894408a53e708d925d63f3e*7C5c7d0b28bdf8410caa934df372b16203*7C1* >>> 7 >>> C0*7C637582424394899812*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi >>> L >>> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3 >>> D >>> 1*2BBbHF*2Fq9PcNucuoI32A1qlGVfsV2yKX*2Btz80loBUj8*3D%26amp%3Breserve >>> d >>> %3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJQ!!JFdNOqOXpB6UZW0!6qpRG5NAI9z >>> 1 >>> J0vKAjYRLInh8G_Ce4PMXR9wecE-uPtkM38P9Arf76U-o7jLMfi7F465LQ%24&da >>> t
_______________________________________________ xCAT-user mailing list xCAT-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xcat-user
