On Thursday, March 3, 2022 at 11:42:54 AM UTC+1 Konstantin Khomoutov wrote:
> On Wed, Mar 02, 2022 at 06:03:46PM +0000, Skybuck Flying wrote: > > >> Search for "4GB" in this blog post [1] which discusses the GfW 2.35 > release > >> notes. > >> > >> 1. https://github.blog/2022-01-24-highlights-from-git-2-35/ > > > Thanks for the update, however I don't see how this could effect git > > checkout of linux kernel, since most linux source files are only a few > > kilobytes... > > Because Git has several data storage/transfer optimization tricks. One of > them > is using of so-called "pack files" (basically that's the behind-the-scenes > mechanism that allows Git to remain fast and have modest storage > requirements > while its data model states that each commit is a snapshot of the whole > project). Pack files are used for both storing stuff on the filesystem, > and > for data transfer; they basically are gzipped archives with added index > files > for fast random access to the contained data. > Depends on how these pack files are used, if only parts of them are read/written then it does not need to be a problem :) Also I did not see any big spikes in memory consumption, at least when I was paying attention. > > My take is that processing of such a pack file hits some limit of the > current > GfW implementation, which has nothing to do with the OS limits (given that > up-to-date API calls are used). > What is GfW ? Firefox also fails regularly when many tabs open, it only does this when pagefile.sys is off. So as far as I am concerned it clearly has something to do with the memory system in windows which is probably less well tested when pagefile.sys is off because most systems/default installation have a pagefile.sys, therefore I recommend any serious programmer working on serious project to turn off pagefile.sys or swap space and observe the memory system ! I am sure you will be surprised ! =D > > While I think such problems were actually solved some time ago, some may > still > remain. In any case, it's hard to say something w/o proper bug report - > either > in the project's tracker of at least on the git-for-windows mailing list. > Why would that make any difference ? Are you hoping for developer feedback on this issue ? I did mention the error message but in case you missed it here it is one more time: fatal: Out of memory, malloc failed (tried to allocate 214601 bytes) fatal: index-pack failed So from this I can see two things: 1. It was trying to allocate a 214 kilobyte block which is quite large and makes the fragmentation hypothesis somewhat believable since windows uses 4 kilobyte pages as far as I know and these are stored in "lists". So to forfill this request it needs 214601 / 4096 = 53 pages. Now maybe a statiscian can somehow calculate the chance of 53 consequetive pages being present without any gaps/missing ones, especially when the system is under heavy use trying to create many little C files or whatever it was doing. >From what I remember it was resolving deltas... so it seems git was "replaying" changes and these could be made out of many many many little changes, hence this makes the memory fragmentation hypothesis believeable. Also what kind of file would need 214 kilobytes ? Perhaps linux has a file of that size, maybe some documentation or so, though I have also seen big lookup tables in other software like bitcoin 256psek or something and pascalcoin and regex/unicode lookup table which is slowing down or freezing analysis tools so this is not good. 2. Indeed it's doing something with an index-pack file... so this is probably what you based your analysis on so you did read the bug/error report that I posted :) So this could mean it was extracting something or maybe indeed packing something up, but it wasn't that big... and it was computing some index file. For now the easy assumption is to assume the system simply run out of memory, but this is a suspicious/somewhat bad conclusion because the chart in the video does not really back that hypothesis up, and also as far as I can re-call the browser was not crashing and seemed to be doing fine, now maybe it simply had enough memory to do it's thing, but usually the browser can suddenly take up a lot of memory and crash quite easily... then again... since git is using many little files... maybe it just hit it sooner and is more sensitive to out of memory or something, though it could also be memory fragmentation or a combination. For now the memory fragmentation or in combination with system cache taking up precious memory is believeable because other people have noticed strange behaviour with windows 7. > By the way, do you use a 64-bit install? If, for some reason, you're using > a > 32-bit version, the limit of circa 4 GB (actaually lower) will be > "native". > > new@new-PC MINGW64 / $ file $( which git ) /mingw64/bin/git: PE32+ executable (console) x86-64, for MS Windows new@new-PC MINGW64 / $ git --version --build-options git version 2.29.2.windows.2 cpu: x86_64 built from commit: 3464b98ce6803c98bf8fb34390cd150d66e4a0d3 sizeof-long: 4 sizeof-size_t: 8 shell-path: /bin/sh Something confusing and conflicting information, however if a 64 bit version is available I usually download the 64 bit version. The OS is 64 bit so it doesn't really make sense to run 32 bit software and artificially limit it's capabilities. However I did not make git so maybe deep inside git it might be using some 32 bit components and/or limitations, this I don't know. The PE32+ statement is somewhat confusing, but I guess it means an extended version of PE32 and is therefore PE64 ? or is this a wrong assumption/guess ? At least mingw seems to be 64 bit, x86_64 seems to indicate a 64 bit version of x86 called x64... sizeof-long: 4 is a bit worriesome on delphi this would be 8, not sure what C defines as a long, but a long long is probably 8, so a long would be 4. size_t is something I don't understand and don't care about lol. So all in all not too terribly clear, but the 64 says enough for me ! ;) So 64 bit it is most likely ! ;) Bye, Skybuck =D -- You received this message because you are subscribed to the Google Groups "Git for human beings" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/git-users/5b4d4a0b-3d2b-473f-90f0-7d691d687696n%40googlegroups.com.
