It would be nice if GIT would respect the following limits: new@new-PC MINGW64 / $ ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited open files (-n) 256 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 2036 cpu time (seconds, -t) unlimited max user processes (-u) 256 virtual memory (kbytes, -v) unlimited
new@new-PC MINGW64 / $ Then I could try and set these limits, however I fear if I set them now it will limit GIT even more and it will lead to even sooner crashes... though it's worth a try to see what happens. Right now I have no time to experiment with this, but could be an idea for the future. Setting limitations for GIT itself does not seem to be well documented or at least google don't take me there, I found this via google: Not sure if this will help or what it does exactly: git config --global pack.windowMemory "100m" git config --global pack.packSizeLimit "100m" git config --global pack.threads "1" How this is related to checkout I don't know, could try it out, but kinda hate to try things out and waste my time if it doesn't work. Things like this should be tested by git developers. Here is official documentation, many commands and settings to go through, low on time right now: https://git-scm.com/docs/git-config Searching for limit gives: core.packedGitLimit Maximum number of bytes to map simultaneously into memory from pack files. If Git needs to access more than this many bytes at once to complete an operation it will unmap existing regions to reclaim virtual address space within the process. Default is 256 MiB on 32 bit platforms and 32 TiB (effectively unlimited) on 64 bit platforms. This should be reasonable for all users/operating systems, except on the largest projects. You probably do not need to adjust this value. Common unit suffixes of *k*, *m*, or *g* are supported. core.deltaBaseCacheLimit Maximum number of bytes per thread to reserve for caching base objects that may be referenced by multiple deltified objects. By storing the entire decompressed base objects in a cache Git is able to avoid unpacking and decompressing frequently used base objects multiple times. Default is 96 MiB on all platforms. This should be reasonable for all users/operating systems, except on the largest projects. You probably do not need to adjust this value. Common unit suffixes of *k*, *m*, or *g* are supported. core.bigFileThreshold Files larger than this size are stored deflated, without attempting delta compression. Storing large files without delta compression avoids excessive memory usage, at the slight expense of increased disk usage. Additionally files larger than this size are always treated as binary. Default is 512 MiB on all platforms. This should be reasonable for most projects as source code and other text files can still be delta compressed, but larger binary media files won’t be. Common unit suffixes of *k*, *m*, or *g* are supported. THIS COULD BE INTERESTING: core.sparseCheckout Enable "sparse checkout" feature. See git-sparse-checkout[1] <https://git-scm.com/docs/git-sparse-checkout> for more information. core.sparseCheckoutCone Enables the "cone mode" of the sparse checkout feature. When the sparse-checkout file contains a limited set of patterns, then this mode provides significant performance advantages. See git-sparse-checkout[1] <https://git-scm.com/docs/git-sparse-checkout> for more information. iff.statGraphWidth Limit the width of the graph part in --stat output. If set, applies to all commands generating --stat output except format-patch. diff.renameLimit The number of files to consider in the exhaustive portion of copy/rename detection; equivalent to the *git diff* option -l. If not set, the default value is currently 1000. This setting has no effect if rename detection is turned off. fastimport.unpackLimit If the number of objects imported by git-fast-import[1] <https://git-scm.com/docs/git-fast-import> is below this limit, then the objects will be unpacked into loose object files. However if the number of imported objects equals or exceeds this limit then the pack will be stored as a pack. Storing the pack from a fast-import can make the import operation complete faster, especially on slow filesystems. If not set, the value of transfer.unpackLimit is used instead. gc.auto When there are approximately more than this many loose objects in the repository, git gc --auto will pack them. Some Porcelain commands use this command to perform a light-weight garbage collection from time to time. The default value is 6700. Setting this to 0 disables not only automatic packing based on the number of loose objects, but any other heuristic git gc --auto will otherwise use to determine if there’s work to do, such as gc.autoPackLimit. gc.autoPackLimit When there are more than this many packs that are not marked with *.keep file in the repository, git gc --auto consolidates them into one larger pack. The default value is 50. Setting this to 0 disables it. Setting gc.auto to 0 will also disable this. See the gc.bigPackThreshold configuration variable below. When in use, it’ll affect how the auto pack limit works. gc.bigPackThreshold If non-zero, all packs larger than this limit are kept when git gc is run. This is very similar to --keep-largest-pack except that all packs that meet the threshold are kept, not just the largest pack. Defaults to zero. Common unit suffixes of *k*, *m*, or *g* are supported. Note that if the number of kept packs is more than gc.autoPackLimit, this configuration variable is ignored, all packs except the base pack will be repacked. After this the number of packs should go below gc.autoPackLimit and gc.bigPackThreshold should be respected again. If the amount of memory estimated for git repack to run smoothly is not available and gc.bigPackThreshold is not set, the largest pack will also be excluded (this is the equivalent of running git gc with --keep-largest-pack ). http.postBuffer Maximum size in bytes of the buffer used by smart HTTP transports when POSTing data to the remote system. For requests larger than this buffer size, HTTP/1.1 and Transfer-Encoding: chunked is used to avoid creating a massive pack file locally. Default is 1 MiB, which is sufficient for most requests. Note that raising this limit is only effective for disabling chunked transfer encoding and therefore should be used only where the remote server or a proxy only supports HTTP/1.0 or is noncompliant with the HTTP standard. Raising this is not, in general, an effective solution for most push problems, but can increase memory consumption significantly since the entire buffer is allocated even for small pushes. http.lowSpeedLimit, http.lowSpeedTime If the HTTP transfer speed is less than *http.lowSpeedLimit* for longer than *http.lowSpeedTime* seconds, the transfer is aborted. Can be overridden by the GIT_HTTP_LOW_SPEED_LIMIT and GIT_HTTP_LOW_SPEED_TIME environment variables. merge.renameLimit The number of files to consider in the exhaustive portion of rename detection during a merge. If not specified, defaults to the value of diff.renameLimit. If neither merge.renameLimit nor diff.renameLimit are specified, currently defaults to 7000. This setting has no effect if rename detection is turned off. pack.windowMemory The maximum size of memory that is consumed by each thread in git-pack-objects[1] <https://git-scm.com/docs/git-pack-objects> for pack window memory when no limit is given on the command line. The value can be suffixed with "k", "m", or "g". When left unconfigured (or set explicitly to 0), there will be no limit. pack.deltaCacheSize The maximum memory in bytes used for caching deltas in git-pack-objects[1] <https://git-scm.com/docs/git-pack-objects> before writing them out to a pack. This cache is used to speed up the writing object phase by not having to recompute the final delta result once the best match for all objects is found. Repacking large repositories on machines which are tight with memory might be badly impacted by this though, especially if this cache pushes the system into swapping. A value of 0 means no limit. The smallest size of 1 byte may be used to virtually disable this cache. Defaults to 256 MiB. pack.deltaCacheLimit The maximum size of a delta, that is cached in git-pack-objects[1] <https://git-scm.com/docs/git-pack-objects>. This cache is used to speed up the writing object phase by not having to recompute the final delta result once the best match for all objects is found. Defaults to 1000. Maximum value is 65535. pack.packSizeLimit The maximum size of a pack. This setting only affects packing to a file when repacking, i.e. the git:// protocol is unaffected. It can be overridden by the --max-pack-size option of git-repack[1] <https://git-scm.com/docs/git-repack>. Reaching this limit results in the creation of multiple packfiles. Note that this option is rarely useful, and may result in a larger total on-disk size (because Git will not store deltas between packs), as well as worse runtime performance (object lookup within multiple packs is slower than a single pack, and optimizations like reachability bitmaps cannot cope with multiple packs). If you need to actively run Git using smaller packfiles (e.g., because your filesystem does not support large files), this option may help. But if your goal is to transmit a packfile over a medium that supports limited sizes (e.g., removable media that cannot store the whole repository), you are likely better off creating a single large packfile and splitting it using a generic multi-volume archive tool (e.g., Unix split). The minimum size allowed is limited to 1 MiB. The default is unlimited. Common unit suffixes of *k*, *m*, or *g* are supported. receive.unpackLimit If the number of objects received in a push is below this limit then the objects will be unpacked into loose object files. However if the number of received objects equals or exceeds this limit then the received pack will be stored as a pack, after adding any missing delta bases. Storing the pack from a push can make the push operation complete faster, especially on slow filesystems. If not set, the value of transfer.unpackLimit is used instead. receive.maxInputSize If the size of the incoming pack stream is larger than this limit, then git-receive-pack will error out, instead of accepting the pack file. If not set or set to 0, then the size is unlimited. status.renameLimit The number of files to consider when performing rename detection in git-status[1] <https://git-scm.com/docs/git-status> and git-commit[1] <https://git-scm.com/docs/git-commit>. Defaults to the value of diff.renameLimit. transfer.unpackLimit When fetch.unpackLimit or receive.unpackLimit are not set, the value of this variable is used instead. The default value is 100. uploadpackfilter.<filter>.allow Explicitly allow or ban the object filter corresponding to <filter>, where <filter> may be one of: blob:none, blob:limit, object:type, tree, sparse:oid, or combine. If using combined filters, both combine and all of the nested filter kinds must be allowed. Defaults to uploadpackfilter.allow. There are apperently many limits that GIT could run into. It would be very usefull if GIT could display which limit was hit and caused the failure/error. Ofcourse it would be better if GIT tries to function within these limitations so it does not error. Bye for now, Skybuck. On Thursday, December 2, 2021 at 6:01:36 AM UTC+1 skybuck2000 wrote: > Hello, > > I would like to report that GIT failed to checkout the linux kernel on a 6 > GB system. > > For now I believe it ran out of memory. I only tried once. > > PAGEFILE.SYS on this system was disabled. > > Apperently GIT relies on PAGEFILE.SYS to cover any out of memory > situations. > > This kinda sucks. > > My recommendation is to disable PAGEFILE.SYS on your system, or try > removing some RAM chips. > > And then try and checkout the linux kernel yourself to see how GIT fails > to checkout. > > Also displaying a LOG seemed to consume also a lot of RAM. > > I am not exactly sure why GIT needs so much memory. Perhaps it's a > unzipping issue or a delta-ing issue, not sure. > > But it would be nice if GIT can do it's operations in > batches/chunks/pieces of memory. > > Gradually so it can handle any database size. Right now it seems to be > limited to the ammount of system RAM/virtual memory available. > > Bye for now, > Skybuck. > > > -- You received this message because you are subscribed to the Google Groups "Git for human beings" group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/git-users/44672e27-cca5-4c80-bfca-f38d967432fdn%40googlegroups.com.
