[PATCH 1/1] git-p4: auto-delete named temporary file
From: "Philip.McGraw"
Take new approach using the NamedTemporaryFile()
file-like object as input to the ZipFile() which
auto-deletes after implicit close leaving with scope.
Original code produced double-open problems on Windows
platform from using already open NamedTemporaryFile()
generated filename instead of object.
Thanks to Andrey for patiently suggesting several
iterations on this change for avoiding exceptions!
Also print error details after resulting IOError to make
debugging cause of exception less mysterious when it has
nothing to do with "git version recent enough."
Signed-off-by: Philip.McGraw
---
git-p4.py | 13 ++---
1 file changed, 6 insertions(+), 7 deletions(-)
diff --git a/git-p4.py b/git-p4.py
index c71a6832e2..33bdb14fd1 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -1160,13 +1160,11 @@ def exceedsLargeFileThreshold(self, relPath, contents):
if contentsSize <=
gitConfigInt('git-p4.largeFileCompressedThreshold'):
return False
contentTempFile = self.generateTempFile(contents)
-compressedContentFile =
tempfile.NamedTemporaryFile(prefix='git-p4-large-file', delete=False)
-zf = zipfile.ZipFile(compressedContentFile.name, mode='w')
-zf.write(contentTempFile, compress_type=zipfile.ZIP_DEFLATED)
-zf.close()
-compressedContentsSize = zf.infolist()[0].compress_size
+compressedContentFile =
tempfile.NamedTemporaryFile(prefix='git-p4-large-file', delete=True)
+with zipfile.ZipFile(compressedContentFile, mode='w') as zf:
+zf.write(contentTempFile, compress_type=zipfile.ZIP_DEFLATED)
+compressedContentsSize = zf.infolist()[0].compress_size
os.remove(contentTempFile)
-os.remove(compressedContentFile.name)
if compressedContentsSize >
gitConfigInt('git-p4.largeFileCompressedThreshold'):
return True
return False
@@ -3514,8 +3512,9 @@ def importHeadRevision(self, revision):
self.updateOptionDict(details)
try:
self.commit(details,
self.extractFilesFromCommit(details), self.branch)
- except IOError:
+ except IOError as err:
print("IO error with git fast-import. Is your git version
recent enough?")
+print("IO error details: {}".format(err))
print(self.gitError.read())
def openStreams(self):
--
gitgitgadget
Keep your secrets safe!
Hello! I am a representative of the ChaosCC hacker group. In the period from 23/06/2019 to 11/08/2019 we got access to your account git@vger.kernel.org by hacking one of the domain.com mail servers. Your pass for above account on moment of hack was: 28121971 You already changed the password? Sumptuously! But my program fixes this every time. And every time I know your new password! Using access to your account, it turned out to be easy to infect the OS of your device. At the moment, all your contacts are known to us. We also have access to your messengers and to your correspondence. All this information is already stored with us. We are also aware of your intimate adventures on the Internet. We know that you adore adult sites and we know about your sexual addictions. You have a very interesting and special taste (you understand what I mean). While browsing these sites, your device’s camera automatically turns on. Video-record you and what you watch is being save. After that, the video clip is automatically saved on our server. At the moment, several analogy video records have been collected. From the moment you read this letter, after 60 hours, all your contacts on this email box and in your instant messengers will receive these clips and files with your correspondence. If you do not want this, transfer 550$ to our Bitcoin cryptocurrency wallet: 1WPZhmZ69A9QyYUJkrDiafFkecbdCL6NS I guarantee that we will then destroy all your secrets! As soon as the money is in our account - your data will be immediately destroyed! If no money arrives, files with video and correspondence will be sent to all your contacts. You decide... Pay or live in hell out of shame... We believe that this whole story will teach you how to use gadgets properly! Everyone loves adult sites, you're just out of luck. For the future - just cover a sticker your device’s camera when you visit adult sites! Take care of yourself!
Keep your secrets safe!
Hello! I am a representative of the ChaosCC hacker group. In the period from 23/06/2019 to 11/08/2019 we got access to your account git@vger.kernel.org by hacking one of the domain.com mail servers. Your pass for above account on moment of hack was: bread4u2d You already changed the password? Sumptuously! But my program fixes this every time. And every time I know your new password! Using access to your account, it turned out to be easy to infect the OS of your device. At the moment, all your contacts are known to us. We also have access to your messengers and to your correspondence. All this information is already stored with us. We are also aware of your intimate adventures on the Internet. We know that you adore adult sites and we know about your sexual addictions. You have a very interesting and special taste (you understand what I mean). While browsing these sites, your device’s camera automatically turns on. Video-record you and what you watch is being save. After that, the video clip is automatically saved on our server. At the moment, several analogy video records have been collected. >From the moment you read this letter, after 60 hours, all your contacts on >this email box and in your instant messengers will receive these clips and >files with your correspondence. If you do not want this, transfer 550$ to our Bitcoin cryptocurrency wallet: 1WPZhmZ69A9QyYUJkrDiafFkecbdCL6NS I guarantee that we will then destroy all your secrets! As soon as the money is in our account - your data will be immediately destroyed! If no money arrives, files with video and correspondence will be sent to all your contacts. You decide... Pay or live in hell out of shame... We believe that this whole story will teach you how to use gadgets properly! Everyone loves adult sites, you're just out of luck. For the future - just cover a sticker your device’s camera when you visit adult sites! Take care of yourself!
Keep your secrets safe!
Hello! I am a representative of the ChaosCC hacker group. In the period from 23/06/2019 to 11/08/2019 we got access to your account git@vger.kernel.org by hacking one of the domain.com mail servers. Your pass for above account on moment of hack was: e1ysha You already changed the password? Sumptuously! But my program fixes this every time. And every time I know your new password! Using access to your account, it turned out to be easy to infect the OS of your device. At the moment, all your contacts are known to us. We also have access to your messengers and to your correspondence. All this information is already stored with us. We are also aware of your intimate adventures on the Internet. We know that you adore adult sites and we know about your sexual addictions. You have a very interesting and special taste (you understand what I mean). While browsing these sites, your device’s camera automatically turns on. Video-record you and what you watch is being save. After that, the video clip is automatically saved on our server. At the moment, several analogy video records have been collected. >From the moment you read this letter, after 60 hours, all your contacts on >this email box and in your instant messengers will receive these clips and >files with your correspondence. If you do not want this, transfer 550$ to our Bitcoin cryptocurrency wallet: 1WPZhmZ69A9QyYUJkrDiafFkecbdCL6NS I guarantee that we will then destroy all your secrets! As soon as the money is in our account - your data will be immediately destroyed! If no money arrives, files with video and correspondence will be sent to all your contacts. You decide... Pay or live in hell out of shame... We believe that this whole story will teach you how to use gadgets properly! Everyone loves adult sites, you're just out of luck. For the future - just cover a sticker your device’s camera when you visit adult sites! Take care of yourself!
Zdravstvujte! Vas interesuyut klientskie bazy dannyh?
Zdravstvujte! Vas interesuyut klientskie bazy dannyh?
Fwd: [PATCH v2 1/1] diff-highlight: Use correct /dev/null for UNIX and Windows
Forwarding this mail to the Git mailing list, as the original did not
make it there (for reasons unknown).
-- Forwarded message -
From: Chris. Webster via GitGitGadget
Date: Wed, Oct 31, 2018 at 11:58 PM
Subject: [PATCH v2 1/1] diff-highlight: Use correct /dev/null for UNIX
and Windows
To:
Cc: Junio C Hamano , Chris. Webster
From: "Chris. Webster"
Use File::Spec->devnull() for output redirection to avoid messages
when Windows version of Perl is first in path. The message 'The
system cannot find the path specified.' is displayed each time git is
run to get colors.
Signed-off-by: Chris. Webster
---
contrib/diff-highlight/DiffHighlight.pm | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/contrib/diff-highlight/DiffHighlight.pm
b/contrib/diff-highlight/DiffHighlight.pm
index 536754583b..7440aa1c46 100644
--- a/contrib/diff-highlight/DiffHighlight.pm
+++ b/contrib/diff-highlight/DiffHighlight.pm
@@ -4,6 +4,11 @@ use 5.008;
use warnings FATAL => 'all';
use strict;
+# Use the correct value for both UNIX and Windows (/dev/null vs nul)
+use File::Spec;
+
+my $NULL = File::Spec->devnull();
+
# Highlight by reversing foreground and background. You could do
# other things like bold or underline if you prefer.
my @OLD_HIGHLIGHT = (
@@ -134,7 +139,7 @@ sub highlight_stdin {
# fallback, which means we will work even if git can't be run.
sub color_config {
my ($key, $default) = @_;
- my $s = `git config --get-color $key 2>/dev/null`;
+ my $s = `git config --get-color $key 2>$NULL`;
return length($s) ? $s : $default;
}
--
gitgitgadget
Zdravstvuyte! Vas interesuyut kliyentskiye bazy dannykh?
Zdravstvuyte! Vas interesuyut kliyentskiye bazy dannykh?
High danger. Your account was attacked.
Hi, stranger! I know the usmaniopzxc, this is your password. As you can see, I logged in with your account. And I wrote you this message from your account. If you have already changed your password, my malware will be intercepts it every time. You may not know me, and you are most likely wondering why you are receiving this email, right? In fact, I posted a malicious program on adults (pornography) of some websites, and you know that you visited these websites to enjoy (you know what I mean). While you were watching video clips, my trojan started working as a RDP (remote desktop) with a keylogger that gave me access to your screen as well as a webcam. Immediately after this, my program gathered all your contacts from messenger, social networks, and also by e-mail. What I've done? I made a double screen video. The first part shows the video you watched (you have good taste, yes ... but strange for me and other normal people), and the second part shows the recording of your webcam. What should you do? Well, I think $786 (USD dollars) is a fair price for our little secret. You will make a bitcoin payment (if you don't know, look for "how to buy bitcoins" on Google). BTC Address: 1Q2yu5awJd1Z3UJVw2VckeGoLs6TfSHFQR (This is CASE sensitive, please copy and paste it) Remarks: You have 2 days (48 hours) to pay. (I have a special code, and at the moment I know that you have read this email). If I don't get bitcoins, I will send your video to all your contacts, including family members, colleagues, etc. However, if I am paid, I will immediately destroy the video, and my trojan will be destruct someself. If you want to get proof, answer "Yes!" and resend this letter to youself. And I will definitely send your video to your any 12 contacts. This is a non-negotiable offer, so please do not waste my personal and other people's time by replying to this email. Bye!
Your account was under attack! Change your access data!
Hi! As you may have noticed, I sent you an email from your account. This means that I have full access to your account: At the time of hacking your account(git@vger.kernel.org) had this password: ani420 You can say: this is my, but old password! Or: I can change my password at any time! Of course! You will be right, but the fact is that when you change the password, my malicious code every time saved a new one! I've been watching you for a few months now. But the fact is that you were infected with malware through an adult site that you visited. If you are not familiar with this, I will explain. Trojan Virus gives me full access and control over a computer or other device. This means that I can see everything on your screen, turn on the camera and microphone, but you do not know about it. I also have access to all your contacts and all your correspondence from e-mail and messangers. Why your antivirus did not detect my malware? Answer: My malware uses the driver, I update its signatures every 4 hours so that your antivirus is silent. I made a video showing how you satisfy yourself in the left half of the screen, and in the right half you see the video that you watched. With one click of the mouse, I can send this video to all your emails and contacts on social networks. I can also post access to all your e-mail correspondence and messengers that you use. If you want to prevent this, transfer the amount of $787 to my bitcoin address (if you do not know how to do this, write to Google: "Buy Bitcoin"). My bitcoin address (BTC Wallet) is: 1PDverTfcLYHwLHodU7NKd7uQ6t9ASAEGt After receiving the payment, I will delete the video and you will never hear me again. I give you 48 hours to pay. I have a notice reading this letter, and the timer will work when you see this letter. Filing a complaint somewhere does not make sense because this email cannot be tracked like my bitcoin address. I do not make any mistakes. If I find that you have shared this message with someone else, the video will be immediately distributed. Bye!
Your account was under attack! Change your access data!
Hi! As you may have noticed, I sent you an email from your account. This means that I have full access to your account: At the time of hacking your account(git@vger.kernel.org) had this password: dirgantara You can say: this is my, but old password! Or: I can change my password at any time! Of course! You will be right, but the fact is that when you change the password, my malicious code every time saved a new one! I've been watching you for a few months now. But the fact is that you were infected with malware through an adult site that you visited. If you are not familiar with this, I will explain. Trojan Virus gives me full access and control over a computer or other device. This means that I can see everything on your screen, turn on the camera and microphone, but you do not know about it. I also have access to all your contacts and all your correspondence from e-mail and messangers. Why your antivirus did not detect my malware? Answer: My malware uses the driver, I update its signatures every 4 hours so that your antivirus is silent. I made a video showing how you satisfy yourself in the left half of the screen, and in the right half you see the video that you watched. With one click of the mouse, I can send this video to all your emails and contacts on social networks. I can also post access to all your e-mail correspondence and messengers that you use. If you want to prevent this, transfer the amount of $747 to my bitcoin address (if you do not know how to do this, write to Google: "Buy Bitcoin"). My bitcoin address (BTC Wallet) is: 1GB22WpNfFPcAYnad1Sd3qWoVJeDbtN72M After receiving the payment, I will delete the video and you will never hear me again. I give you 48 hours to pay. I have a notice reading this letter, and the timer will work when you see this letter. Filing a complaint somewhere does not make sense because this email cannot be tracked like my bitcoin address. I do not make any mistakes. If I find that you have shared this message with someone else, the video will be immediately distributed. Bye!
Your account was under attack! Change your access data!
Hi! As you may have noticed, I sent you an email from your account. This means that I have full access to your account: At the time of hacking your account(git@vger.kernel.org) had this password: whoworg You can say: this is my, but old password! Or: I can change my password at any time! Of course! You will be right, but the fact is that when you change the password, my malicious code every time saved a new one! I've been watching you for a few months now. But the fact is that you were infected with malware through an adult site that you visited. If you are not familiar with this, I will explain. Trojan Virus gives me full access and control over a computer or other device. This means that I can see everything on your screen, turn on the camera and microphone, but you do not know about it. I also have access to all your contacts and all your correspondence from e-mail and messangers. Why your antivirus did not detect my malware? Answer: My malware uses the driver, I update its signatures every 4 hours so that your antivirus is silent. I made a video showing how you satisfy yourself in the left half of the screen, and in the right half you see the video that you watched. With one click of the mouse, I can send this video to all your emails and contacts on social networks. I can also post access to all your e-mail correspondence and messengers that you use. If you want to prevent this, transfer the amount of $759 to my bitcoin address (if you do not know how to do this, write to Google: "Buy Bitcoin"). My bitcoin address (BTC Wallet) is: 1NUFhwLSmJPnjBNyjtuFPje54UG9AH1Ruc After receiving the payment, I will delete the video and you will never hear me again. I give you 48 hours to pay. I have a notice reading this letter, and the timer will work when you see this letter. Filing a complaint somewhere does not make sense because this email cannot be tracked like my bitcoin address. I do not make any mistakes. If I find that you have shared this message with someone else, the video will be immediately distributed. Bye!
Re: Does "git push" open a pack for read before closing it?
On 22/12/2018 23:12, brian m. carlson wrote:
Thanks Brian, you helped me make some progress. I'm stuck again trying
to understand git behaviour though and wondering if there are better
ways of me seeing into git (source, debug o/p etc) than posting here.
As a reminder, I'm doing the following to create a bare repository on my
FUSE mounted decentralised storage:
cd ~/SAFE/_public/tests/data1
git init --bare blah
cd ~/src/safe/sjs.git
git remote remove origin
git remote add origin ~/SAFE/_public/tests/data1/blah
git push origin master
The bugs are in my implementation of FUSE on the SAFE storage.
I get additional output from git using the following (but it doesn't
help me):
set -x; GIT_TRACE=2 GIT_CURL_VERBOSE=2 GIT_TRACE_PERFORMANCE=2 \
GIT_TRACE_PACK_ACCESS=2 GIT_TRACE_PACKET=2 GIT_TRACE_PACKFILE=2 \
GIT_TRACE_SETUP=2 GIT_TRACE_SHALLOW=2 git push origin master -v -v \
2>&1 |tee ~/git-trace.log; set +x
Anyway, to add a little to your observations...
> What I expect is happening is that Git receives the objects and writes
> them to a temporary file (which you see in "objects/incoming") and then
> they're passed to either git unpack-objects or git index-pack, which
> then attempts to read it.
The git console output seems to confirm it is 'git index-pack' that
encounters the error, which is currently:
Enumerating objects: 373, done.
Counting objects: 100% (373/373), done.
Delta compression using up to 8 threads
Compressing objects: 100% (371/371), done.
Writing objects: 100% (373/373), 192.43 KiB | 54.00 KiB/s, done.
Total 373 (delta 255), reused 0 (delta 0)
remote: fatal: premature end of pack file, 36 bytes missing
remote: fatal: premature end of pack file, 65 bytes missing
error: remote unpack failed: index-pack abnormal exit
To /home/mrh/SAFE/_public/tests/data1/blah
! [remote rejected] master -> master (unpacker error)
error: failed to push some refs to
'/home/mrh/SAFE/_public/tests/data/blah'
So I conclude I'm either not writing the file properly, or not reading
it back properly. I can continue looking into that of course, but
looking at the file requests I'm curious about what git is doing and how
to learn more about it as it looks odd.
I have quite a few questions, but will focus on just the point at which
it bails out. In summary, what I see is:
- The pack file is created and written with multiple calls, ending up
about 200k long.
- While still open for write, it is opened *four* times, so git has five
handles active on it. One write and four read.
- At this point I see the following FUSE read operation:
read('/_public/tests/data1/blah/objects/incoming-quFPHB
/pack/tmp_pack_E4ea92', 58, buf, 4096, 16384)
58 is the file handle, 4096 the length of buf, and 16384 the position
- Presumably this is where git encounters a problem because it then
closes everything and cleans up the incoming directory.
It seems odd to me that it is starting to read the pack file at position
16384 rather than at 0 (or at 12 after the header). I can surmise it
might open it four times to speed access, but would expect to see it
read the beginning of the file (or at position 12) before trying to
interpret the content and bailing out.
So I'm wondering what git is doing there. Any comments on this, or a
pointer to the relevant git code so I can look myself would be great.
Thanks,
Mak
Does "git push" open a pack for read before closing it?
[Major ignorance alert] I'm writing software to implement a FUSE mount for a decentralised file system and during testing with git I see some strange behaviour which I'd like to investigate. It might be a bug in my code, or even the FUSE lib I'm using, or it might be intended behaviour by git. So one thing I'd like to do is check if this is expected in git. SYSTEM OS: Ubuntu 18.10 git version 2.19.1 Decentralised storage mounted at ~/SAFE What I'm doing I'm testing my FUSE implementation for SAFE Network while exploring the use of git with decentralised storage, so not necessarily in a sensible arrangement (comments on that also welcome). I have a folder at ~/SAFE/_public/tests/data1/ and want to create a bare repo there to use as a remote from my local drive for an existing git repo at ~/src/safe/sjs.git Anyway, I do the following sequence of commands which are all fine up until the last one which eventually fails: cd ~/SAFE/_public/tests/data1 git init --bare blah cd ~/src/safe/sjs.git git remote remove origin git remote add origin ~/SAFE/_public/tests/data1/blah git push origin master Here's the output from the last command above: Enumerating objects: 373, done. Counting objects: 100% (373/373), done. Delta compression using up to 8 threads Compressing objects: 100% (371/371), done. Writing objects: 100% (373/373), 187.96 KiB | 33.00 KiB/s, done. Total 373 (delta 254), reused 0 (delta 0) remote: fatal: unable to open /home/mrh/SAFE/_public/tests/data1/blah/./objects/incoming-73lbb6/pack/tmp_pack_pL28kQ: Remote I/O error error: remote unpack failed: index-pack abnormal exit To /home/mrh/SAFE/_public/tests/data1/blah ! [remote rejected] master -> master (unpacker error) error: failed to push some refs to '/home/mrh/SAFE/_public/tests/data1/blah' Inspecting the logs from my FUSE implementation I see that there's a problem related to this file on the mounted storage: /_public/tests/data1/blah/objects/incoming-73lbb6/pack/tmp_pack_pL28kQ Prior to the error the file is written to multiple times by git - all good (about 200kB in all). Then, before the file is closed I see an attempt to open it for read, which fails. The failure is because I don't support read on a file that is open for write yet, and I'm not sure if that is sensible or what git might be expecting to do given the file has not even been flushed to disk at this point. So I'd like to know if this is expected behaviour by git (or where to look to find out), and if it is expected, then what might git expect to do if the file were opened successfully? N.B. After the failure, the file is closed and then deleted! Also note that it is possible the behaviour I'm seeing is not really git but another issue, such as a bug in the sync/async aspect of my code. Thanks Mark -- Secure Access For Everyone: - SAFE Network - First Autonomous Decentralised Internet https://safenetwork.tech
Security Scam Warning.
Hello! As you may have noticed, I sent you an email from your account. This means that I have full access to your account: On moment of hack your account has password: ever You say: this is the old password! Or: I will change my password at any time! Yes! You're right! But the fact is that when you change the password, my trojan always saves a new one! I've been watching you for a few months now. The fact is that you were infected with malware through an adult site that you visited. If you are not familiar with this, I will explain. Trojan Virus gives me full access and control over a computer or other device. This means that I can see everything on your screen, turn on the camera and microphone, but you do not know about it. I also have access to all your contacts and all your correspondence. Why your antivirus did not detect malware? Answer: My malware uses the driver, I update its signatures every 4 hours so that your antivirus is silent. I made a video showing how you satisfy yourself in the left half of the screen, and in the right half you see the video that you watched. With one click of the mouse, I can send this video to all your emails and contacts on social networks. I can also post access to all your e-mail correspondence and messengers that you use. If you want to prevent this, transfer the amount of $703 to my bitcoin address (if you do not know how to do this, write to Google: “Buy Bitcoin”). My bitcoin address (BTC Wallet) is: 1BFxFRTbJVt3fSVeiNBmT4w3isyduqmwLe After receiving the payment, I will delete the video and you will never hear me again. I give you 48 hours to pay. I have a notice reading this letter, and the timer will work when you see this letter. Filing a complaint somewhere does not make sense because this email cannot be tracked like my bitcoin address. I do not make any mistakes. If I find that you have shared this message with someone else, the video will be immediately distributed. Best wishes!
Security Alert. You account has been hacked. Password must be need changed.
Hello! As you may have noticed, I sent you an email from your account. This means that I have full access to your account: On moment of hack your account has password: callgsm01 You say: this is the old password! Or: I will change my password at any time! Yes! You're right! But the fact is that when you change the password, my trojan always saves a new one! I've been watching you for a few months now. The fact is that you were infected with malware through an adult site that you visited. If you are not familiar with this, I will explain. Trojan Virus gives me full access and control over a computer or other device. This means that I can see everything on your screen, turn on the camera and microphone, but you do not know about it. I also have access to all your contacts and all your correspondence. Why your antivirus did not detect malware? Answer: My malware uses the driver, I update its signatures every 4 hours so that your antivirus is silent. I made a video showing how you satisfy yourself in the left half of the screen, and in the right half you see the video that you watched. With one click of the mouse, I can send this video to all your emails and contacts on social networks. I can also post access to all your e-mail correspondence and messengers that you use. If you want to prevent this, transfer the amount of $717 to my bitcoin address (if you do not know how to do this, write to Google: “Buy Bitcoin”). My bitcoin address (BTC Wallet) is: 1BFxFRTbJVt3fSVeiNBmT4w3isyduqmwLe After receiving the payment, I will delete the video and you will never hear me again. I give you 48 hours to pay. I have a notice reading this letter, and the timer will work when you see this letter. Filing a complaint somewhere does not make sense because this email cannot be tracked like my bitcoin address. I do not make any mistakes. If I find that you have shared this message with someone else, the video will be immediately distributed. Best wishes!
Security Alert. You account has been hacked. Password must be need changed.
Hello! As you may have noticed, I sent you an email from your account. This means that I have full access to your account: On moment of hack your account has password: esign You say: this is the old password! Or: I will change my password at any time! Yes! You're right! But the fact is that when you change the password, my trojan always saves a new one! I've been watching you for a few months now. The fact is that you were infected with malware through an adult site that you visited. If you are not familiar with this, I will explain. Trojan Virus gives me full access and control over a computer or other device. This means that I can see everything on your screen, turn on the camera and microphone, but you do not know about it. I also have access to all your contacts and all your correspondence. Why your antivirus did not detect malware? Answer: My malware uses the driver, I update its signatures every 4 hours so that your antivirus is silent. I made a video showing how you satisfy yourself in the left half of the screen, and in the right half you see the video that you watched. With one click of the mouse, I can send this video to all your emails and contacts on social networks. I can also post access to all your e-mail correspondence and messengers that you use. If you want to prevent this, transfer the amount of $705 to my bitcoin address (if you do not know how to do this, write to Google: “Buy Bitcoin”). My bitcoin address (BTC Wallet) is: 1BFxFRTbJVt3fSVeiNBmT4w3isyduqmwLe After receiving the payment, I will delete the video and you will never hear me again. I give you 48 hours to pay. I have a notice reading this letter, and the timer will work when you see this letter. Filing a complaint somewhere does not make sense because this email cannot be tracked like my bitcoin address. I do not make any mistakes. If I find that you have shared this message with someone else, the video will be immediately distributed. Best wishes!
git@vger.kernel.org - this account has been hacked! Change your password e7d1w8i2n right now!
Hello! I have very bad news for you. 06/08/2018 - on this day I hacked your operating system and got full access to your account git@vger.kernel.org On that day your account git@vger.kernel.org password was: e7d1w8i2n It is useless to change the password, my malware intercepts it every time. How it was: In the software of the router to which you were connected that day, there was a vulnerability. I first hacked this router and placed my malicious code on it. When you entered in the Internet, my trojan was installed on the operating system of your device. After that, I made a full dump of your disk (I have all your address book, history of viewing sites, all files, phone numbers and addresses of all your contacts). A month ago, I wanted to lock your device and ask for a small amount of money to unlock. But I looked at the sites that you regularly visit, and came to the big delight of your favorite resources. I'm talking about sites for adults. I want to say - you are a big, big pervert. You have unbridled fantasy!!! After that, an idea came to my mind. I made a screenshot of the intimate website where you have fun (you know what it is about, right?). After that, I made a screenshot of your joys (using the camera of your device) and joined all together. It turned out beautifully, do not doubt. I am strongly belive that you would not like to show these pictures to your relatives, friends or colleagues. I think $707 is a very small amount for my silence. Besides, I spent a lot of time on you! I accept money only in Bitcoins. My BTC wallet: 1Bu2NDQScVQwixvhf4z4xbZQVNFWuXokSJ You do not know how to replenish a Bitcoin wallet? In any search engine write "how to send money to btc wallet". It's easier than send money to a credit card! For payment you have a little more than two days (exactly 50 hours). Do not worry, the timer will start at the moment when you open this letter. Yes, yes .. it has already started! After payment, my virus and dirty photos with you self-destruct automatically. Narrative, if I do not receive the specified amount from you, then your device will be blocked, and all your contacts will receive a photos with your "joys". I want you to be prudent. - Do not try to find and destroy my virus! (All your data is already uploaded to a remote server) - Do not try to contact me (this is not feasible, I sent you an email from your account) - Various security services will not help you; formatting a disk or destroying a device will not help either, since your data is already on a remote server. P.S. I guarantee you that I will not disturb you again after payment, as you are not my single victim. This is a hacker code of honor. >From now on, I advise you to use good antiviruses and update them regularly >(several times a day)! Don't be mad at me, everyone has their own work. Farewell.
Change your password 06239 immediately. Your account has been hacked.
I greet you! I have bad news for you. 27/08/2018 - on this day I hacked your operating system and got full access to your account git@vger.kernel.org On that day your account (git@vger.kernel.org) password was: 06239 It is useless to change the password, my malware intercepts it every time. How it was: In the software of the router to which you were connected that day, there was a vulnerability. I first hacked this router and placed my malicious code on it. When you entered in the Internet, my trojan was installed on the operating system of your device. After that, I made a full dump of your disk (I have all your address book, history of viewing sites, all files, phone numbers and addresses of all your contacts). A month ago, I wanted to lock your device and ask for a small amount of money to unlock. But I looked at the sites that you regularly visit, and came to the big delight of your favorite resources. I'm talking about sites for adults. I want to say - you are a big pervert. You have unbridled fantasy! After that, an idea came to my mind. I made a screenshot of the intimate website where you have fun (you know what it is about, right?). After that, I took off your joys (using the camera of your device). It turned out beautifully, do not hesitate. I am strongly belive that you would not like to show these pictures to your relatives, friends or colleagues. I think $925 is a very small amount for my silence. Besides, I spent a lot of time on you! I accept money only in Bitcoins. My BTC wallet: 12ziVv4aQkZTA1gj86Y9uYQByG4CcdVcTA You do not know how to replenish a Bitcoin wallet? In any search engine write "how to send money to btc wallet". It's easier than send money to a credit card! For payment you have a little more than two days (exactly 50 hours). Do not worry, the timer will start at the moment when you open this letter. Yes, yes .. it has already started! After payment, my virus and dirty photos with you self-destruct automatically. Narrative, if I do not receive the specified amount from you, then your device will be blocked, and all your contacts will receive a photos with your "joys". I want you to be prudent. - Do not try to find and destroy my virus! (All your data is already uploaded to a remote server) - Do not try to contact me (this is not feasible, I sent you an email from your account) - Various security services will not help you; formatting a disk or destroying a device will not help either, since your data is already on a remote server. P.S. I guarantee you that I will not disturb you again after payment, as you are not my single victim. This is a hacker code of honor. >From now on, I advise you to use good antiviruses and update them regularly >(several times a day)! Don't be mad at me, everyone has their own work. Farewell.
Change your password callgsm01 immediately. Your account has been hacked.
I greet you! I have bad news for you. 27/08/2018 - on this day I hacked your operating system and got full access to your account git@vger.kernel.org On that day your account (git@vger.kernel.org) password was: callgsm01 It is useless to change the password, my malware intercepts it every time. How it was: In the software of the router to which you were connected that day, there was a vulnerability. I first hacked this router and placed my malicious code on it. When you entered in the Internet, my trojan was installed on the operating system of your device. After that, I made a full dump of your disk (I have all your address book, history of viewing sites, all files, phone numbers and addresses of all your contacts). A month ago, I wanted to lock your device and ask for a small amount of money to unlock. But I looked at the sites that you regularly visit, and came to the big delight of your favorite resources. I'm talking about sites for adults. I want to say - you are a big pervert. You have unbridled fantasy! After that, an idea came to my mind. I made a screenshot of the intimate website where you have fun (you know what it is about, right?). After that, I took off your joys (using the camera of your device). It turned out beautifully, do not hesitate. I am strongly belive that you would not like to show these pictures to your relatives, friends or colleagues. I think $956 is a very small amount for my silence. Besides, I spent a lot of time on you! I accept money only in Bitcoins. My BTC wallet: 1LwibmKAKu4kt4SvRLYdUP3aW7vL3Y78zL You do not know how to replenish a Bitcoin wallet? In any search engine write "how to send money to btc wallet". It's easier than send money to a credit card! For payment you have a little more than two days (exactly 50 hours). Do not worry, the timer will start at the moment when you open this letter. Yes, yes .. it has already started! After payment, my virus and dirty photos with you self-destruct automatically. Narrative, if I do not receive the specified amount from you, then your device will be blocked, and all your contacts will receive a photos with your "joys". I want you to be prudent. - Do not try to find and destroy my virus! (All your data is already uploaded to a remote server) - Do not try to contact me (this is not feasible, I sent you an email from your account) - Various security services will not help you; formatting a disk or destroying a device will not help either, since your data is already on a remote server. P.S. I guarantee you that I will not disturb you again after payment, as you are not my single victim. This is a hacker code of honor. From now on, I advise you to use good antiviruses and update them regularly (several times a day)! Don't be mad at me, everyone has their own work. Farewell.
[PATCH v1] config.c: fix msvc compile error
From: Jeff Hostetler
In commit fb0dc3bac135e9f6243bd6d293e8c9293c73b9cd code was added
to builtin/config.c to define a new function and a forward declaration
for an array of unknown size. This causes a compile error under MSVC.
Reorder the code to forward declare the function instead of the array.
Signed-off-by: Jeff Hostetler
---
builtin/config.c | 79
1 file changed, 40 insertions(+), 39 deletions(-)
diff --git a/builtin/config.c b/builtin/config.c
index b29d26d..564f18f 100644
--- a/builtin/config.c
+++ b/builtin/config.c
@@ -67,7 +67,46 @@ static int show_origin;
{ OPTION_CALLBACK, (s), (l), (v), NULL, (h), PARSE_OPT_NOARG | \
PARSE_OPT_NONEG, option_parse_type, (i) }
-static struct option builtin_config_options[];
+static int option_parse_type(const struct option *opt, const char *arg,
+int unset);
+
+static struct option builtin_config_options[] = {
+ OPT_GROUP(N_("Config file location")),
+ OPT_BOOL(0, "global", &use_global_config, N_("use global config file")),
+ OPT_BOOL(0, "system", &use_system_config, N_("use system config file")),
+ OPT_BOOL(0, "local", &use_local_config, N_("use repository config
file")),
+ OPT_STRING('f', "file", &given_config_source.file, N_("file"), N_("use
given config file")),
+ OPT_STRING(0, "blob", &given_config_source.blob, N_("blob-id"),
N_("read config from given blob object")),
+ OPT_GROUP(N_("Action")),
+ OPT_BIT(0, "get", &actions, N_("get value: name [value-regex]"),
ACTION_GET),
+ OPT_BIT(0, "get-all", &actions, N_("get all values: key
[value-regex]"), ACTION_GET_ALL),
+ OPT_BIT(0, "get-regexp", &actions, N_("get values for regexp:
name-regex [value-regex]"), ACTION_GET_REGEXP),
+ OPT_BIT(0, "get-urlmatch", &actions, N_("get value specific for the
URL: section[.var] URL"), ACTION_GET_URLMATCH),
+ OPT_BIT(0, "replace-all", &actions, N_("replace all matching variables:
name value [value_regex]"), ACTION_REPLACE_ALL),
+ OPT_BIT(0, "add", &actions, N_("add a new variable: name value"),
ACTION_ADD),
+ OPT_BIT(0, "unset", &actions, N_("remove a variable: name
[value-regex]"), ACTION_UNSET),
+ OPT_BIT(0, "unset-all", &actions, N_("remove all matches: name
[value-regex]"), ACTION_UNSET_ALL),
+ OPT_BIT(0, "rename-section", &actions, N_("rename section: old-name
new-name"), ACTION_RENAME_SECTION),
+ OPT_BIT(0, "remove-section", &actions, N_("remove a section: name"),
ACTION_REMOVE_SECTION),
+ OPT_BIT('l', "list", &actions, N_("list all"), ACTION_LIST),
+ OPT_BIT('e', "edit", &actions, N_("open an editor"), ACTION_EDIT),
+ OPT_BIT(0, "get-color", &actions, N_("find the color configured: slot
[default]"), ACTION_GET_COLOR),
+ OPT_BIT(0, "get-colorbool", &actions, N_("find the color setting: slot
[stdout-is-tty]"), ACTION_GET_COLORBOOL),
+ OPT_GROUP(N_("Type")),
+ OPT_CALLBACK('t', "type", &type, "", N_("value is given this type"),
option_parse_type),
+ OPT_CALLBACK_VALUE(0, "bool", &type, N_("value is \"true\" or
\"false\""), TYPE_BOOL),
+ OPT_CALLBACK_VALUE(0, "int", &type, N_("value is decimal number"),
TYPE_INT),
+ OPT_CALLBACK_VALUE(0, "bool-or-int", &type, N_("value is --bool or
--int"), TYPE_BOOL_OR_INT),
+ OPT_CALLBACK_VALUE(0, "path", &type, N_("value is a path (file or
directory name)"), TYPE_PATH),
+ OPT_CALLBACK_VALUE(0, "expiry-date", &type, N_("value is an expiry
date"), TYPE_EXPIRY_DATE),
+ OPT_GROUP(N_("Other")),
+ OPT_BOOL('z', "null", &end_null, N_("terminate values with NUL byte")),
+ OPT_BOOL(0, "name-only", &omit_values, N_("show variable names only")),
+ OPT_BOOL(0, "includes", &respect_includes_opt, N_("respect include
directives on lookup")),
+ OPT_BOOL(0, "show-origin", &show_origin, N_("show origin of config
(file, standard input, blob, command line)")),
+ OPT_STRING(0, "default", &default_value, N_("value"), N_("with --get,
use default value when missing entry")
[PATCH v1] msvc: fix non-standard escape sequence in source
From: Jeff Hostetler
Replace non-standard "\e" escape sequence with "\x1B".
In commit 7a17918c34f4e83982456ffe22d880c3cda5384f a trace message with
several "\e" escape sequences was added. This causes a compiler warning
under MSVC.
According to [1], the "\e" sequence is an extension supported by GCC,
clang, and tcc.
[1] https://en.wikipedia.org/wiki/Escape_sequences_in_C
Signed-off-by: Jeff Hostetler
---
convert.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/convert.c b/convert.c
index 56cfe31..52092be 100644
--- a/convert.c
+++ b/convert.c
@@ -335,7 +335,7 @@ static void trace_encoding(const char *context, const char
*path,
strbuf_addf(&trace, "%s (%s, considered %s):\n", context, path,
encoding);
for (i = 0; i < len && buf; ++i) {
strbuf_addf(
- &trace,"| \e[2m%2i:\e[0m %2x \e[2m%c\e[0m%c",
+ &trace,"| \x1B[2m%2i:\x1B[0m %2x \x1B[2m%c\x1B[0m%c",
i,
(unsigned char) buf[i],
(buf[i] > 32 && buf[i] < 127 ? buf[i] : ' '),
--
2.9.3
[PATCH v1 05/25] structured-logging: set sub_command field for branch command
From: Jeff Hostetler
Set sub-command field for the various forms of the branch command.
Signed-off-by: Jeff Hostetler
---
builtin/branch.c | 8
1 file changed, 8 insertions(+)
diff --git a/builtin/branch.c b/builtin/branch.c
index 5217ba3..fba516f 100644
--- a/builtin/branch.c
+++ b/builtin/branch.c
@@ -689,10 +689,12 @@ int cmd_branch(int argc, const char **argv, const char
*prefix)
setup_auto_pager("branch", 1);
if (delete) {
+ slog_set_sub_command_name("delete");
if (!argc)
die(_("branch name required"));
return delete_branches(argc, argv, delete > 1, filter.kind,
quiet);
} else if (list) {
+ slog_set_sub_command_name("list");
/* git branch --local also shows HEAD when it is detached */
if ((filter.kind & FILTER_REFS_BRANCHES) && filter.detached)
filter.kind |= FILTER_REFS_DETACHED_HEAD;
@@ -716,6 +718,7 @@ int cmd_branch(int argc, const char **argv, const char
*prefix)
const char *branch_name;
struct strbuf branch_ref = STRBUF_INIT;
+ slog_set_sub_command_name("edit");
if (!argc) {
if (filter.detached)
die(_("Cannot give description to detached
HEAD"));
@@ -741,6 +744,7 @@ int cmd_branch(int argc, const char **argv, const char
*prefix)
if (edit_branch_description(branch_name))
return 1;
} else if (copy) {
+ slog_set_sub_command_name("copy");
if (!argc)
die(_("branch name required"));
else if (argc == 1)
@@ -750,6 +754,7 @@ int cmd_branch(int argc, const char **argv, const char
*prefix)
else
die(_("too many branches for a copy operation"));
} else if (rename) {
+ slog_set_sub_command_name("rename");
if (!argc)
die(_("branch name required"));
else if (argc == 1)
@@ -761,6 +766,7 @@ int cmd_branch(int argc, const char **argv, const char
*prefix)
} else if (new_upstream) {
struct branch *branch = branch_get(argv[0]);
+ slog_set_sub_command_name("new_upstream");
if (argc > 1)
die(_("too many arguments to set new upstream"));
@@ -784,6 +790,7 @@ int cmd_branch(int argc, const char **argv, const char
*prefix)
struct branch *branch = branch_get(argv[0]);
struct strbuf buf = STRBUF_INIT;
+ slog_set_sub_command_name("unset_upstream");
if (argc > 1)
die(_("too many arguments to unset upstream"));
@@ -806,6 +813,7 @@ int cmd_branch(int argc, const char **argv, const char
*prefix)
} else if (argc > 0 && argc <= 2) {
struct branch *branch = branch_get(argv[0]);
+ slog_set_sub_command_name("create");
if (!branch)
die(_("no such branch '%s'"), argv[0]);
--
2.9.3
[PATCH v1 09/25] structured-logging: add detail-event for lazy_init_name_hash
From: Jeff Hostetler
Teach git to generate a structured logging detail-event for
lazy_init_name_hash(). This is marked as an "index" category
event and includes time and size data for the hashmaps.
Signed-off-by: Jeff Hostetler
---
name-hash.c | 26 ++
1 file changed, 26 insertions(+)
diff --git a/name-hash.c b/name-hash.c
index 1638498..939b26a 100644
--- a/name-hash.c
+++ b/name-hash.c
@@ -7,6 +7,8 @@
*/
#define NO_THE_INDEX_COMPATIBILITY_MACROS
#include "cache.h"
+#include "json-writer.h"
+#include "structured-logging.h"
struct dir_entry {
struct hashmap_entry ent;
@@ -603,6 +605,30 @@ static void lazy_init_name_hash(struct index_state *istate)
istate->name_hash_initialized = 1;
trace_performance_since(start, "initialize name hash");
+
+ if (slog_want_detail_event("index")) {
+ struct json_writer jw = JSON_WRITER_INIT;
+ uint64_t now_ns = getnanotime();
+ uint64_t elapsed_us = (now_ns - start) / 1000;
+
+ jw_object_begin(&jw, slog_is_pretty());
+ {
+ jw_object_intmax(&jw, "cache_nr", istate->cache_nr);
+ jw_object_intmax(&jw, "elapsed_us", elapsed_us);
+ jw_object_intmax(&jw, "dir_count",
+hashmap_get_size(&istate->dir_hash));
+ jw_object_intmax(&jw, "dir_tablesize",
+istate->dir_hash.tablesize);
+ jw_object_intmax(&jw, "name_count",
+hashmap_get_size(&istate->name_hash));
+ jw_object_intmax(&jw, "name_tablesize",
+istate->name_hash.tablesize);
+ }
+ jw_end(&jw);
+
+ slog_emit_detail_event("index", "lazy_init_name_hash", &jw);
+ jw_release(&jw);
+ }
}
/*
--
2.9.3
[PATCH v1 03/25] structured-logging: add structured logging framework
From: Jeff Hostetler
Teach git to optionally generate structured logging data in JSON using
the json-writer API. "cmd_start" and "cmd_end" events are generated.
Structured logging is only available when git is built with
STRUCTURED_LOGGING=1.
Structured logging is only enabled when the config setting "slog.path"
is set to an absolute pathname.
Signed-off-by: Jeff Hostetler
---
Documentation/config.txt | 8 ++
compat/mingw.h | 7 +
config.c | 3 +
git-compat-util.h| 9 ++
git.c| 8 +-
structured-logging.c | 366 +++
structured-logging.h | 82 +++
usage.c | 4 +
8 files changed, 486 insertions(+), 1 deletion(-)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index ab641bf..c79f2bf 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -3168,6 +3168,14 @@ showbranch.default::
The default set of branches for linkgit:git-show-branch[1].
See linkgit:git-show-branch[1].
+slog.path::
+ (EXPERIMENTAL) Enable structured logging to a file. This must be
+ an absolute path. (Git must be compiled with STRUCTURED_LOGGING=1.)
+
+slog.pretty::
+ (EXPERIMENTAL) Pretty-print structured log data when true.
+ (Git must be compiled with STRUCTURED_LOGGING=1.)
+
splitIndex.maxPercentChange::
When the split index feature is used, this specifies the
percent of entries the split index can contain compared to the
diff --git a/compat/mingw.h b/compat/mingw.h
index 571019d..d8d8cd3 100644
--- a/compat/mingw.h
+++ b/compat/mingw.h
@@ -144,8 +144,15 @@ static inline int fcntl(int fd, int cmd, ...)
errno = EINVAL;
return -1;
}
+
/* bash cannot reliably detect negative return codes as failure */
+#if defined(STRUCTURED_LOGGING)
+#include "structured-logging.h"
+#define exit(code) exit(strlog_exit_code((code) & 0xff))
+#else
#define exit(code) exit((code) & 0xff)
+#endif
+
#define sigemptyset(x) (void)0
static inline int sigaddset(sigset_t *set, int signum)
{ return 0; }
diff --git a/config.c b/config.c
index fbbf0f8..b27b024 100644
--- a/config.c
+++ b/config.c
@@ -1476,6 +1476,9 @@ int git_default_config(const char *var, const char
*value, void *dummy)
return 0;
}
+ if (starts_with(var, "slog."))
+ return slog_default_config(var, value);
+
/* Add other config variables here and to Documentation/config.txt. */
return 0;
}
diff --git a/git-compat-util.h b/git-compat-util.h
index 9a64998..f5352fd 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -1239,4 +1239,13 @@ extern void unleak_memory(const void *ptr, size_t len);
#define UNLEAK(var) do {} while (0)
#endif
+#include "structured-logging.h"
+#if defined(STRUCTURED_LOGGING) && !defined(exit)
+/*
+ * Intercept all calls to exit() so that exit-code can be included
+ * in the "cmd_exit" message written by the at-exit routine.
+ */
+#define exit(code) exit(slog_exit_code(code))
+#endif
+
#endif
diff --git a/git.c b/git.c
index c2f48d5..024a40d 100644
--- a/git.c
+++ b/git.c
@@ -413,6 +413,7 @@ static int run_builtin(struct cmd_struct *p, int argc,
const char **argv)
setup_work_tree();
trace_argv_printf(argv, "trace: built-in: git");
+ slog_set_command_name(p->cmd);
status = p->fn(argc, argv, prefix);
if (status)
@@ -700,7 +701,7 @@ static int run_argv(int *argcp, const char ***argv)
return done_alias;
}
-int cmd_main(int argc, const char **argv)
+static int real_cmd_main(int argc, const char **argv)
{
const char *cmd;
int done_help = 0;
@@ -779,3 +780,8 @@ int cmd_main(int argc, const char **argv)
return 1;
}
+
+int cmd_main(int argc, const char **argv)
+{
+ return slog_wrap_main(real_cmd_main, argc, argv);
+}
diff --git a/structured-logging.c b/structured-logging.c
index 702fd84..afa2224 100644
--- a/structured-logging.c
+++ b/structured-logging.c
@@ -1,3 +1,10 @@
+#include "cache.h"
+#include "config.h"
+#include "version.h"
+#include "json-writer.h"
+#include "sigchain.h"
+#include "argv-array.h"
+
#if !defined(STRUCTURED_LOGGING)
/*
* Structured logging is not available.
@@ -6,4 +13,363 @@
#else
+#define SLOG_VERSION 0
+
+static uint64_t my__start_time;
+static uint64_t my__exit_time;
+static int my__is_config_loaded;
+static int my__is_enabled;
+static int my__is_pretty;
+static int my__signal;
+static int my__exit_code;
+static int my__pid;
+static int my__wrote_start_event;
+static int my__log_fd = -1;
+
+static char *my__log_path;
+static char *my__command_name;
+static char *my__sub_command_name;
+
+static struct argv_array my__argv = ARGV_ARRAY_INIT;
+static struct jso
[PATCH v1 11/25] structured-logging: add timer around do_read_index
From: Jeff Hostetler
Use a SLOG timer to record the time spent in do_read_index()
and report it in the "cmd_exit" event.
Signed-off-by: Jeff Hostetler
---
read-cache.c | 5 +
1 file changed, 5 insertions(+)
diff --git a/read-cache.c b/read-cache.c
index 3725882..df5dc87 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1900,6 +1900,7 @@ static void freshen_shared_index(const char
*shared_index, int warn)
int read_index_from(struct index_state *istate, const char *path,
const char *gitdir)
{
+ int slog_tid;
uint64_t start = getnanotime();
struct split_index *split_index;
int ret;
@@ -1910,7 +1911,9 @@ int read_index_from(struct index_state *istate, const
char *path,
if (istate->initialized)
return istate->cache_nr;
+ slog_tid = slog_start_timer("index", "do_read_index");
ret = do_read_index(istate, path, 0);
+ slog_stop_timer(slog_tid);
trace_performance_since(start, "read cache %s", path);
split_index = istate->split_index;
@@ -1926,7 +1929,9 @@ int read_index_from(struct index_state *istate, const
char *path,
base_oid_hex = oid_to_hex(&split_index->base_oid);
base_path = xstrfmt("%s/sharedindex.%s", gitdir, base_oid_hex);
+ slog_tid = slog_start_timer("index", "do_read_index");
ret = do_read_index(split_index->base, base_path, 1);
+ slog_stop_timer(slog_tid);
if (oidcmp(&split_index->base_oid, &split_index->base->oid))
die("broken index, expect %s in %s, got %s",
base_oid_hex, base_path,
--
2.9.3
[PATCH v1 15/25] structured-logging: t0420 tests for timers
From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/t0420-structured-logging.sh | 48 +++ 1 file changed, 48 insertions(+) diff --git a/t/t0420-structured-logging.sh b/t/t0420-structured-logging.sh index a594af3..37c7e83 100755 --- a/t/t0420-structured-logging.sh +++ b/t/t0420-structured-logging.sh @@ -140,4 +140,52 @@ test_expect_success PERLJSON 'parse JSON for checkout command' ' grep "row\[2\]\.sub_command path" /dev/null && + + grep -f key_cmd_exit "$LOGFILE" >event_exit && + + perl "$TEST_DIRECTORY"/t0420/parse_json.perl parsed_exit && + + grep "row\[0\]\.version\.slog 0" /dev/null && + + grep -f key_cmd_exit "$LOGFILE" >event_exit && + + perl "$TEST_DIRECTORY"/t0420/parse_json.perl parsed_exit && + + grep "row\[0\]\.version\.slog 0"
[PATCH v1 17/25] structured-logging: add aux-data for index size
From: Jeff Hostetler
Teach do_read_index() and do_write_index() to record the size of the index
in aux-data. This will be reported in the "cmd_exit" event.
Signed-off-by: Jeff Hostetler
---
read-cache.c | 4
1 file changed, 4 insertions(+)
diff --git a/read-cache.c b/read-cache.c
index 7fe66b5..b6e2cfa 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1916,6 +1916,8 @@ int read_index_from(struct index_state *istate, const
char *path,
slog_stop_timer(slog_tid);
trace_performance_since(start, "read cache %s", path);
+ slog_aux_intmax("index", "cache_nr", istate->cache_nr);
+
split_index = istate->split_index;
if (!split_index || is_null_oid(&split_index->base_oid)) {
post_read_index_from(istate);
@@ -1937,6 +1939,8 @@ int read_index_from(struct index_state *istate, const
char *path,
base_oid_hex, base_path,
oid_to_hex(&split_index->base->oid));
+ slog_aux_intmax("index", "split_index_cache_nr",
split_index->base->cache_nr);
+
freshen_shared_index(base_path, 0);
merge_base_index(istate);
post_read_index_from(istate);
--
2.9.3
[PATCH v1 10/25] structured-logging: add timer facility
From: Jeff Hostetler
Add timer facility to structured logging. This allows stopwatch-like
operations over the life of the git process. Timer data is summarized
in the "cmd_exit" event.
Signed-off-by: Jeff Hostetler
---
Documentation/config.txt | 6 ++
structured-logging.c | 180 +++
structured-logging.h | 19 +
3 files changed, 205 insertions(+)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 88f93fe..7817966 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -3189,6 +3189,12 @@ code.
This is intended to be an extendable facility where new events can easily
be added (possibly only for debugging or performance testing purposes).
+slog.timers::
+ (EXPERIMENTAL) May be set to a boolean value or a list of comma
+ separated tokens. Controls which categories of SLOG timers are
+ enabled. Defaults to off. Data for enabled timers is added to
+ the `cmd_exit` event.
+
splitIndex.maxPercentChange::
When the split index feature is used, this specifies the
percent of entries the split index can contain compared to the
diff --git a/structured-logging.c b/structured-logging.c
index 9cbf3bd..215138c 100644
--- a/structured-logging.c
+++ b/structured-logging.c
@@ -15,6 +15,26 @@
#define SLOG_VERSION 0
+struct timer_data {
+ char *category;
+ char *name;
+ uint64_t total_ns;
+ uint64_t min_ns;
+ uint64_t max_ns;
+ uint64_t start_ns;
+ int count;
+ int started;
+};
+
+struct timer_data_array {
+ struct timer_data **array;
+ size_t nr, alloc;
+};
+
+static struct timer_data_array my__timers;
+static void format_timers(struct json_writer *jw);
+static void free_timers(void);
+
static uint64_t my__start_time;
static uint64_t my__exit_time;
static int my__is_config_loaded;
@@ -41,6 +61,7 @@ struct category_filter
};
static struct category_filter my__detail_categories;
+static struct category_filter my__timer_categories;
static void set_want_categories(struct category_filter *cf, const char *value)
{
@@ -228,6 +249,12 @@ static void emit_exit_event(void)
jw_object_intmax(&jw, "slog", SLOG_VERSION);
}
jw_end(&jw);
+
+ if (my__timers.nr) {
+ jw_object_inline_begin_object(&jw, "timers");
+ format_timers(&jw);
+ jw_end(&jw);
+ }
}
jw_end(&jw);
@@ -294,6 +321,12 @@ static int cfg_detail(const char *key, const char *value)
return 0;
}
+static int cfg_timers(const char *key, const char *value)
+{
+ set_want_categories(&my__timer_categories, value);
+ return 0;
+}
+
int slog_default_config(const char *key, const char *value)
{
const char *sub;
@@ -314,6 +347,8 @@ int slog_default_config(const char *key, const char *value)
return cfg_pretty(key, value);
if (!strcmp(sub, "detail"))
return cfg_detail(key, value);
+ if (!strcmp(sub, "timers"))
+ return cfg_timers(key, value);
}
return 0;
@@ -371,6 +406,7 @@ static void do_final_steps(int in_signal)
argv_array_clear(&my__argv);
jw_release(&my__errors);
strbuf_release(&my__session_id);
+ free_timers();
}
static void slog_atexit(void)
@@ -519,4 +555,148 @@ void slog_emit_detail_event(const char *category, const
char *label,
emit_detail_event(category, label, data);
}
+int slog_start_timer(const char *category, const char *name)
+{
+ int k;
+ struct timer_data *td;
+
+ if (!want_category(&my__timer_categories, category))
+ return SLOG_UNDEFINED_TIMER_ID;
+ if (!name || !*name)
+ return SLOG_UNDEFINED_TIMER_ID;
+
+ for (k = 0; k < my__timers.nr; k++) {
+ td = my__timers.array[k];
+ if (!strcmp(category, td->category) && !strcmp(name, td->name))
+ goto start_timer;
+ }
+
+ td = xcalloc(1, sizeof(struct timer_data));
+ td->category = xstrdup(category);
+ td->name = xstrdup(name);
+ td->min_ns = UINT64_MAX;
+
+ ALLOC_GROW(my__timers.array, my__timers.nr + 1, my__timers.alloc);
+ my__timers.array[my__timers.nr++] = td;
+
+start_timer:
+ if (td->started)
+ BUG("slog.timer '%s:%s' already started",
+ td->category, td->name);
+
+ td->start_ns = getnanotime();
+ td->started = 1;
+
+ return k;
+}
+
+static void stop_timer(struct timer_data *td)
+{
+ uint64_t delta_ns = getnanotime() - td->start_ns;
+
+ td->count++;
+ td->tota
[PATCH v1 22/25] structured-logging: add child process classification
From: Jeff Hostetler
Teach git to classify child processes as "editor", "pager", "subprocess",
"alias", "shell", or "other".
Add the child process classification to the child detail events.
Mark child processes of class "editor" or "pager" as interactive in the
child detail event.
Add child summary to cmd_exit event grouping child process by class.
Signed-off-by: Jeff Hostetler
---
editor.c | 1 +
git.c| 2 +
pager.c | 1 +
run-command.h| 1 +
structured-logging.c | 119 +++++++
sub-process.c| 1 +
6 files changed, 125 insertions(+)
diff --git a/editor.c b/editor.c
index 9a9b4e1..6f5ccf3 100644
--- a/editor.c
+++ b/editor.c
@@ -66,6 +66,7 @@ int launch_editor(const char *path, struct strbuf *buffer,
const char *const *en
p.argv = args;
p.env = env;
p.use_shell = 1;
+ p.slog_child_class = "editor";
if (start_command(&p) < 0)
return error("unable to start editor '%s'", editor);
diff --git a/git.c b/git.c
index 024a40d..f1cb29e 100644
--- a/git.c
+++ b/git.c
@@ -328,6 +328,7 @@ static int handle_alias(int *argcp, const char ***argv)
commit_pager_choice();
child.use_shell = 1;
+ child.slog_child_class = "alias";
argv_array_push(&child.args, alias_string + 1);
argv_array_pushv(&child.args, (*argv) + 1);
@@ -651,6 +652,7 @@ static void execv_dashed_external(const char **argv)
cmd.clean_on_exit = 1;
cmd.wait_after_clean = 1;
cmd.silent_exec_failure = 1;
+ cmd.slog_child_class = "alias";
trace_argv_printf(cmd.args.argv, "trace: exec:");
diff --git a/pager.c b/pager.c
index a768797..5939077 100644
--- a/pager.c
+++ b/pager.c
@@ -100,6 +100,7 @@ void prepare_pager_args(struct child_process
*pager_process, const char *pager)
argv_array_push(&pager_process->args, pager);
pager_process->use_shell = 1;
setup_pager_env(&pager_process->env_array);
+ pager_process->slog_child_class = "pager";
}
void setup_pager(void)
diff --git a/run-command.h b/run-command.h
index 89c89cf..8c99bd1 100644
--- a/run-command.h
+++ b/run-command.h
@@ -13,6 +13,7 @@ struct child_process {
struct argv_array env_array;
pid_t pid;
int slog_child_id;
+ const char *slog_child_class;
/*
* Using .in, .out, .err:
* - Specify 0 for no redirections (child inherits stdin, stdout,
diff --git a/structured-logging.c b/structured-logging.c
index dbe60b7..2571e79 100644
--- a/structured-logging.c
+++ b/structured-logging.c
@@ -49,13 +49,30 @@ struct aux_data_array {
static struct aux_data_array my__aux_data;
static void format_and_free_aux_data(struct json_writer *jw);
+struct child_summary_data {
+ char *child_class;
+ uint64_t total_ns;
+ int count;
+};
+
+struct child_summary_data_array {
+ struct child_summary_data **array;
+ size_t nr, alloc;
+};
+
+static struct child_summary_data_array my__child_summary_data;
+static void format_child_summary_data(struct json_writer *jw);
+static void free_child_summary_data(void);
+
struct child_data {
uint64_t start_ns;
uint64_t end_ns;
struct json_writer jw_argv;
+ char *child_class;
unsigned int is_running:1;
unsigned int is_git_cmd:1;
unsigned int use_shell:1;
+ unsigned int is_interactive:1;
};
struct child_data_array {
@@ -293,6 +310,12 @@ static void emit_exit_event(void)
format_and_free_aux_data(&jw);
jw_end(&jw);
}
+
+ if (my__child_summary_data.nr) {
+ jw_object_inline_begin_object(&jw, "child_summary");
+ format_child_summary_data(&jw);
+ jw_end(&jw);
+ }
}
jw_end(&jw);
@@ -453,6 +476,7 @@ static void do_final_steps(int in_signal)
argv_array_clear(&my__argv);
jw_release(&my__errors);
strbuf_release(&my__session_id);
+ free_child_summary_data();
free_timers();
free_children();
}
@@ -835,6 +859,85 @@ static void format_and_free_aux_data(struct json_writer
*jw)
my__aux_data.alloc = 0;
}
+static struct child_summary_data *find_child_summary_data(
+ const struct child_data *cd)
+{
+ struct child_summary_data *csd;
+ char *child_class;
+ int k;
+
+ child_class = cd->child_class;
+ if (!child_class || !*child_class) {
+ if (cd->use
[PATCH v1 08/25] structured-logging: add detail-event facility
From: Jeff Hostetler
Add a generic "detail-event" to structured logging. This can be used
to emit context-specific events for performance or debugging purposes.
These are conceptually similar to the various GIT_TRACE_ messages.
Signed-off-by: Jeff Hostetler
---
Documentation/config.txt | 13 +++
structured-logging.c | 95
structured-logging.h | 16
3 files changed, 124 insertions(+)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index c79f2bf..88f93fe 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -3176,6 +3176,19 @@ slog.pretty::
(EXPERIMENTAL) Pretty-print structured log data when true.
(Git must be compiled with STRUCTURED_LOGGING=1.)
+slog.detail::
+ (EXPERIMENTAL) May be set to a boolean value or a list of comma
+ separated tokens. Controls which categories of optional "detail"
+ events are generated. Default to off. This is conceptually
+ similar to the different GIT_TRACE_ values.
++
+Detail events are generic events with a context-specific payload. This
+may represent a single function call or a section of performance sensitive
+code.
++
+This is intended to be an extendable facility where new events can easily
+be added (possibly only for debugging or performance testing purposes).
+
splitIndex.maxPercentChange::
When the split index feature is used, this specifies the
percent of entries the split index can contain compared to the
diff --git a/structured-logging.c b/structured-logging.c
index 289140f..9cbf3bd 100644
--- a/structured-logging.c
+++ b/structured-logging.c
@@ -34,6 +34,34 @@ static struct argv_array my__argv = ARGV_ARRAY_INIT;
static struct strbuf my__session_id = STRBUF_INIT;
static struct json_writer my__errors = JSON_WRITER_INIT;
+struct category_filter
+{
+ char *categories;
+ int want;
+};
+
+static struct category_filter my__detail_categories;
+
+static void set_want_categories(struct category_filter *cf, const char *value)
+{
+ FREE_AND_NULL(cf->categories);
+
+ cf->want = git_parse_maybe_bool(value);
+ if (cf->want == -1)
+ cf->categories = xstrdup(value);
+}
+
+static int want_category(const struct category_filter *cf, const char
*category)
+{
+ if (cf->want == 0 || cf->want == 1)
+ return cf->want;
+
+ if (!category || !*category)
+ return 0;
+
+ return !!strstr(cf->categories, category);
+}
+
/*
* Compute a new session id for the current process. Build string
* with the start time and PID of the current process and append
@@ -207,6 +235,40 @@ static void emit_exit_event(void)
jw_release(&jw);
}
+static void emit_detail_event(const char *category, const char *label,
+ const struct json_writer *data)
+{
+ struct json_writer jw = JSON_WRITER_INIT;
+ uint64_t clock_us = getnanotime() / 1000;
+
+ /* build "detail" event */
+ jw_object_begin(&jw, my__is_pretty);
+ {
+ jw_object_string(&jw, "event", "detail");
+ jw_object_intmax(&jw, "clock_us", (intmax_t)clock_us);
+ jw_object_intmax(&jw, "pid", (intmax_t)my__pid);
+ jw_object_string(&jw, "sid", my__session_id.buf);
+
+ if (my__command_name && *my__command_name)
+ jw_object_string(&jw, "command", my__command_name);
+ if (my__sub_command_name && *my__sub_command_name)
+ jw_object_string(&jw, "sub_command",
my__sub_command_name);
+
+ jw_object_inline_begin_object(&jw, "detail");
+ {
+ jw_object_string(&jw, "category", category);
+ jw_object_string(&jw, "label", label);
+ if (data)
+ jw_object_sub_jw(&jw, "data", data);
+ }
+ jw_end(&jw);
+ }
+ jw_end(&jw);
+
+ emit_event(&jw, "detail");
+ jw_release(&jw);
+}
+
static int cfg_path(const char *key, const char *value)
{
if (is_absolute_path(value)) {
@@ -226,6 +288,12 @@ static int cfg_pretty(const char *key, const char *value)
return 0;
}
+static int cfg_detail(const char *key, const char *value)
+{
+ set_want_categories(&my__detail_categories, value);
+ return 0;
+}
+
int slog_default_config(const char *key, const char *value)
{
const char *sub;
@@ -244,6 +312,8 @@ int slog_default_config(const char *key, const char *value)
return cfg_path(key, value);
if (!strcmp(sub, "pretty"))
[PATCH v1 07/25] structured-logging: t0420 basic tests
From: Jeff Hostetler
Add structured logging prereq definition "SLOG" to test-lib.sh.
Create t0420 test script with some basic tests.
Signed-off-by: Jeff Hostetler
---
t/t0420-structured-logging.sh | 143 ++
t/t0420/parse_json.perl | 52 +++
t/test-lib.sh | 1 +
3 files changed, 196 insertions(+)
create mode 100755 t/t0420-structured-logging.sh
create mode 100644 t/t0420/parse_json.perl
diff --git a/t/t0420-structured-logging.sh b/t/t0420-structured-logging.sh
new file mode 100755
index 000..a594af3
--- /dev/null
+++ b/t/t0420-structured-logging.sh
@@ -0,0 +1,143 @@
+#!/bin/sh
+
+test_description='structured logging tests'
+
+. ./test-lib.sh
+
+if ! test_have_prereq SLOG
+then
+ skip_all='skipping structured logging tests'
+ test_done
+fi
+
+LOGFILE=$TRASH_DIRECTORY/test.log
+
+test_expect_success 'setup' '
+ test_commit hello &&
+ cat >key_cmd_start <<-\EOF &&
+ "event":"cmd_start"
+ EOF
+ cat >key_cmd_exit <<-\EOF &&
+ "event":"cmd_exit"
+ EOF
+ cat >key_exit_code_0 <<-\EOF &&
+ "exit_code":0
+ EOF
+ cat >key_exit_code_129 <<-\EOF &&
+ "exit_code":129
+ EOF
+ git config --local slog.pretty false &&
+ git config --local slog.path "$LOGFILE"
+'
+
+test_expect_success 'basic events' '
+ test_when_finished "rm \"$LOGFILE\"" &&
+ git status >/dev/null &&
+ grep -f key_cmd_start "$LOGFILE" &&
+ grep -f key_cmd_exit "$LOGFILE" &&
+ grep -f key_exit_code_0 "$LOGFILE"
+'
+
+test_expect_success 'basic error code and message' '
+ test_when_finished "rm \"$LOGFILE\" event_exit" &&
+ test_expect_code 129 git status --xyzzy >/dev/null 2>/dev/null &&
+ grep -f key_cmd_exit "$LOGFILE" >event_exit &&
+ grep -f key_exit_code_129 event_exit &&
+ grep "\"errors\":" event_exit
+'
+
+test_lazy_prereq PERLJSON '
+ perl -MJSON -e "exit 0"
+'
+
+# Let perl parse the resulting JSON and dump it out.
+#
+# Since the output contains PIDs, SIDs, clock values, and the full path to
+# git[.exe] we cannot have a HEREDOC with the expected result, so we look
+# for a few key fields.
+#
+test_expect_success PERLJSON 'parse JSON for basic command' '
+ test_when_finished "rm \"$LOGFILE\" event_exit" &&
+ git status >/dev/null &&
+
+ grep -f key_cmd_exit "$LOGFILE" >event_exit &&
+
+ perl "$TEST_DIRECTORY"/t0420/parse_json.perl parsed_exit &&
+
+ grep "row\[0\]\.version\.slog 0" /dev/null &&
+ git branch --all >/dev/null &&
+ git branch new_branch >/dev/null &&
+
+ grep -f key_cmd_exit "$LOGFILE" >event_exit &&
+
+ perl "$TEST_DIRECTORY"/t0420/parse_json.perl parsed_exit &&
+
+ grep "row\[0\]\.version\.slog 0" /dev/null &&
+ git checkout master >/dev/null &&
+ git checkout -- hello.t >/dev/null &&
+
+ grep -f key_cmd_exit "$LOGFILE" >event_exit &&
+
+ perl "$TEST_DIRECTORY"/t0420/parse_json.perl parsed_exit &&
+
+ grep "row\[0\]\.version\.slog 0" 0) ? "$label_in.$k" : "$k";
+ my $value = $obj{$k};
+
+ dump_item($label, $value);
+}
+}
+
+sub dump_item {
+my ($label_in, $value) = @_;
+if (ref($value) eq 'ARRAY') {
+ print "$label_in array\n";
+ dump_array($label_in, $value);
+} elsif (ref($value) eq 'HASH') {
+ print "$label_in hash\n";
+ dump_hash($label_in, $value);
+} elsif (defined $value) {
+ print "$label_in $value\n";
+} else {
+ print "$label_in null\n";
+}
+}
+
+my $row = 0;
+while (<>) {
+my $data = decode_json( $_ );
+my $label = "row[$row]";
+
+dump_hash($label, $data);
+$row++;
+}
+
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 2831570..3d38bc7 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -1071,6 +1071,7 @@ test -n "$USE_LIBPCRE1$USE_LIBPCRE2" && test_set_prereq
PCRE
test -n "$USE_LIBPCRE1" && test_set_prereq LIBPCRE1
test -n "$USE_LIBPCRE2" && test_set_prereq LIBPCRE2
test -z "$NO_GETTEXT" && test_set_prereq GETTEXT
+test -z "$STRUCTURED_LOGGING" || test_set_prereq SLOG
# Can we rely on git's output in the C locale?
if test -n "$GETTEXT_POISON"
--
2.9.3
[PATCH v1 14/25] structured-logging: add timer around preload_index
From: Jeff Hostetler
Use a SLOG timer to record the time spend in preload_index() and
report it in the "cmd_exit" event.
Signed-off-by: Jeff Hostetler
---
preload-index.c | 6 ++
1 file changed, 6 insertions(+)
diff --git a/preload-index.c b/preload-index.c
index 4d08d44..572bb56 100644
--- a/preload-index.c
+++ b/preload-index.c
@@ -116,8 +116,14 @@ static void preload_index(struct index_state *index,
int read_index_preload(struct index_state *index,
const struct pathspec *pathspec)
{
+ int slog_tid;
int retval = read_index(index);
+ slog_tid = slog_start_timer("index", "preload");
+
preload_index(index, pathspec);
+
+ slog_stop_timer(slog_tid);
+
return retval;
}
--
2.9.3
[PATCH v1 23/25] structured-logging: t0420 tests for child process detail events
From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/t0420-structured-logging.sh | 39 +++ 1 file changed, 39 insertions(+) diff --git a/t/t0420-structured-logging.sh b/t/t0420-structured-logging.sh index 2e06cd7..4ac404d 100755 --- a/t/t0420-structured-logging.sh +++ b/t/t0420-structured-logging.sh @@ -26,6 +26,9 @@ test_expect_success 'setup' ' cat >key_exit_code_129 <<-\EOF && "exit_code":129 EOF + cat >key_detail <<-\EOF && + "event":"detail" + EOF git config --local slog.pretty false && git config --local slog.path "$LOGFILE" ' @@ -221,4 +224,40 @@ test_expect_success PERLJSON 'turn on aux-data, verify a few fields' ' grep "row\[0\]\.aux\.index\[.*\]\[0\] sparse_checkout_count" event_exit && + grep -f key_detail "$LOGFILE" >event_detail && + + perl "$TEST_DIRECTORY"/t0420/parse_json.perl parsed_exit && + perl "$TEST_DIRECTORY"/t0420/parse_json.perl parsed_detail && + + grep "row\[0\]\.event cmd_exit"
[PATCH v1 16/25] structured-logging: add aux-data facility
From: Jeff Hostetler
Add facility to add extra data to the structured logging data allowing
arbitrary key/value pair data to be added to the "cmd_exit" event.
Signed-off-by: Jeff Hostetler
---
Documentation/config.txt | 6 +++
structured-logging.c | 116 +++
structured-logging.h | 21 +
3 files changed, 143 insertions(+)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 7817966..ca78d4c 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -3195,6 +3195,12 @@ slog.timers::
enabled. Defaults to off. Data for enabled timers is added to
the `cmd_exit` event.
+slog.aux::
+ (EXPERIMENTAL) May be set to a boolean value or a list of
+ comma separated tokens. Controls which categories of SLOG
+ "aux" data are enabled. Defaults to off. "Aux" data is added
+ to the `cmd_exit` event.
+
splitIndex.maxPercentChange::
When the split index feature is used, this specifies the
percent of entries the split index can contain compared to the
diff --git a/structured-logging.c b/structured-logging.c
index 215138c..584f70a 100644
--- a/structured-logging.c
+++ b/structured-logging.c
@@ -35,6 +35,19 @@ static struct timer_data_array my__timers;
static void format_timers(struct json_writer *jw);
static void free_timers(void);
+struct aux_data {
+ char *category;
+ struct json_writer jw;
+};
+
+struct aux_data_array {
+ struct aux_data **array;
+ size_t nr, alloc;
+};
+
+static struct aux_data_array my__aux_data;
+static void format_and_free_aux_data(struct json_writer *jw);
+
static uint64_t my__start_time;
static uint64_t my__exit_time;
static int my__is_config_loaded;
@@ -62,6 +75,7 @@ struct category_filter
static struct category_filter my__detail_categories;
static struct category_filter my__timer_categories;
+static struct category_filter my__aux_categories;
static void set_want_categories(struct category_filter *cf, const char *value)
{
@@ -255,6 +269,12 @@ static void emit_exit_event(void)
format_timers(&jw);
jw_end(&jw);
}
+
+ if (my__aux_data.nr) {
+ jw_object_inline_begin_object(&jw, "aux");
+ format_and_free_aux_data(&jw);
+ jw_end(&jw);
+ }
}
jw_end(&jw);
@@ -327,6 +347,12 @@ static int cfg_timers(const char *key, const char *value)
return 0;
}
+static int cfg_aux(const char *key, const char *value)
+{
+ set_want_categories(&my__aux_categories, value);
+ return 0;
+}
+
int slog_default_config(const char *key, const char *value)
{
const char *sub;
@@ -349,6 +375,8 @@ int slog_default_config(const char *key, const char *value)
return cfg_detail(key, value);
if (!strcmp(sub, "timers"))
return cfg_timers(key, value);
+ if (!strcmp(sub, "aux"))
+ return cfg_aux(key, value);
}
return 0;
@@ -699,4 +727,92 @@ static void free_timers(void)
my__timers.alloc = 0;
}
+int slog_want_aux(const char *category)
+{
+ return want_category(&my__aux_categories, category);
+}
+
+static struct aux_data *find_aux_data(const char *category)
+{
+ struct aux_data *ad;
+ int k;
+
+ if (!slog_want_aux(category))
+ return NULL;
+
+ for (k = 0; k < my__aux_data.nr; k++) {
+ ad = my__aux_data.array[k];
+ if (!strcmp(category, ad->category))
+ return ad;
+ }
+
+ ad = xcalloc(1, sizeof(struct aux_data));
+ ad->category = xstrdup(category);
+
+ jw_array_begin(&ad->jw, my__is_pretty);
+ /* leave per-category object unterminated for now */
+
+ ALLOC_GROW(my__aux_data.array, my__aux_data.nr + 1, my__aux_data.alloc);
+ my__aux_data.array[my__aux_data.nr++] = ad;
+
+ return ad;
+}
+
+#define add_to_aux(c, k, v, fn)
\
+ do {\
+ struct aux_data *ad = find_aux_data((c)); \
+ if (ad) { \
+ jw_array_inline_begin_array(&ad->jw); \
+ { \
+ jw_array_string(&ad->jw, (k)); \
+ (fn)(&ad->jw, (v)); \
+ } \
+ jw_end(&ad->jw);\
+ }
[PATCH v1 20/25] structured-logging: add structured logging to remote-curl
From: Jeff Hostetler
remote-curl is not a builtin command and therefore, does not inherit
the common cmd_main() startup in git.c
Wrap cmd_main() with slog_cmd_main() in remote-curl to initialize
logging.
Add slog timers around push, fetch, and list verbs.
Signed-off-by: Jeff Hostetler
---
remote-curl.c | 16 +++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/remote-curl.c b/remote-curl.c
index 99b0bed..ed910f8 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -1322,8 +1322,9 @@ static int stateless_connect(const char *service_name)
return 0;
}
-int cmd_main(int argc, const char **argv)
+static int real_cmd_main(int argc, const char **argv)
{
+ int slog_tid;
struct strbuf buf = STRBUF_INIT;
int nongit;
@@ -1333,6 +1334,8 @@ int cmd_main(int argc, const char **argv)
return 1;
}
+ slog_set_command_name("remote-curl");
+
options.verbosity = 1;
options.progress = !!isatty(2);
options.thin = 1;
@@ -1362,14 +1365,20 @@ int cmd_main(int argc, const char **argv)
if (starts_with(buf.buf, "fetch ")) {
if (nongit)
die("remote-curl: fetch attempted without a
local repo");
+ slog_tid = slog_start_timer("curl", "fetch");
parse_fetch(&buf);
+ slog_stop_timer(slog_tid);
} else if (!strcmp(buf.buf, "list") || starts_with(buf.buf,
"list ")) {
int for_push = !!strstr(buf.buf + 4, "for-push");
+ slog_tid = slog_start_timer("curl", "list");
output_refs(get_refs(for_push));
+ slog_stop_timer(slog_tid);
} else if (starts_with(buf.buf, "push ")) {
+ slog_tid = slog_start_timer("curl", "push");
parse_push(&buf);
+ slog_stop_timer(slog_tid);
} else if (skip_prefix(buf.buf, "option ", &arg)) {
char *value = strchr(arg, ' ');
@@ -1411,3 +1420,8 @@ int cmd_main(int argc, const char **argv)
return 0;
}
+
+int cmd_main(int argc, const char **argv)
+{
+ return slog_wrap_main(real_cmd_main, argc, argv);
+}
--
2.9.3
[PATCH v1 21/25] structured-logging: add detail-events for child processes
From: Jeff Hostetler
Teach git to emit "detail" events with category "child" before a child
process is started and after it finishes. These events can be used to
infer time spent by git waiting for child processes to complete.
These events are controlled by the slog.detail config setting. Set to
true or add the token "child" to it.
Signed-off-by: Jeff Hostetler
---
run-command.c| 14 -
run-command.h| 1 +
structured-logging.c | 154 ++-
structured-logging.h | 15 +
4 files changed, 181 insertions(+), 3 deletions(-)
diff --git a/run-command.c b/run-command.c
index 84b883c..30fb4c5 100644
--- a/run-command.c
+++ b/run-command.c
@@ -710,6 +710,8 @@ int start_command(struct child_process *cmd)
fflush(NULL);
+ cmd->slog_child_id = slog_child_starting(cmd);
+
#ifndef GIT_WINDOWS_NATIVE
{
int notify_pipe[2];
@@ -923,6 +925,9 @@ int start_command(struct child_process *cmd)
close_pair(fderr);
else if (cmd->err)
close(cmd->err);
+
+ slog_child_ended(cmd->slog_child_id, cmd->pid, failed_errno);
+
child_process_clear(cmd);
errno = failed_errno;
return -1;
@@ -949,13 +954,20 @@ int start_command(struct child_process *cmd)
int finish_command(struct child_process *cmd)
{
int ret = wait_or_whine(cmd->pid, cmd->argv[0], 0);
+
+ slog_child_ended(cmd->slog_child_id, cmd->pid, ret);
+
child_process_clear(cmd);
return ret;
}
int finish_command_in_signal(struct child_process *cmd)
{
- return wait_or_whine(cmd->pid, cmd->argv[0], 1);
+ int ret = wait_or_whine(cmd->pid, cmd->argv[0], 1);
+
+ slog_child_ended(cmd->slog_child_id, cmd->pid, ret);
+
+ return ret;
}
diff --git a/run-command.h b/run-command.h
index 3932420..89c89cf 100644
--- a/run-command.h
+++ b/run-command.h
@@ -12,6 +12,7 @@ struct child_process {
struct argv_array args;
struct argv_array env_array;
pid_t pid;
+ int slog_child_id;
/*
* Using .in, .out, .err:
* - Specify 0 for no redirections (child inherits stdin, stdout,
diff --git a/structured-logging.c b/structured-logging.c
index 584f70a..dbe60b7 100644
--- a/structured-logging.c
+++ b/structured-logging.c
@@ -4,6 +4,7 @@
#include "json-writer.h"
#include "sigchain.h"
#include "argv-array.h"
+#include "run-command.h"
#if !defined(STRUCTURED_LOGGING)
/*
@@ -48,6 +49,23 @@ struct aux_data_array {
static struct aux_data_array my__aux_data;
static void format_and_free_aux_data(struct json_writer *jw);
+struct child_data {
+ uint64_t start_ns;
+ uint64_t end_ns;
+ struct json_writer jw_argv;
+ unsigned int is_running:1;
+ unsigned int is_git_cmd:1;
+ unsigned int use_shell:1;
+};
+
+struct child_data_array {
+ struct child_data **array;
+ size_t nr, alloc;
+};
+
+static struct child_data_array my__child_data;
+static void free_children(void);
+
static uint64_t my__start_time;
static uint64_t my__exit_time;
static int my__is_config_loaded;
@@ -283,10 +301,11 @@ static void emit_exit_event(void)
}
static void emit_detail_event(const char *category, const char *label,
+ uint64_t clock_ns,
const struct json_writer *data)
{
struct json_writer jw = JSON_WRITER_INIT;
- uint64_t clock_us = getnanotime() / 1000;
+ uint64_t clock_us = clock_ns / 1000;
/* build "detail" event */
jw_object_begin(&jw, my__is_pretty);
@@ -435,6 +454,7 @@ static void do_final_steps(int in_signal)
jw_release(&my__errors);
strbuf_release(&my__session_id);
free_timers();
+ free_children();
}
static void slog_atexit(void)
@@ -580,7 +600,7 @@ void slog_emit_detail_event(const char *category, const
char *label,
BUG("unterminated slog.detail data: '%s' '%s' '%s'",
category, label, data->json.buf);
- emit_detail_event(category, label, data);
+ emit_detail_event(category, label, getnanotime(), data);
}
int slog_start_timer(const char *category, const char *name)
@@ -815,4 +835,134 @@ static void format_and_free_aux_data(struct json_writer
*jw)
my__aux_data.alloc = 0;
}
+static struct child_data *alloc_child_data(const struct child_process *cmd)
+{
+ struct child_data *cd = xcalloc(1, sizeof(struct child_data));
+
+ cd->start_ns = getnanotime();
+ cd->is_running = 1;
+ cd->is_git_cmd = cmd->git_cmd;
+ cd->use_shell = cmd->use_shell;
+
+ jw_init(&cd->jw_argv);
+
+ jw_array_begin(&cd-
[PATCH v1 25/25] structured-logging: add config data facility
From: Jeff Hostetler
Add "config" section to "cmd_exit" event to record important
configuration settings in the log.
Add the value of "slog.detail", "slog.timers", and "slog.aux" config
settings to the log. These values control the filtering of the log.
Knowing the filter settings can help post-processors reason about
the contents of the log.
Signed-off-by: Jeff Hostetler
---
structured-logging.c | 132 +++
structured-logging.h | 13 +++++
2 files changed, 145 insertions(+)
diff --git a/structured-logging.c b/structured-logging.c
index 2571e79..0e3f79e 100644
--- a/structured-logging.c
+++ b/structured-logging.c
@@ -83,6 +83,20 @@ struct child_data_array {
static struct child_data_array my__child_data;
static void free_children(void);
+struct config_data {
+ char *group;
+ struct json_writer jw;
+};
+
+struct config_data_array {
+ struct config_data **array;
+ size_t nr, alloc;
+};
+
+static struct config_data_array my__config_data;
+static void format_config_data(struct json_writer *jw);
+static void free_config_data(void);
+
static uint64_t my__start_time;
static uint64_t my__exit_time;
static int my__is_config_loaded;
@@ -132,6 +146,15 @@ static int want_category(const struct category_filter *cf,
const char *category)
return !!strstr(cf->categories, category);
}
+static void set_config_data_from_category(const struct category_filter *cf,
+ const char *key)
+{
+ if (cf->want == 0 || cf->want == 1)
+ slog_set_config_data_intmax(key, cf->want);
+ else
+ slog_set_config_data_string(key, cf->categories);
+}
+
/*
* Compute a new session id for the current process. Build string
* with the start time and PID of the current process and append
@@ -249,6 +272,18 @@ static void emit_exit_event(void)
struct json_writer jw = JSON_WRITER_INIT;
uint64_t atexit_time = getnanotime() / 1000;
+ /*
+* Copy important (and non-obvious) config settings into the
+* "config" section of the "cmd_exit" event. The values of
+* "slog.detail", "slog.timers", and "slog.aux" are used in
+* category want filtering, so post-processors should know the
+* filter settings so that they can tell if an event is missing
+* because of filtering or an error.
+*/
+ set_config_data_from_category(&my__detail_categories, "slog.detail");
+ set_config_data_from_category(&my__timer_categories, "slog.timers");
+ set_config_data_from_category(&my__aux_categories, "slog.aux");
+
/* close unterminated forms */
if (my__errors.json.len)
jw_end(&my__errors);
@@ -299,6 +334,12 @@ static void emit_exit_event(void)
}
jw_end(&jw);
+ if (my__config_data.nr) {
+ jw_object_inline_begin_object(&jw, "config");
+ format_config_data(&jw);
+ jw_end(&jw);
+ }
+
if (my__timers.nr) {
jw_object_inline_begin_object(&jw, "timers");
format_timers(&jw);
@@ -479,6 +520,7 @@ static void do_final_steps(int in_signal)
free_child_summary_data();
free_timers();
free_children();
+ free_config_data();
}
static void slog_atexit(void)
@@ -1084,4 +1126,94 @@ static void free_children(void)
my__child_data.alloc = 0;
}
+/*
+ * Split into . (for example "slog.path" into "slog" and
"path")
+ * Find or insert in config_data_array[].
+ *
+ * Return config_data_arary[].
+ */
+static struct config_data *find_config_data(const char *key, const char
**sub_key)
+{
+ struct config_data *cd;
+ char *dot;
+ size_t group_len;
+ int k;
+
+ dot = strchr(key, '.');
+ if (!dot)
+ return NULL;
+
+ *sub_key = dot + 1;
+
+ group_len = dot - key;
+
+ for (k = 0; k < my__config_data.nr; k++) {
+ cd = my__config_data.array[k];
+ if (!strncmp(key, cd->group, group_len))
+ return cd;
+ }
+
+ cd = xcalloc(1, sizeof(struct config_data));
+ cd->group = xstrndup(key, group_len);
+
+ jw_object_begin(&cd->jw, my__is_pretty);
+ /* leave per-group object unterminated for now */
+
+ ALLOC_GROW(my__config_data.array, my__config_data.nr + 1,
+ my__config_data.alloc);
+ my__config_data.array[my__config_data.nr++] = cd;
+
+ return cd;
+}
+
+void slog_set_config_data_string(const char *key, const char *value)
+{
+ const char *sub_key;
[PATCH v1 13/25] structured-logging: add timer around wt-status functions
From: Jeff Hostetler
Use a SLOG timer to record the time spend in wt_status_collect_worktree(),
wt_status_collect_changes_initial(), wt_status_collect_changes_index(),
and wt_status_collect_untracked(). These are reported in the "cmd_exit"
event.
Signed-off-by: Jeff Hostetler
---
wt-status.c | 20
1 file changed, 20 insertions(+)
diff --git a/wt-status.c b/wt-status.c
index d1c0514..f663a37 100644
--- a/wt-status.c
+++ b/wt-status.c
@@ -580,8 +580,11 @@ static void wt_status_collect_updated_cb(struct
diff_queue_struct *q,
static void wt_status_collect_changes_worktree(struct wt_status *s)
{
+ int slog_tid;
struct rev_info rev;
+ slog_tid = slog_start_timer("status", "worktree");
+
init_revisions(&rev, NULL);
setup_revisions(0, NULL, &rev, NULL);
rev.diffopt.output_format |= DIFF_FORMAT_CALLBACK;
@@ -600,13 +603,18 @@ static void wt_status_collect_changes_worktree(struct
wt_status *s)
rev.diffopt.rename_score = s->rename_score >= 0 ? s->rename_score :
rev.diffopt.rename_score;
copy_pathspec(&rev.prune_data, &s->pathspec);
run_diff_files(&rev, 0);
+
+ slog_stop_timer(slog_tid);
}
static void wt_status_collect_changes_index(struct wt_status *s)
{
+ int slog_tid;
struct rev_info rev;
struct setup_revision_opt opt;
+ slog_tid = slog_start_timer("status", "changes_index");
+
init_revisions(&rev, NULL);
memset(&opt, 0, sizeof(opt));
opt.def = s->is_initial ? empty_tree_oid_hex() : s->reference;
@@ -636,12 +644,17 @@ static void wt_status_collect_changes_index(struct
wt_status *s)
rev.diffopt.rename_score = s->rename_score >= 0 ? s->rename_score :
rev.diffopt.rename_score;
copy_pathspec(&rev.prune_data, &s->pathspec);
run_diff_index(&rev, 1);
+
+ slog_stop_timer(slog_tid);
}
static void wt_status_collect_changes_initial(struct wt_status *s)
{
+ int slog_tid;
int i;
+ slog_tid = slog_start_timer("status", "changes_initial");
+
for (i = 0; i < active_nr; i++) {
struct string_list_item *it;
struct wt_status_change_data *d;
@@ -672,10 +685,13 @@ static void wt_status_collect_changes_initial(struct
wt_status *s)
oidcpy(&d->oid_index, &ce->oid);
}
}
+
+ slog_stop_timer(slog_tid);
}
static void wt_status_collect_untracked(struct wt_status *s)
{
+ int slog_tid;
int i;
struct dir_struct dir;
uint64_t t_begin = getnanotime();
@@ -683,6 +699,8 @@ static void wt_status_collect_untracked(struct wt_status *s)
if (!s->show_untracked_files)
return;
+ slog_tid = slog_start_timer("status", "untracked");
+
memset(&dir, 0, sizeof(dir));
if (s->show_untracked_files != SHOW_ALL_UNTRACKED_FILES)
dir.flags |=
@@ -722,6 +740,8 @@ static void wt_status_collect_untracked(struct wt_status *s)
if (advice_status_u_option)
s->untracked_in_ms = (getnanotime() - t_begin) / 100;
+
+ slog_stop_timer(slog_tid);
}
void wt_status_collect(struct wt_status *s)
--
2.9.3
[PATCH v1 24/25] structured-logging: t0420 tests for interacitve child_summary
From: Jeff Hostetler Test running a command with a fake pager and verify that a child_summary is generated. Signed-off-by: Jeff Hostetler --- t/t0420-structured-logging.sh | 30 ++ 1 file changed, 30 insertions(+) diff --git a/t/t0420-structured-logging.sh b/t/t0420-structured-logging.sh index 4ac404d..69f811a 100755 --- a/t/t0420-structured-logging.sh +++ b/t/t0420-structured-logging.sh @@ -260,4 +260,34 @@ test_expect_success PERLJSON 'verify child start/end events during clone' ' grep "row\[1\]\.detail\.data\.child_exit_code 0" event_exit && + + perl "$TEST_DIRECTORY"/t0420/parse_json.perl parsed_exit && + + grep "row\[0\]\.child_summary\.pager\.count 1"
[PATCH v1 12/25] structured-logging: add timer around do_write_index
From: Jeff Hostetler
Use a SLOG timer to record the time spend in do_write_index() and
report it in the "cmd_exit" event.
Signed-off-by: Jeff Hostetler
---
read-cache.c | 5 +
1 file changed, 5 insertions(+)
diff --git a/read-cache.c b/read-cache.c
index df5dc87..7fe66b5 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -2433,7 +2433,9 @@ static int commit_locked_index(struct lock_file *lk)
static int do_write_locked_index(struct index_state *istate, struct lock_file
*lock,
unsigned flags)
{
+ int slog_timer = slog_start_timer("index", "do_write_index");
int ret = do_write_index(istate, lock->tempfile, 0);
+ slog_stop_timer(slog_timer);
if (ret)
return ret;
if (flags & COMMIT_LOCK)
@@ -2514,11 +2516,14 @@ static int clean_shared_index_files(const char
*current_hex)
static int write_shared_index(struct index_state *istate,
struct tempfile **temp)
{
+ int slog_tid = SLOG_UNDEFINED_TIMER_ID;
struct split_index *si = istate->split_index;
int ret;
move_cache_to_base_index(istate);
+ slog_tid = slog_start_timer("index", "do_write_index");
ret = do_write_index(si->base, *temp, 1);
+ slog_stop_timer(slog_tid);
if (ret)
return ret;
ret = adjust_shared_perm(get_tempfile_path(*temp));
--
2.9.3
[PATCH v1 18/25] structured-logging: add aux-data for size of sparse-checkout file
From: Jeff Hostetler
Teach unpack_trees() to record the number of entries in the sparse-checkout
file in the aux-data. This will be reported in the "cmd_exit" event.
Signed-off-by: Jeff Hostetler
---
unpack-trees.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/unpack-trees.c b/unpack-trees.c
index 3a85a02..71b1b93 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1285,8 +1285,10 @@ int unpack_trees(unsigned len, struct tree_desc *t,
struct unpack_trees_options
char *sparse = git_pathdup("info/sparse-checkout");
if (add_excludes_from_file_to_list(sparse, "", 0, &el, NULL) <
0)
o->skip_sparse_checkout = 1;
- else
+ else {
o->el = ⪙
+ slog_aux_intmax("index", "sparse_checkout_count",
el.nr);
+ }
free(sparse);
}
--
2.9.3
[PATCH v1 19/25] structured-logging: t0420 tests for aux-data
From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/t0420-structured-logging.sh | 33 + 1 file changed, 33 insertions(+) diff --git a/t/t0420-structured-logging.sh b/t/t0420-structured-logging.sh index 37c7e83..2e06cd7 100755 --- a/t/t0420-structured-logging.sh +++ b/t/t0420-structured-logging.sh @@ -188,4 +188,37 @@ test_expect_success PERLJSON 'turn on index timers only' ' test_expect_code 1 grep "row\[0\]\.timers\.status\.untracked\.total_us" .git/info/sparse-checkout && + git config --local core.sparsecheckout true && + git config --local slog.aux foo,index,bar && + rm -f "$LOGFILE" && + + git checkout HEAD && + + grep -f key_cmd_exit "$LOGFILE" >event_exit && + + perl "$TEST_DIRECTORY"/t0420/parse_json.perl parsed_exit && + + grep "row\[0\]\.version\.slog 0" ][0] cache_nr + # row[0].aux.index[][1] 1 + # row[0].aux.index[][0] sparse_checkout_count + # row[0].aux.index[][1] 1 + # + # But do not assume values for and (in case the sorting changes + # or other "aux" fields are added later). + + grep "row\[0\]\.aux\.index\[.*\]\[0\] cache_nr"
[PATCH v1 00/25] RFC: structured logging
From: Jeff Hostetler
This RFC patch series adds structured logging to git. The motivation,
background, and limitations of this feature are described at the
beginning of the design document in the first commit. The design
document also contains a section comparing this feature with the
existing GIT_TRACE feature. So I won't go into great detail here in
the cover letter.
My primary focus in this RFC is to reach agreement on the structured
logging facility. This includes the basic approach and the various
logging fields and timers.
This patch series also includes several example usage commits, such as
adding timers around do_{read,write}_index, that demonstrate the
capabilities of the structured logging facility. I only added a few
examples for things that I think we'll want long-term. I did not
attempt to instrument everything.
This patch series requires V11 of my json-writer patch series.
Jeff Hostetler (25):
structured-logging: design document
structured-logging: add STRUCTURED_LOGGING=1 to Makefile
structured-logging: add structured logging framework
structured-logging: add session-id to log events
structured-logging: set sub_command field for branch command
structured-logging: set sub_command field for checkout command
structured-logging: t0420 basic tests
structured-logging: add detail-event facility
structured-logging: add detail-event for lazy_init_name_hash
structured-logging: add timer facility
structured-logging: add timer around do_read_index
structured-logging: add timer around do_write_index
structured-logging: add timer around wt-status functions
structured-logging: add timer around preload_index
structured-logging: t0420 tests for timers
structured-logging: add aux-data facility
structured-logging: add aux-data for index size
structured-logging: add aux-data for size of sparse-checkout file
structured-logging: t0420 tests for aux-data
structured-logging: add structured logging to remote-curl
structured-logging: add detail-events for child processes
structured-logging: add child process classification
structured-logging: t0420 tests for child process detail events
structured-logging: t0420 tests for interacitve child_summary
structured-logging: add config data facility
Documentation/config.txt | 33 +
Documentation/git.txt |6 +
Documentation/technical/structured-logging.txt | 816
Makefile |8 +
builtin/branch.c |8 +
builtin/checkout.c |7 +
compat/mingw.h |7 +
config.c |3 +
editor.c | 1 +
git-compat-util.h |9 +
git.c | 10 +-
name-hash.c| 26 +
pager.c|1 +
preload-index.c|6 +
read-cache.c | 14 +
remote-curl.c | 16 +-
run-command.c | 14 +-
run-command.h |2 +
structured-logging.c | 1219
structured-logging.h | 179
sub-process.c |1 +
t/t0001-init.sh|1 +
t/t0420-structured-logging.sh | 293 ++
t/t0420/parse_json.perl| 52 +
t/test-lib.sh |1 +
unpack-trees.c |4 +-
usage.c|4 +
wt-status.c| 20 +
28 files changed, 2757 insertions(+), 4 deletions(-)
create mode 100644 Documentation/technical/structured-logging.txt
create mode 100644 structured-logging.c
create mode 100644 structured-logging.h
create mode 100755 t/t0420-structured-logging.sh
create mode 100644 t/t0420/parse_json.perl
--
2.9.3
[PATCH v1 02/25] structured-logging: add STRUCTURED_LOGGING=1 to Makefile
From: Jeff Hostetler
Teach the Makefile to take STRUCTURED_LOGGING=1 variable to
compile in/out structured logging feature.
Signed-off-by: Jeff Hostetler
---
Makefile | 8
structured-logging.c | 9 +
structured-logging.h | 13 +
3 files changed, 30 insertions(+)
create mode 100644 structured-logging.c
create mode 100644 structured-logging.h
diff --git a/Makefile b/Makefile
index 39ca66b..ccc39bf 100644
--- a/Makefile
+++ b/Makefile
@@ -442,6 +442,8 @@ all::
# When cross-compiling, define HOST_CPU as the canonical name of the CPU on
# which the built Git will run (for instance "x86_64").
#
+# Define STRUCTURED_LOGGING if you want structured logging to be available.
+#
# Define RUNTIME_PREFIX to configure Git to resolve its ancillary tooling and
# support files relative to the location of the runtime binary, rather than
# hard-coding them into the binary. Git installations built with RUNTIME_PREFIX
@@ -955,6 +957,7 @@ LIB_OBJS += split-index.o
LIB_OBJS += strbuf.o
LIB_OBJS += streaming.o
LIB_OBJS += string-list.o
+LIB_OBJS += structured-logging.o
LIB_OBJS += submodule.o
LIB_OBJS += submodule-config.o
LIB_OBJS += sub-process.o
@@ -1326,6 +1329,10 @@ ifdef ZLIB_PATH
endif
EXTLIBS += -lz
+ifdef STRUCTURED_LOGGING
+ BASIC_CFLAGS += -DSTRUCTURED_LOGGING
+endif
+
ifndef NO_OPENSSL
OPENSSL_LIBSSL = -lssl
ifdef OPENSSLDIR
@@ -2543,6 +2550,7 @@ GIT-BUILD-OPTIONS: FORCE
@echo TAR=\''$(subst ','\'',$(subst ','\'',$(TAR)))'\' >>$@+
@echo NO_CURL=\''$(subst ','\'',$(subst ','\'',$(NO_CURL)))'\' >>$@+
@echo NO_EXPAT=\''$(subst ','\'',$(subst ','\'',$(NO_EXPAT)))'\' >>$@+
+ @echo STRUCTURED_LOGGING=\''$(subst ','\'',$(subst
','\'',$(STRUCTURED_LOGGING)))'\' >>$@+
@echo USE_LIBPCRE1=\''$(subst ','\'',$(subst
','\'',$(USE_LIBPCRE1)))'\' >>$@+
@echo USE_LIBPCRE2=\''$(subst ','\'',$(subst
','\'',$(USE_LIBPCRE2)))'\' >>$@+
@echo NO_LIBPCRE1_JIT=\''$(subst ','\'',$(subst
','\'',$(NO_LIBPCRE1_JIT)))'\' >>$@+
diff --git a/structured-logging.c b/structured-logging.c
new file mode 100644
index 000..702fd84
--- /dev/null
+++ b/structured-logging.c
@@ -0,0 +1,9 @@
+#if !defined(STRUCTURED_LOGGING)
+/*
+ * Structured logging is not available.
+ * Stub out all API routines.
+ */
+
+#else
+
+#endif
diff --git a/structured-logging.h b/structured-logging.h
new file mode 100644
index 000..c9e8c1d
--- /dev/null
+++ b/structured-logging.h
@@ -0,0 +1,13 @@
+#ifndef STRUCTURED_LOGGING_H
+#define STRUCTURED_LOGGING_H
+
+#if !defined(STRUCTURED_LOGGING)
+/*
+ * Structured logging is not available.
+ * Stub out all API routines.
+ */
+
+#else
+
+#endif /* STRUCTURED_LOGGING */
+#endif /* STRUCTURED_LOGGING_H */
--
2.9.3
[PATCH v1 01/25] structured-logging: design document
From: Jeff Hostetler
Signed-off-by: Jeff Hostetler
---
Documentation/technical/structured-logging.txt | 816 +
1 file changed, 816 insertions(+)
create mode 100644 Documentation/technical/structured-logging.txt
diff --git a/Documentation/technical/structured-logging.txt
b/Documentation/technical/structured-logging.txt
new file mode 100644
index 000..794c614
--- /dev/null
+++ b/Documentation/technical/structured-logging.txt
@@ -0,0 +1,816 @@
+Structured Logging
+==
+
+Structured Logging (SLOG) is an optional feature to allow Git to
+generate structured log data for executed commands. This includes
+command line arguments, command run times, error codes and messages,
+child process information, time spent in various critical functions,
+and repository data-shape information. Data is written to a target
+log file in JSON[1,2,3] format.
+
+SLOG is disabled by default. Several steps are required to enable it:
+
+1. Add the compile-time flag "STRUCTURED_LOGGING=1" when building git
+ to include the SLOG routines in the git executable.
+
+2. Set "slog.*" config settings[5] to enable SLOG in your repo.
+
+
+Motivation
+==
+
+Git users may be faced with scenarios that are surprisingly slow or
+produce unexpected results. And Git developers may have difficulty
+reproducing these experiences. Structured logging allows users to
+provide developers with additional usage, performance and error data
+that can help diagnose and debug issues.
+
+Many Git hosting providers and users with many developers have bespoke
+efforts to help troubleshoot problems; for example, command wrappers,
+custom pre- and post-command hooks, and custom instrumentation of Git
+code. These are inefficient and/or difficult to maintain. The goal
+of SLOG is to provide this data as efficiently as possible.
+
+And having structured rather than free format log data, will help
+developers with their analysis.
+
+
+Background (Git Merge 2018 Barcelona)
+=
+
+Performance and/or error logging was discussed during the contributor's
+summit in Barcelona. Here are the relevant notes from the meeting
+minutes[6].
+
+> Performance misc (Ævar)
+> ---
+> [...]
+> - central error reporting for git
+>- `git status` logging
+>- git config that collects data, pushes to known endpoint with `git push`
+>- pre_command and post_command hooks, for logs
+>- `gvfs diagnose` that looks at packfiles, etc
+>- detect BSODs, etc
+>- Dropbox writes out json with index properties and command-line
+>information for status/fetch/push, fork/execs external tool to upload
+>- windows trace facility; would be nice to have cross-platform
+>- would hosting providers care?
+>- zipfile of logs to give when debugging
+>- sanitizing data is harder
+>- more in a company setting
+>- fileshare to upload zipfile
+>- most of the errors are proxy when they shouldn't, wrong proxy, proxy
+>specific to particular URL; so upload endpoint wouldn't work
+>- GIT_TRACE is supposed to be that (for proxy)
+>- but we need more trace variables
+>- series to make tracing cheaper
+>- except that curl selects the proxy
+>- trace should have an API, so it can call an executable
+>- dump to .git/traces/... and everything else happens externally
+>- tools like visual studio can't set GIT_TRACE, so
+>- sourcetree has seen user environments where commands just take forever
+>- third-party tools like perf/strace - could we be better leveraging
those?
+>- distribute turn-key solution to handout to collect more data?
+
+
+A Quick Example
+===
+
+Note: JSON pretty-printing is enabled in all of the examples shown in
+this document. When pretty-printing is turned off, each event is
+written on a single line. Pretty-printing is intended for debugging.
+It should be turned off in production to make post-processing easier.
+
+$ git config slog.pretty
+
+Here is a quick example showing SLOG data for "git status". This
+example has all optional features turned off. It contains 2 events.
+The first is generated when the command started and the second when it
+ended.
+
+{
+ "event": "cmd_start",
+ "clock_us": 1530273550667800,
+ "pid": 107270,
+ "sid": "1530273550667800-107270",
+ "command": "status",
+ "argv": [
+"./git",
+"status"
+ ]
+}
+{
+ "event": "cmd_exit",
+ "clock_us": 1530273550680460,
+ "pid": 107270,
+ "sid": "1530273550667800-107270",
+ "command": "status",
+ "argv": [
+"./git",
+"statu
[PATCH v1 04/25] structured-logging: add session-id to log events
From: Jeff Hostetler
Teach git to create a unique session id (SID) during structured
logging initialization and use that SID in all log events.
This SID is exported into a transient environment variable and
inherited by child processes. This allows git child processes
to be related back to the parent git process event if there are
intermediate /bin/sh processes between them.
Update t0001 to ignore the environment variable GIT_SLOG_PARENT_SID.
Signed-off-by: Jeff Hostetler
---
Documentation/git.txt | 6 ++
structured-logging.c | 52 +++
t/t0001-init.sh | 1 +
3 files changed, 59 insertions(+)
diff --git a/Documentation/git.txt b/Documentation/git.txt
index dba7f0c..a24f399 100644
--- a/Documentation/git.txt
+++ b/Documentation/git.txt
@@ -766,6 +766,12 @@ standard output.
adequate and support for it is likely to be removed in the
foreseeable future (along with the variable).
+`GIT_SLOG_PARENT_SID`::
+ (Experimental) A transient environment variable set by top-level
+ Git commands and inherited by child Git commands. It contains
+ a session id that will be written the structured logging output
+ to help associate child and parent processes.
+
Discussion[[Discussion]]
diff --git a/structured-logging.c b/structured-logging.c
index afa2224..289140f 100644
--- a/structured-logging.c
+++ b/structured-logging.c
@@ -31,9 +31,57 @@ static char *my__command_name;
static char *my__sub_command_name;
static struct argv_array my__argv = ARGV_ARRAY_INIT;
+static struct strbuf my__session_id = STRBUF_INIT;
static struct json_writer my__errors = JSON_WRITER_INIT;
/*
+ * Compute a new session id for the current process. Build string
+ * with the start time and PID of the current process and append
+ * the inherited session id from our parent process (if present).
+ * The parent session id may include its parent session id.
+ *
+ * sid := '-' [ ':' [ ... ] ]
+ */
+static void compute_our_sid(void)
+{
+ const char *parent_sid;
+
+ if (my__session_id.len)
+ return;
+
+ /*
+* A "session id" (SID) is a cheap, unique-enough string to
+* associate child process with the hierarchy of invoking git
+* processes.
+*
+* This is stronger than a simple parent-pid because we may
+* have an intermediate shell between a top-level Git command
+* and a child Git command. It also isolates from issues
+* about how the OS recycles PIDs.
+*
+* This could be a UUID/GUID, but that is overkill for our
+* needs here and more expensive to compute.
+*
+* Consumers should consider this an unordered opaque string
+* in case we decide to switch to a real UUID in the future.
+*/
+ strbuf_addf(&my__session_id, "%"PRIuMAX"-%"PRIdMAX,
+ (uintmax_t)my__start_time, (intmax_t)my__pid);
+
+ parent_sid = getenv("GIT_SLOG_PARENT_SID");
+ if (parent_sid && *parent_sid) {
+ strbuf_addch(&my__session_id, ':');
+ strbuf_addstr(&my__session_id, parent_sid);
+ }
+
+ /*
+* Install our SID into the environment for our child processes
+* to inherit.
+*/
+ setenv("GIT_SLOG_PARENT_SID", my__session_id.buf, 1);
+}
+
+/*
* Write a single event to the structured log file.
*/
static void emit_event(struct json_writer *jw, const char *event_name)
@@ -75,6 +123,7 @@ static void emit_start_event(void)
jw_object_string(&jw, "event", "cmd_start");
jw_object_intmax(&jw, "clock_us", (intmax_t)my__start_time);
jw_object_intmax(&jw, "pid", (intmax_t)my__pid);
+ jw_object_string(&jw, "sid", my__session_id.buf);
if (my__command_name && *my__command_name)
jw_object_string(&jw, "command", my__command_name);
@@ -112,6 +161,7 @@ static void emit_exit_event(void)
jw_object_string(&jw, "event", "cmd_exit");
jw_object_intmax(&jw, "clock_us", (intmax_t)atexit_time);
jw_object_intmax(&jw, "pid", (intmax_t)my__pid);
+ jw_object_string(&jw, "sid", my__session_id.buf);
if (my__command_name && *my__command_name)
jw_object_string(&jw, "command", my__command_name);
@@ -250,6 +300,7 @@ static void do_final_steps(int in_signal)
free(my__sub_command_name);
argv_array_clear(&my__argv);
jw_release(&my__errors);
+ strbuf_release(&my__session_id);
}
static void slog_atexit(void)
@@
[PATCH v1 06/25] structured-logging: set sub_command field for checkout command
From: Jeff Hostetler
Signed-off-by: Jeff Hostetler
---
builtin/checkout.c | 7 +++
1 file changed, 7 insertions(+)
diff --git a/builtin/checkout.c b/builtin/checkout.c
index 2e1d237..d05890b 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -249,6 +249,8 @@ static int checkout_paths(const struct checkout_opts *opts,
int errs = 0;
struct lock_file lock_file = LOCK_INIT;
+ slog_set_sub_command_name(opts->patch_mode ? "patch" : "path");
+
if (opts->track != BRANCH_TRACK_UNSPECIFIED)
die(_("'%s' cannot be used with updating paths"), "--track");
@@ -826,6 +828,9 @@ static int switch_branches(const struct checkout_opts *opts,
void *path_to_free;
struct object_id rev;
int flag, writeout_error = 0;
+
+ slog_set_sub_command_name("switch_branch");
+
memset(&old_branch_info, 0, sizeof(old_branch_info));
old_branch_info.path = path_to_free = resolve_refdup("HEAD", 0, &rev,
&flag);
if (old_branch_info.path)
@@ -1037,6 +1042,8 @@ static int switch_unborn_to_new_branch(const struct
checkout_opts *opts)
int status;
struct strbuf branch_ref = STRBUF_INIT;
+ slog_set_sub_command_name("switch_unborn_to_new_branch");
+
if (!opts->new_branch)
die(_("You are on a branch yet to be born"));
strbuf_addf(&branch_ref, "refs/heads/%s", opts->new_branch);
--
2.9.3
[PATCH v11] json_writer: new routines to create JSON data
From: Jeff Hostetler
Add "struct json_writer" and a series of jw_ routines to compose JSON
data into a string buffer. The resulting string may then be printed by
commands wanting to support a JSON-like output format.
The json_writer is limited to correctly formatting structured data for
output. It does not attempt to build an object model of the JSON data.
We say "JSON-like" because we do not enforce the Unicode (usually UTF-8)
requirement on string fields. Internally, Git does not necessarily have
Unicode/UTF-8 data for most fields, so it is currently unclear the best
way to enforce that requirement. For example, on Linux pathnames can
contain arbitrary 8-bit character data, so a command like "status" would
not know how to encode the reported pathnames. We may want to revisit
this (or double encode such strings) in the future.
Helped-by: Eric Sunshine
Helped-by: René Scharfe
Helped-by: Wink Saville
Helped-by: Ramsay Jones
Signed-off-by: Jeff Hostetler
---
Makefile| 2 +
json-writer.c | 414
json-writer.h | 105
t/helper/test-json-writer.c | 565
t/helper/test-tool.c| 1 +
t/helper/test-tool.h| 1 +
t/t0019-json-writer.sh | 331 ++
t/t0019/parse_json.perl | 52
8 files changed, 1471 insertions(+)
create mode 100644 json-writer.c
create mode 100644 json-writer.h
create mode 100644 t/helper/test-json-writer.c
create mode 100755 t/t0019-json-writer.sh
create mode 100644 t/t0019/parse_json.perl
diff --git a/Makefile b/Makefile
index e4b503d..39ca66b 100644
--- a/Makefile
+++ b/Makefile
@@ -709,6 +709,7 @@ TEST_BUILTINS_OBJS += test-example-decorate.o
TEST_BUILTINS_OBJS += test-genrandom.o
TEST_BUILTINS_OBJS += test-hashmap.o
TEST_BUILTINS_OBJS += test-index-version.o
+TEST_BUILTINS_OBJS += test-json-writer.o
TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o
TEST_BUILTINS_OBJS += test-match-trees.o
TEST_BUILTINS_OBJS += test-mergesort.o
@@ -871,6 +872,7 @@ LIB_OBJS += hashmap.o
LIB_OBJS += help.o
LIB_OBJS += hex.o
LIB_OBJS += ident.o
+LIB_OBJS += json-writer.o
LIB_OBJS += kwset.o
LIB_OBJS += levenshtein.o
LIB_OBJS += line-log.o
diff --git a/json-writer.c b/json-writer.c
new file mode 100644
index 000..aadb9db
--- /dev/null
+++ b/json-writer.c
@@ -0,0 +1,414 @@
+#include "cache.h"
+#include "json-writer.h"
+
+void jw_init(struct json_writer *jw)
+{
+ strbuf_init(&jw->json, 0);
+ strbuf_init(&jw->open_stack, 0);
+ jw->need_comma = 0;
+ jw->pretty = 0;
+}
+
+void jw_release(struct json_writer *jw)
+{
+ strbuf_release(&jw->json);
+ strbuf_release(&jw->open_stack);
+}
+
+/*
+ * Append JSON-quoted version of the given string to 'out'.
+ */
+static void append_quoted_string(struct strbuf *out, const char *in)
+{
+ unsigned char c;
+
+ strbuf_addch(out, '"');
+ while ((c = *in++) != '\0') {
+ if (c == '"')
+ strbuf_addstr(out, "\\\"");
+ else if (c == '\\')
+ strbuf_addstr(out, "");
+ else if (c == '\n')
+ strbuf_addstr(out, "\\n");
+ else if (c == '\r')
+ strbuf_addstr(out, "\\r");
+ else if (c == '\t')
+ strbuf_addstr(out, "\\t");
+ else if (c == '\f')
+ strbuf_addstr(out, "\\f");
+ else if (c == '\b')
+ strbuf_addstr(out, "\\b");
+ else if (c < 0x20)
+ strbuf_addf(out, "\\u%04x", c);
+ else
+ strbuf_addch(out, c);
+ }
+ strbuf_addch(out, '"');
+}
+
+static void indent_pretty(struct json_writer *jw)
+{
+ int k;
+
+ for (k = 0; k < jw->open_stack.len; k++)
+ strbuf_addstr(&jw->json, " ");
+}
+
+/*
+ * Begin an object or array (either top-level or nested within the currently
+ * open object or array).
+ */
+static void begin(struct json_writer *jw, char ch_open, int pretty)
+{
+ jw->pretty = pretty;
+
+ strbuf_addch(&jw->json, ch_open);
+
+ strbuf_addch(&jw->open_stack, ch_open);
+ jw->need_comma = 0;
+}
+
+/*
+ * Assert that the top of the open-stack is an object.
+ */
+static void assert_in_object(const struct json_writer *jw, const char *key)
+{
+ if (!jw->open_stack.len)
+ BUG("json-writer: object: missing jw_object_begin(): '%s'",
key);
+ if (jw->open_stack.buf
[PATCH v11] json_writer V10
From: Jeff Hostetler Here is V11 of my json-writer patch. I fixed a minor initialization bug in V10 and rebased it onto v2.18.0 (from an 2.18.0-RC version). Jeff Hostetler (1): json_writer: new routines to create JSON data Makefile| 2 + json-writer.c | 414 json-writer.h | 105 t/helper/test-json-writer.c | 565 t/helper/test-tool.c| 1 + t/helper/test-tool.h| 1 + t/t0019-json-writer.sh | 331 ++ t/t0019/parse_json.perl | 52 8 files changed, 1471 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh create mode 100644 t/t0019/parse_json.perl -- 2.9.3
[PATCH v10] json_writer V10
From: Jeff Hostetler Here is V10 of my json-writer patch. I fixed the DEVELOPER=1 warnings that Eric pointed out and refactored the indent_pretty() code as Junio suggested. Jeff Hostetler (1): json_writer: new routines to create JSON data Makefile| 2 + json-writer.c | 414 json-writer.h | 105 t/helper/test-json-writer.c | 565 t/helper/test-tool.c| 1 + t/helper/test-tool.h| 1 + t/t0019-json-writer.sh | 331 ++ t/t0019/parse_json.perl | 52 8 files changed, 1471 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh create mode 100644 t/t0019/parse_json.perl -- 2.9.3
[PATCH v10] json_writer: new routines to create JSON data
From: Jeff Hostetler
Add "struct json_writer" and a series of jw_ routines to compose JSON
data into a string buffer. The resulting string may then be printed by
commands wanting to support a JSON-like output format.
The json_writer is limited to correctly formatting structured data for
output. It does not attempt to build an object model of the JSON data.
We say "JSON-like" because we do not enforce the Unicode (usually UTF-8)
requirement on string fields. Internally, Git does not necessarily have
Unicode/UTF-8 data for most fields, so it is currently unclear the best
way to enforce that requirement. For example, on Linux pathnames can
contain arbitrary 8-bit character data, so a command like "status" would
not know how to encode the reported pathnames. We may want to revisit
this (or double encode such strings) in the future.
Helped-by: Eric Sunshine
Helped-by: René Scharfe
Helped-by: Wink Saville
Helped-by: Ramsay Jones
Signed-off-by: Jeff Hostetler
---
Makefile| 2 +
json-writer.c | 414
json-writer.h | 105
t/helper/test-json-writer.c | 565
t/helper/test-tool.c| 1 +
t/helper/test-tool.h| 1 +
t/t0019-json-writer.sh | 331 ++
t/t0019/parse_json.perl | 52
8 files changed, 1471 insertions(+)
create mode 100644 json-writer.c
create mode 100644 json-writer.h
create mode 100644 t/helper/test-json-writer.c
create mode 100755 t/t0019-json-writer.sh
create mode 100644 t/t0019/parse_json.perl
diff --git a/Makefile b/Makefile
index 1d27f36..5a781e2 100644
--- a/Makefile
+++ b/Makefile
@@ -709,6 +709,7 @@ TEST_BUILTINS_OBJS += test-example-decorate.o
TEST_BUILTINS_OBJS += test-genrandom.o
TEST_BUILTINS_OBJS += test-hashmap.o
TEST_BUILTINS_OBJS += test-index-version.o
+TEST_BUILTINS_OBJS += test-json-writer.o
TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o
TEST_BUILTINS_OBJS += test-match-trees.o
TEST_BUILTINS_OBJS += test-mergesort.o
@@ -871,6 +872,7 @@ LIB_OBJS += hashmap.o
LIB_OBJS += help.o
LIB_OBJS += hex.o
LIB_OBJS += ident.o
+LIB_OBJS += json-writer.o
LIB_OBJS += kwset.o
LIB_OBJS += levenshtein.o
LIB_OBJS += line-log.o
diff --git a/json-writer.c b/json-writer.c
new file mode 100644
index 000..115de3e
--- /dev/null
+++ b/json-writer.c
@@ -0,0 +1,414 @@
+#include "cache.h"
+#include "json-writer.h"
+
+void jw_init(struct json_writer *jw)
+{
+ strbuf_reset(&jw->json);
+ strbuf_reset(&jw->open_stack);
+ jw->need_comma = 0;
+ jw->pretty = 0;
+}
+
+void jw_release(struct json_writer *jw)
+{
+ strbuf_release(&jw->json);
+ strbuf_release(&jw->open_stack);
+}
+
+/*
+ * Append JSON-quoted version of the given string to 'out'.
+ */
+static void append_quoted_string(struct strbuf *out, const char *in)
+{
+ unsigned char c;
+
+ strbuf_addch(out, '"');
+ while ((c = *in++) != '\0') {
+ if (c == '"')
+ strbuf_addstr(out, "\\\"");
+ else if (c == '\\')
+ strbuf_addstr(out, "");
+ else if (c == '\n')
+ strbuf_addstr(out, "\\n");
+ else if (c == '\r')
+ strbuf_addstr(out, "\\r");
+ else if (c == '\t')
+ strbuf_addstr(out, "\\t");
+ else if (c == '\f')
+ strbuf_addstr(out, "\\f");
+ else if (c == '\b')
+ strbuf_addstr(out, "\\b");
+ else if (c < 0x20)
+ strbuf_addf(out, "\\u%04x", c);
+ else
+ strbuf_addch(out, c);
+ }
+ strbuf_addch(out, '"');
+}
+
+static void indent_pretty(struct json_writer *jw)
+{
+ int k;
+
+ for (k = 0; k < jw->open_stack.len; k++)
+ strbuf_addstr(&jw->json, " ");
+}
+
+/*
+ * Begin an object or array (either top-level or nested within the currently
+ * open object or array).
+ */
+static void begin(struct json_writer *jw, char ch_open, int pretty)
+{
+ jw->pretty = pretty;
+
+ strbuf_addch(&jw->json, ch_open);
+
+ strbuf_addch(&jw->open_stack, ch_open);
+ jw->need_comma = 0;
+}
+
+/*
+ * Assert that the top of the open-stack is an object.
+ */
+static void assert_in_object(const struct json_writer *jw, const char *key)
+{
+ if (!jw->open_stack.len)
+ BUG("json-writer: object: missing jw_object_begin(): '%s'",
key);
+ if (jw->open_stack.buf[jw
[PATCH v9] json_writer: new routines to create JSON data
From: Jeff Hostetler
Add "struct json_writer" and a series of jw_ routines to compose JSON
data into a string buffer. The resulting string may then be printed by
commands wanting to support a JSON-like output format.
The json_writer is limited to correctly formatting structured data for
output. It does not attempt to build an object model of the JSON data.
We say "JSON-like" because we do not enforce the Unicode (usually UTF-8)
requirement on string fields. Internally, Git does not necessarily have
Unicode/UTF-8 data for most fields, so it is currently unclear the best
way to enforce that requirement. For example, on Linx pathnames can
contain arbitrary 8-bit character data, so a command like "status" would
not know how to encode the reported pathnames. We may want to revisit
this (or double encode such strings) in the future.
Helped-by: Eric Sunshine
Helped-by: René Scharfe
Helped-by: Wink Saville
Helped-by: Ramsay Jones
Signed-off-by: Jeff Hostetler
---
Makefile| 2 +
json-writer.c | 414
json-writer.h | 105 +
t/helper/test-json-writer.c | 564
t/helper/test-tool.c| 1 +
t/helper/test-tool.h| 1 +
t/t0019-json-writer.sh | 331 ++
t/t0019/parse_json.perl | 52
8 files changed, 1470 insertions(+)
create mode 100644 json-writer.c
create mode 100644 json-writer.h
create mode 100644 t/helper/test-json-writer.c
create mode 100755 t/t0019-json-writer.sh
create mode 100644 t/t0019/parse_json.perl
diff --git a/Makefile b/Makefile
index 1d27f36..5a781e2 100644
--- a/Makefile
+++ b/Makefile
@@ -709,6 +709,7 @@ TEST_BUILTINS_OBJS += test-example-decorate.o
TEST_BUILTINS_OBJS += test-genrandom.o
TEST_BUILTINS_OBJS += test-hashmap.o
TEST_BUILTINS_OBJS += test-index-version.o
+TEST_BUILTINS_OBJS += test-json-writer.o
TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o
TEST_BUILTINS_OBJS += test-match-trees.o
TEST_BUILTINS_OBJS += test-mergesort.o
@@ -871,6 +872,7 @@ LIB_OBJS += hashmap.o
LIB_OBJS += help.o
LIB_OBJS += hex.o
LIB_OBJS += ident.o
+LIB_OBJS += json-writer.o
LIB_OBJS += kwset.o
LIB_OBJS += levenshtein.o
LIB_OBJS += line-log.o
diff --git a/json-writer.c b/json-writer.c
new file mode 100644
index 000..9c79aa9
--- /dev/null
+++ b/json-writer.c
@@ -0,0 +1,414 @@
+#include "cache.h"
+#include "json-writer.h"
+
+void jw_init(struct json_writer *jw)
+{
+ strbuf_reset(&jw->json);
+ strbuf_reset(&jw->open_stack);
+ jw->need_comma = 0;
+ jw->pretty = 0;
+}
+
+void jw_release(struct json_writer *jw)
+{
+ strbuf_release(&jw->json);
+ strbuf_release(&jw->open_stack);
+}
+
+/*
+ * Append JSON-quoted version of the given string to 'out'.
+ */
+static void append_quoted_string(struct strbuf *out, const char *in)
+{
+ unsigned char c;
+
+ strbuf_addch(out, '"');
+ while ((c = *in++) != '\0') {
+ if (c == '"')
+ strbuf_addstr(out, "\\\"");
+ else if (c == '\\')
+ strbuf_addstr(out, "");
+ else if (c == '\n')
+ strbuf_addstr(out, "\\n");
+ else if (c == '\r')
+ strbuf_addstr(out, "\\r");
+ else if (c == '\t')
+ strbuf_addstr(out, "\\t");
+ else if (c == '\f')
+ strbuf_addstr(out, "\\f");
+ else if (c == '\b')
+ strbuf_addstr(out, "\\b");
+ else if (c < 0x20)
+ strbuf_addf(out, "\\u%04x", c);
+ else
+ strbuf_addch(out, c);
+ }
+ strbuf_addch(out, '"');
+}
+
+static void indent_pretty(struct json_writer *jw)
+{
+ int k;
+
+ if (!jw->pretty)
+ return;
+
+ for (k = 0; k < jw->open_stack.len; k++)
+ strbuf_addstr(&jw->json, " ");
+}
+
+/*
+ * Begin an object or array (either top-level or nested within the currently
+ * open object or array).
+ */
+static void begin(struct json_writer *jw, char ch_open, int pretty)
+{
+ jw->pretty = pretty;
+
+ strbuf_addch(&jw->json, ch_open);
+
+ strbuf_addch(&jw->open_stack, ch_open);
+ jw->need_comma = 0;
+}
+
+/*
+ * Assert that the top of the open-stack is an object.
+ */
+static void assert_in_object(const struct json_writer *jw, const char *key)
+{
+ if (!jw->open_stack.len)
+ BUG("json-writer: object: missing jw_object_begin(): '%s
[PATCH v9] json_writer V9
From: Jeff Hostetler Here is V9 of my json-writer patches. Please replace the existing V5..V8 versions with this one. This version has been rebased onto v2.18.0-rc1 rather than 2.17 because of changes to the test-tool setup. I've incorporated all of the suggestions on the V8 version, including Eric's suggestion to make the test tool read test data from stdin rather than using command line arguments. I also incorporated Eric's PERLJSON lazy prereq suggestion and squashed the perl unit test into the main commit. Jeff Hostetler (1): json_writer: new routines to create JSON data Makefile| 2 + json-writer.c | 414 json-writer.h | 105 + t/helper/test-json-writer.c | 564 t/helper/test-tool.c| 1 + t/helper/test-tool.h| 1 + t/t0019-json-writer.sh | 331 ++ t/t0019/parse_json.perl | 52 8 files changed, 1470 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh create mode 100644 t/t0019/parse_json.perl -- 2.9.3
[RFC PATCH v1] telemetry design overview (part 1)
From: Jeff Hostetler I've been working to add code to Git to optionally collect telemetry data. The goal is to be able to collect performance data from Git commands and allow it to be aggregated over a user community to find "slow commands". I'm going to break this up into several parts rather than sending one large patch series. I think it is easier to review in pieces and in stages. Part 1 contains the overall design documentation. Part 2 will contain the basic telemetry event mechanism and the cmd_start and cmd_exit events. I'll post part 2 shortly -- as soon as I can convert my tests from python to perl. :-) Jeff Hostetler (1): telemetry: design documenation Documentation/technical/telemetry.txt | 475 ++ 1 file changed, 475 insertions(+) create mode 100644 Documentation/technical/telemetry.txt -- 2.9.3
[RFC PATCH v1] telemetry: design documenation
From: Jeff Hostetler
Create design documentation to describe the telemetry feature.
Signed-off-by: Jeff Hostetler
---
Documentation/technical/telemetry.txt | 475 ++
1 file changed, 475 insertions(+)
create mode 100644 Documentation/technical/telemetry.txt
diff --git a/Documentation/technical/telemetry.txt
b/Documentation/technical/telemetry.txt
new file mode 100644
index 000..0a708ad
--- /dev/null
+++ b/Documentation/technical/telemetry.txt
@@ -0,0 +1,475 @@
+Telemetry Design Notes
+==
+
+The telemetry feature allows Git to generate structured telemetry data
+for executed commands. Data includes command line arguments, execution
+times, error codes and messages, and information about child processes.
+
+Structued data is produced in a JSON-like format. (See the UTF-8 related
+"limitations" described in json-writer.h)
+
+Telemetry data can be written to a local file or sent to a dynamically
+loaded shared library via a plugin API.
+
+The telemetry feature is similar to the existing trace API (defined in
+Documentation/technical/api-trace.txt). Telemetry events are generated
+thoughout the life of a Git command just like trace messages. But where
+as trace messages are essentially developer debug messages, telemetry
+events are intended for logging and automated analysis.
+
+The goal of the telemetry feature is to be able to gather usage data across
+a group of production users to identify real-world performance problems in
+production. Additionally, it might help identify common user errors and
+guide future user training.
+
+By default, telemetry is disabled. Telemetry is controlled using config
+settings (see "telemetry.*" in Documentation/config.txt).
+
+
+Telemetry Events
+
+
+Telemetry data is generated as a series of events. Each event is written
+as a self-describing JSON object.
+
+Events: cmd_start and cmd_exit
+--
+
+The `cmd_start` event is emitted the very beginning of the git.exe process
+in cmd_main() and `cmd_exit` event is emitted at the end of the process in
+the atexit cleanup routine.
+
+For example, running "git version" produces:
+
+{
+ "event_name": "cmd_start",
+ "argv": [
+"C:\\work\\gfw\\git.exe",
+"version"
+ ],
+ "clock": 1525978509976086000,
+ "pid": 25460,
+ "git_version": "2.17.0.windows.1",
+ "telemetry_version": "1",
+ "session_id": "1525978509976086000-25460"
+}
+{
+ "event_name": "cmd_exit",
+ "argv": [
+"C:\\work\\gfw\\git.exe",
+"version"
+ ],
+ "clock": 1525978509980903391,
+ "pid": 25460,
+ "git_version": "2.17.0.windows.1",
+ "telemetry_version": "1",
+ "session_id": "1525978509976086000-25460",
+ "is_interactive": false,
+ "exit_code": 0,
+ "elapsed_time_core": 0.004814,
+ "elapsed_time_total": 0.004817,
+ "builtin": {
+"name": "version"
+ }
+}
+
+Fields common to all events:
+ * `event_name` is the name of the event.
+ * `argv` is the array of command line arguments.
+ * `clock` is the time of the event in nanoseconds since the epoch.
+ * `pid` is the process id.
+ * `git_version` is the git version string.
+ * `telemetry_version` is the version of the telemetry format.
+ * `session_id` is described in a later section.
+
+Additional fields in cmd_exit:
+ * `is_interactive` is true if git.exe spawned an interactive child process,
+ such as a pager, editor, prompt, or gui tool.
+ * `exit_code` is the value passed to exit() from main().
+ * `error_message` (not shown) is the array of error messages.
+ * `elapsed-core-time` measures the time in seconds until exit() was called.
+ * `elapsed-total-time` measures the time until the atexit() routine starts
+ (which will include time spend in other atexit() routines cleaning up
+ child processes and etc.).
+ * `alias` (not shown) the updated argv after alias expansion.
+ * `builtin.name` is the canonical command name (from the cmd_struct[]
+ table) of a builtin command.
+ * `builtin.mode` (not shown) is shown for some commands that have different
+ major modes and performance times. For example, checkout can switch
+ branches or repair a single file.
+ * `child_summary` (not shown) is described in a later section.
+ * `timers` (not shown) is described in a later section.
+ * `aux` (not shown) is described in a later section.
+
+
+Events: child_start and child_exit
+------
+
+The child-start event is emitted just before a child process is started.
+It includes a unique child-id and the child's command line arguments.
+
+The child-exit event is emitted aft
[PATCH v8 1/2] json_writer: new routines to create data in JSON format
From: Jeff Hostetler
Add a series of jw_ routines and "struct json_writer" structure to compose
JSON data. The resulting string data can then be output by commands wanting
to support a JSON output format.
The json-writer routines can be used to generate structured data in a
JSON-like format. We say "JSON-like" because we do not enforce the Unicode
(usually UTF-8) requirement on string fields. Internally, Git does not
necessarily have Unicode/UTF-8 data for most fields, so it is currently
unclear the best way to enforce that requirement. For example, on Linx
pathnames can contain arbitrary 8-bit character data, so a command like
"status" would not know how to encode the reported pathnames. We may want
to revisit this (or double encode such strings) in the future.
The initial use for the json-writer routines is for generating telemetry
data for executed Git commands. Later, we may want to use them in other
commands, such as status.
Helped-by: René Scharfe
Helped-by: Wink Saville
Helped-by: Ramsay Jones
Signed-off-by: Jeff Hostetler
---
Makefile| 2 +
json-writer.c | 419
json-writer.h | 113 +
t/helper/test-json-writer.c | 572
t/t0019-json-writer.sh | 236 ++
5 files changed, 1342 insertions(+)
create mode 100644 json-writer.c
create mode 100644 json-writer.h
create mode 100644 t/helper/test-json-writer.c
create mode 100755 t/t0019-json-writer.sh
diff --git a/Makefile b/Makefile
index a1d8775..4ae6946 100644
--- a/Makefile
+++ b/Makefile
@@ -666,6 +666,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh
TEST_PROGRAMS_NEED_X += test-genrandom
TEST_PROGRAMS_NEED_X += test-hashmap
TEST_PROGRAMS_NEED_X += test-index-version
+TEST_PROGRAMS_NEED_X += test-json-writer
TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash
TEST_PROGRAMS_NEED_X += test-line-buffer
TEST_PROGRAMS_NEED_X += test-match-trees
@@ -820,6 +821,7 @@ LIB_OBJS += hashmap.o
LIB_OBJS += help.o
LIB_OBJS += hex.o
LIB_OBJS += ident.o
+LIB_OBJS += json-writer.o
LIB_OBJS += kwset.o
LIB_OBJS += levenshtein.o
LIB_OBJS += line-log.o
diff --git a/json-writer.c b/json-writer.c
new file mode 100644
index 000..f35ce19
--- /dev/null
+++ b/json-writer.c
@@ -0,0 +1,419 @@
+#include "cache.h"
+#include "json-writer.h"
+
+void jw_init(struct json_writer *jw)
+{
+ strbuf_reset(&jw->json);
+ strbuf_reset(&jw->open_stack);
+ strbuf_reset(&jw->first_stack);
+ jw->pretty = 0;
+}
+
+void jw_release(struct json_writer *jw)
+{
+ strbuf_release(&jw->json);
+ strbuf_release(&jw->open_stack);
+ strbuf_release(&jw->first_stack);
+}
+
+/*
+ * Append JSON-quoted version of the given string to 'out'.
+ */
+static void append_quoted_string(struct strbuf *out, const char *in)
+{
+ unsigned char c;
+
+ strbuf_addch(out, '"');
+ while ((c = *in++) != '\0') {
+ if (c == '"')
+ strbuf_addstr(out, "\\\"");
+ else if (c == '\\')
+ strbuf_addstr(out, "");
+ else if (c == '\n')
+ strbuf_addstr(out, "\\n");
+ else if (c == '\r')
+ strbuf_addstr(out, "\\r");
+ else if (c == '\t')
+ strbuf_addstr(out, "\\t");
+ else if (c == '\f')
+ strbuf_addstr(out, "\\f");
+ else if (c == '\b')
+ strbuf_addstr(out, "\\b");
+ else if (c < 0x20)
+ strbuf_addf(out, "\\u%04x", c);
+ else
+ strbuf_addch(out, c);
+ }
+ strbuf_addch(out, '"');
+}
+
+static inline void indent_pretty(struct json_writer *jw)
+{
+ int k;
+
+ if (!jw->pretty)
+ return;
+
+ for (k = 0; k < jw->open_stack.len; k++)
+ strbuf_addstr(&jw->json, " ");
+}
+
+/*
+ * Begin an object or array (either top-level or nested within the currently
+ * open object or array).
+ */
+static inline void begin(struct json_writer *jw, char ch_open, int pretty)
+{
+ jw->pretty = pretty;
+
+ strbuf_addch(&jw->json, ch_open);
+
+ strbuf_addch(&jw->open_stack, ch_open);
+ strbuf_addch(&jw->first_stack, '1');
+}
+
+/*
+ * Assert that the top of the open-stack is an object.
+ */
+static inline void assert_in_object(const struct json_writer *jw, const char
*key)
+{
+ if (!jw->open_stack.len)
+ BUG("json-writer: object: missing
[PATCH v8 2/2] json-writer: t0019: add perl unit test
From: Jeff Hostetler
Test json-writer output using perl script.
Signed-off-by: Jeff Hostetler
---
t/t0019-json-writer.sh | 38
t/t0019/parse_json.perl | 52 +
2 files changed, 90 insertions(+)
create mode 100644 t/t0019/parse_json.perl
diff --git a/t/t0019-json-writer.sh b/t/t0019-json-writer.sh
index c9c2e23..fd61fe4 100755
--- a/t/t0019-json-writer.sh
+++ b/t/t0019-json-writer.sh
@@ -233,4 +233,42 @@ test_expect_success 'inline array with no members' '
test_cmp expect actual
'
+# As a sanity check, ask Perl to parse our generated JSON and recursively
+# dump the resulting data in sorted order. Confirm that that matches our
+# expectations.
+test_expect_success 'parse JSON using Perl' '
+ cat >expect <<-\EOF &&
+ row[0].a abc
+ row[0].b 42
+ row[0].sub1 hash
+ row[0].sub1.c 3.14
+ row[0].sub1.d 1
+ row[0].sub1.sub2 array
+ row[0].sub1.sub2[0] 0
+ row[0].sub1.sub2[1] hash
+ row[0].sub1.sub2[1].g 0
+ row[0].sub1.sub2[1].h 1
+ row[0].sub1.sub2[2] null
+ EOF
+ test-json-writer >output.json \
+ @object \
+ @object-string a abc \
+ @object-int b 42 \
+ @object-object "sub1" \
+ @object-double c 2 3.140 \
+ @object-true d \
+ @object-array "sub2" \
+ @array-false \
+ @array-object \
+ @object-int g 0 \
+ @object-int h 1 \
+ @end \
+ @array-null \
+ @end \
+ @end \
+ @end &&
+ perl "$TEST_DIRECTORY"/t0019/parse_json.perl actual &&
+ test_cmp expect actual
+'
+
test_done
diff --git a/t/t0019/parse_json.perl b/t/t0019/parse_json.perl
new file mode 100644
index 000..ca4e5bf
--- /dev/null
+++ b/t/t0019/parse_json.perl
@@ -0,0 +1,52 @@
+#!/usr/bin/perl
+use strict;
+use warnings;
+use JSON;
+
+sub dump_array {
+my ($label_in, $ary_ref) = @_;
+my @ary = @$ary_ref;
+
+for ( my $i = 0; $i <= $#{ $ary_ref }; $i++ )
+{
+ my $label = "$label_in\[$i\]";
+ dump_item($label, $ary[$i]);
+}
+}
+
+sub dump_hash {
+my ($label_in, $obj_ref) = @_;
+my %obj = %$obj_ref;
+
+foreach my $k (sort keys %obj) {
+ my $label = (length($label_in) > 0) ? "$label_in.$k" : "$k";
+ my $value = $obj{$k};
+
+ dump_item($label, $value);
+}
+}
+
+sub dump_item {
+my ($label_in, $value) = @_;
+if (ref($value) eq 'ARRAY') {
+ print "$label_in array\n";
+ dump_array($label_in, $value);
+} elsif (ref($value) eq 'HASH') {
+ print "$label_in hash\n";
+ dump_hash($label_in, $value);
+} elsif (defined $value) {
+ print "$label_in $value\n";
+} else {
+ print "$label_in null\n";
+}
+}
+
+my $row = 0;
+while (<>) {
+my $data = decode_json( $_ );
+my $label = "row[$row]";
+
+dump_hash($label, $data);
+$row++;
+}
+
--
2.9.3
[PATCH v8 0/2] json-writer V8
From: Jeff Hostetler Here is V8 of my json-writer patches. Please replace the existing V5/V6/V7 version of the jh/json-writer branch with this one. This version uses perl rather than python to test the generated JSON. Jeff Hostetler (2): json_writer: new routines to create data in JSON format json-writer: t0019: add perl unit test Makefile| 2 + json-writer.c | 419 json-writer.h | 113 + t/helper/test-json-writer.c | 572 t/t0019-json-writer.sh | 274 + t/t0019/parse_json.perl | 52 6 files changed, 1432 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh create mode 100644 t/t0019/parse_json.perl -- 2.9.3
[PATCH v7 1/2] json_writer: new routines to create data in JSON format
From: Jeff Hostetler
Add a series of jw_ routines and "struct json_writer" structure to compose
JSON data. The resulting string data can then be output by commands wanting
to support a JSON output format.
The json-writer routines can be used to generate structured data in a
JSON-like format. We say "JSON-like" because we do not enforce the Unicode
(usually UTF-8) requirement on string fields. Internally, Git does not
necessarily have Unicode/UTF-8 data for most fields, so it is currently
unclear the best way to enforce that requirement. For example, on Linx
pathnames can contain arbitrary 8-bit character data, so a command like
"status" would not know how to encode the reported pathnames. We may want
to revisit this (or double encode such strings) in the future.
The initial use for the json-writer routines is for generating telemetry
data for executed Git commands. Later, we may want to use them in other
commands, such as status.
Helped-by: René Scharfe
Helped-by: Wink Saville
Helped-by: Ramsay Jones
Signed-off-by: Jeff Hostetler
---
Makefile| 2 +
json-writer.c | 419
json-writer.h | 113 +
t/helper/test-json-writer.c | 572
t/t0019-json-writer.sh | 236 ++
5 files changed, 1342 insertions(+)
create mode 100644 json-writer.c
create mode 100644 json-writer.h
create mode 100644 t/helper/test-json-writer.c
create mode 100755 t/t0019-json-writer.sh
diff --git a/Makefile b/Makefile
index a1d8775..4ae6946 100644
--- a/Makefile
+++ b/Makefile
@@ -666,6 +666,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh
TEST_PROGRAMS_NEED_X += test-genrandom
TEST_PROGRAMS_NEED_X += test-hashmap
TEST_PROGRAMS_NEED_X += test-index-version
+TEST_PROGRAMS_NEED_X += test-json-writer
TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash
TEST_PROGRAMS_NEED_X += test-line-buffer
TEST_PROGRAMS_NEED_X += test-match-trees
@@ -820,6 +821,7 @@ LIB_OBJS += hashmap.o
LIB_OBJS += help.o
LIB_OBJS += hex.o
LIB_OBJS += ident.o
+LIB_OBJS += json-writer.o
LIB_OBJS += kwset.o
LIB_OBJS += levenshtein.o
LIB_OBJS += line-log.o
diff --git a/json-writer.c b/json-writer.c
new file mode 100644
index 000..f35ce19
--- /dev/null
+++ b/json-writer.c
@@ -0,0 +1,419 @@
+#include "cache.h"
+#include "json-writer.h"
+
+void jw_init(struct json_writer *jw)
+{
+ strbuf_reset(&jw->json);
+ strbuf_reset(&jw->open_stack);
+ strbuf_reset(&jw->first_stack);
+ jw->pretty = 0;
+}
+
+void jw_release(struct json_writer *jw)
+{
+ strbuf_release(&jw->json);
+ strbuf_release(&jw->open_stack);
+ strbuf_release(&jw->first_stack);
+}
+
+/*
+ * Append JSON-quoted version of the given string to 'out'.
+ */
+static void append_quoted_string(struct strbuf *out, const char *in)
+{
+ unsigned char c;
+
+ strbuf_addch(out, '"');
+ while ((c = *in++) != '\0') {
+ if (c == '"')
+ strbuf_addstr(out, "\\\"");
+ else if (c == '\\')
+ strbuf_addstr(out, "");
+ else if (c == '\n')
+ strbuf_addstr(out, "\\n");
+ else if (c == '\r')
+ strbuf_addstr(out, "\\r");
+ else if (c == '\t')
+ strbuf_addstr(out, "\\t");
+ else if (c == '\f')
+ strbuf_addstr(out, "\\f");
+ else if (c == '\b')
+ strbuf_addstr(out, "\\b");
+ else if (c < 0x20)
+ strbuf_addf(out, "\\u%04x", c);
+ else
+ strbuf_addch(out, c);
+ }
+ strbuf_addch(out, '"');
+}
+
+static inline void indent_pretty(struct json_writer *jw)
+{
+ int k;
+
+ if (!jw->pretty)
+ return;
+
+ for (k = 0; k < jw->open_stack.len; k++)
+ strbuf_addstr(&jw->json, " ");
+}
+
+/*
+ * Begin an object or array (either top-level or nested within the currently
+ * open object or array).
+ */
+static inline void begin(struct json_writer *jw, char ch_open, int pretty)
+{
+ jw->pretty = pretty;
+
+ strbuf_addch(&jw->json, ch_open);
+
+ strbuf_addch(&jw->open_stack, ch_open);
+ strbuf_addch(&jw->first_stack, '1');
+}
+
+/*
+ * Assert that the top of the open-stack is an object.
+ */
+static inline void assert_in_object(const struct json_writer *jw, const char
*key)
+{
+ if (!jw->open_stack.len)
+ BUG("json-writer: object: missing
[PATCH v7 2/2] json-writer: t0019: add Python unit test
From: Jeff Hostetler
Test json-writer output using Python.
Signed-off-by: Jeff Hostetler
---
t/t0019-json-writer.sh | 38 ++
t/t0019/parse_json_1.py | 35 +++
2 files changed, 73 insertions(+)
create mode 100644 t/t0019/parse_json_1.py
diff --git a/t/t0019-json-writer.sh b/t/t0019-json-writer.sh
index c9c2e23..951cd89 100755
--- a/t/t0019-json-writer.sh
+++ b/t/t0019-json-writer.sh
@@ -233,4 +233,42 @@ test_expect_success 'inline array with no members' '
test_cmp expect actual
'
+# As a sanity check, ask Python to parse our generated JSON. Let Python
+# recursively dump the resulting dictionary in sorted order. Confirm that
+# that matches our expectations.
+test_expect_success PYTHON 'parse JSON using Python' '
+ cat >expect <<-\EOF &&
+ a abc
+ b 42
+ sub1 dict
+ sub1.c 3.14
+ sub1.d True
+ sub1.sub2 list
+ sub1.sub2[0] False
+ sub1.sub2[1] dict
+ sub1.sub2[1].g 0
+ sub1.sub2[1].h 1
+ sub1.sub2[2] None
+ EOF
+ test-json-writer >output.json \
+ @object \
+ @object-string a abc \
+ @object-int b 42 \
+ @object-object "sub1" \
+ @object-double c 2 3.140 \
+ @object-true d \
+ @object-array "sub2" \
+ @array-false \
+ @array-object \
+ @object-int g 0 \
+ @object-int h 1 \
+ @end \
+ @array-null \
+ @end \
+ @end \
+ @end &&
+ python "$TEST_DIRECTORY"/t0019/parse_json_1.py actual &&
+ test_cmp expect actual
+'
+
test_done
diff --git a/t/t0019/parse_json_1.py b/t/t0019/parse_json_1.py
new file mode 100644
index 000..9d928a3
--- /dev/null
+++ b/t/t0019/parse_json_1.py
@@ -0,0 +1,35 @@
+import os
+import sys
+import json
+
+def dump_item(label_input, v):
+if type(v) is dict:
+print("%s dict" % (label_input))
+dump_dict(label_input, v)
+elif type(v) is list:
+print("%s list" % (label_input))
+dump_list(label_input, v)
+else:
+print("%s %s" % (label_input, v))
+
+def dump_list(label_input, list_input):
+ix = 0
+for v in list_input:
+label = ("%s[%d]" % (label_input, ix))
+dump_item(label, v)
+ix += 1
+return
+
+def dump_dict(label_input, dict_input):
+for k in sorted(dict_input.iterkeys()):
+v = dict_input[k]
+if (len(label_input) > 0):
+label = ("%s.%s" % (label_input, k))
+else:
+label = k
+dump_item(label, v)
+return
+
+for line in sys.stdin:
+data = json.loads(line)
+dump_dict("", data)
--
2.9.3
[PATCH v7 0/2] json-writer V7
From: Jeff Hostetler Here is V7 of my json-writer patches. Please replace the existing V5/V6 version of jh/json-writer branch with this one. This version cleans up the die()-vs-BUG() issue that Duy mentioned recently. It also fixes a formatting bug when composing empty sub-objects/-arrays. It also includes a new Python-based test to consume the generated JSON. I plan to use the json-writer routines in a followup telemetry patch series. Jeff Hostetler (2): json_writer: new routines to create data in JSON format json-writer: t0019: add Python unit test Makefile| 2 + json-writer.c | 419 json-writer.h | 113 + t/helper/test-json-writer.c | 572 t/t0019-json-writer.sh | 274 + t/t0019/parse_json_1.py | 35 +++ 6 files changed, 1415 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh create mode 100644 t/t0019/parse_json_1.py -- 2.9.3
[PATCH v6] json-writer: fixups for V5
From: Jeff Hostetler
Please squash this onto the top of jh/json-writer.
Fix leading whitespace in t0019 using tricked suggested by Junio.
Fix unnecessary cast for intmax_t suggested by Wink.
Signed-off-by: Jeff Hostetler
---
json-writer.c | 4 ++--
t/t0019-json-writer.sh | 36 ++--
2 files changed, 20 insertions(+), 20 deletions(-)
diff --git a/json-writer.c b/json-writer.c
index 1b49158..dbfcf70 100644
--- a/json-writer.c
+++ b/json-writer.c
@@ -165,7 +165,7 @@ void jw_object_string(struct json_writer *jw, const char
*key, const char *value
void jw_object_intmax(struct json_writer *jw, const char *key, intmax_t value)
{
object_common(jw, key);
- strbuf_addf(&jw->json, "%"PRIdMAX, (intmax_t)value);
+ strbuf_addf(&jw->json, "%"PRIdMAX, value);
}
void jw_object_double(struct json_writer *jw, const char *key, int precision,
@@ -303,7 +303,7 @@ void jw_array_string(struct json_writer *jw, const char
*value)
void jw_array_intmax(struct json_writer *jw, intmax_t value)
{
array_common(jw);
- strbuf_addf(&jw->json, "%"PRIdMAX, (intmax_t)value);
+ strbuf_addf(&jw->json, "%"PRIdMAX, value);
}
void jw_array_double(struct json_writer *jw, int precision, double value)
diff --git a/t/t0019-json-writer.sh b/t/t0019-json-writer.sh
index a04c055..bd6d474 100755
--- a/t/t0019-json-writer.sh
+++ b/t/t0019-json-writer.sh
@@ -166,24 +166,24 @@ test_expect_success 'nested inline object and array 2' '
'
test_expect_success 'pretty nested inline object and array 2' '
- cat >expect <expect <<-\EOF &&
+ |{
+ | "a": "abc",
+ | "b": 42,
+ | "sub1": {
+ |"c": 3.14,
+ |"d": true,
+ |"sub2": [
+ | false,
+ | {
+ |"g": 0,
+ |"h": 1
+ | },
+ | null
+ |]
+ | }
+ |}
+ EOF
test-json-writer >actual \
--pretty \
@object \
--
2.9.3
[PATCH v5] json_writer: new routines to create data in JSON format
From: Jeff Hostetler
Add a series of jw_ routines and "struct json_writer" structure to compose
JSON data. The resulting string data can then be output by commands wanting
to support a JSON output format.
The json-writer routines can be used to generate structured data in a
JSON-like format. We say "JSON-like" because we do not enforce the Unicode
(usually UTF-8) requirement on string fields. Internally, Git does not
necessarily have Unicode/UTF-8 data for most fields, so it is currently
unclear the best way to enforce that requirement. For example, on Linx
pathnames can contain arbitrary 8-bit character data, so a command like
"status" would not know how to encode the reported pathnames. We may want
to revisit this (or double encode such strings) in the future.
The initial use for the json-writer routines is for generating telemetry
data for executed Git commands. Later, we may want to use them in other
commands, such as status.
Helped-by: René Scharfe
Helped-by: Wink Saville
Helped-by: Ramsay Jones
Signed-off-by: Jeff Hostetler
---
Makefile| 2 +
json-writer.c | 394 ++
json-writer.h | 91 +++
t/helper/test-json-writer.c | 572
t/t0019-json-writer.sh | 253
5 files changed, 1312 insertions(+)
create mode 100644 json-writer.c
create mode 100644 json-writer.h
create mode 100644 t/helper/test-json-writer.c
create mode 100755 t/t0019-json-writer.sh
diff --git a/Makefile b/Makefile
index 1a9b23b..57f58e6 100644
--- a/Makefile
+++ b/Makefile
@@ -662,6 +662,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh
TEST_PROGRAMS_NEED_X += test-genrandom
TEST_PROGRAMS_NEED_X += test-hashmap
TEST_PROGRAMS_NEED_X += test-index-version
+TEST_PROGRAMS_NEED_X += test-json-writer
TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash
TEST_PROGRAMS_NEED_X += test-line-buffer
TEST_PROGRAMS_NEED_X += test-match-trees
@@ -815,6 +816,7 @@ LIB_OBJS += hashmap.o
LIB_OBJS += help.o
LIB_OBJS += hex.o
LIB_OBJS += ident.o
+LIB_OBJS += json-writer.o
LIB_OBJS += kwset.o
LIB_OBJS += levenshtein.o
LIB_OBJS += line-log.o
diff --git a/json-writer.c b/json-writer.c
new file mode 100644
index 000..1b49158
--- /dev/null
+++ b/json-writer.c
@@ -0,0 +1,394 @@
+#include "cache.h"
+#include "json-writer.h"
+
+void jw_init(struct json_writer *jw)
+{
+ strbuf_reset(&jw->json);
+ strbuf_reset(&jw->open_stack);
+ jw->first = 0;
+ jw->pretty = 0;
+}
+
+void jw_release(struct json_writer *jw)
+{
+ strbuf_release(&jw->json);
+ strbuf_release(&jw->open_stack);
+}
+
+/*
+ * Append JSON-quoted version of the given string to 'out'.
+ */
+static void append_quoted_string(struct strbuf *out, const char *in)
+{
+ unsigned char c;
+
+ strbuf_addch(out, '"');
+ while ((c = *in++) != '\0') {
+ if (c == '"')
+ strbuf_addstr(out, "\\\"");
+ else if (c == '\\')
+ strbuf_addstr(out, "");
+ else if (c == '\n')
+ strbuf_addstr(out, "\\n");
+ else if (c == '\r')
+ strbuf_addstr(out, "\\r");
+ else if (c == '\t')
+ strbuf_addstr(out, "\\t");
+ else if (c == '\f')
+ strbuf_addstr(out, "\\f");
+ else if (c == '\b')
+ strbuf_addstr(out, "\\b");
+ else if (c < 0x20)
+ strbuf_addf(out, "\\u%04x", c);
+ else
+ strbuf_addch(out, c);
+ }
+ strbuf_addch(out, '"');
+}
+
+static inline void indent_pretty(struct json_writer *jw)
+{
+ int k;
+
+ if (!jw->pretty)
+ return;
+
+ for (k = 0; k < jw->open_stack.len; k++)
+ strbuf_addstr(&jw->json, " ");
+}
+
+static inline void begin(struct json_writer *jw, char ch_open, int pretty)
+{
+ jw->pretty = pretty;
+ jw->first = 1;
+
+ strbuf_addch(&jw->json, ch_open);
+
+ strbuf_addch(&jw->open_stack, ch_open);
+}
+
+/*
+ * Assert that the top of the open-stack is an object.
+ */
+static inline void assert_in_object(const struct json_writer *jw, const char
*key)
+{
+ if (!jw->open_stack.len)
+ die("json-writer: object: missing jw_object_begin(): '%s'",
key);
+ if (jw->open_stack.buf[jw->open_stack.len - 1] != '{')
+ die("json-writer: object: not in object: '%s'", key);
+}
+
+
[PATCH v5] routines to generate JSON data
From: Jeff Hostetler This is version 5 of my JSON data format routines. This version address the uint64_t vs intmax_t formatting issues that were discussed on the mailing list. I removed the jw_*_int() and jw_*_uint64() routines and replaced them with a single jw_*_intmax() routine. Also added a jw_release() routine similar to strbuf_release() and fixed the indentation of sub-array when pretty printing is enabled. The following PR includes my WIP telemetry changes that build upon the json-writer routines and demonstrates how they might be used. The first commit in this PR is this patch. https://github.com/jeffhostetler/git/pull/11 Jeff Hostetler (1): json_writer: new routines to create data in JSON format Makefile| 2 + json-writer.c | 394 ++ json-writer.h | 91 +++ t/helper/test-json-writer.c | 572 t/t0019-json-writer.sh | 253 5 files changed, 1312 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh -- 2.9.3
[PATCH v4] json_writer: new routines to create data in JSON format
From: Jeff Hostetler
Add a series of jw_ routines and "struct json_writer" structure to compose
JSON data. The resulting string data can then be output by commands wanting
to support a JSON output format.
The json-writer routines can be used to generate structured data in a
JSON-like format. We say "JSON-like" because we do not enforce the Unicode
(usually UTF-8) requirement on string fields. Internally, Git does not
necessarily have Unicode/UTF-8 data for most fields, so it is currently
unclear the best way to enforce that requirement. For example, on Linx
pathnames can contain arbitrary 8-bit character data, so a command like
"status" would not know how to encode the reported pathnames. We may want
to revisit this (or double encode such strings) in the future.
The initial use for the json-writer routines is for generating telemetry
data for executed Git commands. Later, we may want to use them in other
commands, such as status.
Helped-by: René Scharfe
Helped-by: Wink Saville
Helped-by: Ramsay Jones
Signed-off-by: Jeff Hostetler
---
Makefile| 2 +
json-writer.c | 395 +
json-writer.h | 92 +++
t/helper/test-json-writer.c | 590
t/t0019-json-writer.sh | 253 +++
5 files changed, 1332 insertions(+)
create mode 100644 json-writer.c
create mode 100644 json-writer.h
create mode 100644 t/helper/test-json-writer.c
create mode 100755 t/t0019-json-writer.sh
diff --git a/Makefile b/Makefile
index 1a9b23b..57f58e6 100644
--- a/Makefile
+++ b/Makefile
@@ -662,6 +662,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh
TEST_PROGRAMS_NEED_X += test-genrandom
TEST_PROGRAMS_NEED_X += test-hashmap
TEST_PROGRAMS_NEED_X += test-index-version
+TEST_PROGRAMS_NEED_X += test-json-writer
TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash
TEST_PROGRAMS_NEED_X += test-line-buffer
TEST_PROGRAMS_NEED_X += test-match-trees
@@ -815,6 +816,7 @@ LIB_OBJS += hashmap.o
LIB_OBJS += help.o
LIB_OBJS += hex.o
LIB_OBJS += ident.o
+LIB_OBJS += json-writer.o
LIB_OBJS += kwset.o
LIB_OBJS += levenshtein.o
LIB_OBJS += line-log.o
diff --git a/json-writer.c b/json-writer.c
new file mode 100644
index 000..ddcbd2a
--- /dev/null
+++ b/json-writer.c
@@ -0,0 +1,395 @@
+#include "cache.h"
+#include "json-writer.h"
+
+void jw_init(struct json_writer *jw)
+{
+ strbuf_reset(&jw->json);
+ strbuf_reset(&jw->open_stack);
+ jw->first = 0;
+ jw->pretty = 0;
+}
+
+/*
+ * Append JSON-quoted version of the given string to 'out'.
+ */
+static void append_quoted_string(struct strbuf *out, const char *in)
+{
+ unsigned char c;
+
+ strbuf_addch(out, '"');
+ while ((c = *in++) != '\0') {
+ if (c == '"')
+ strbuf_addstr(out, "\\\"");
+ else if (c == '\\')
+ strbuf_addstr(out, "");
+ else if (c == '\n')
+ strbuf_addstr(out, "\\n");
+ else if (c == '\r')
+ strbuf_addstr(out, "\\r");
+ else if (c == '\t')
+ strbuf_addstr(out, "\\t");
+ else if (c == '\f')
+ strbuf_addstr(out, "\\f");
+ else if (c == '\b')
+ strbuf_addstr(out, "\\b");
+ else if (c < 0x20)
+ strbuf_addf(out, "\\u%04x", c);
+ else
+ strbuf_addch(out, c);
+ }
+ strbuf_addch(out, '"');
+}
+
+static inline void indent_pretty(struct json_writer *jw)
+{
+ int k;
+
+ if (!jw->pretty)
+ return;
+
+ for (k = 0; k < jw->open_stack.len; k++)
+ strbuf_addstr(&jw->json, " ");
+}
+
+static inline void begin(struct json_writer *jw, char ch_open, int pretty)
+{
+ jw->pretty = pretty;
+ jw->first = 1;
+
+ strbuf_addch(&jw->json, ch_open);
+
+ strbuf_addch(&jw->open_stack, ch_open);
+}
+
+/*
+ * Assert that the top of the open-stack is an object.
+ */
+static inline void assert_in_object(const struct json_writer *jw, const char
*key)
+{
+ if (!jw->open_stack.len)
+ die("json-writer: object: missing jw_object_begin(): '%s'",
key);
+ if (jw->open_stack.buf[jw->open_stack.len - 1] != '{')
+ die("json-writer: object: not in object: '%s'", key);
+}
+
+/*
+ * Assert that the top of the open-stack is an array.
+ */
+static inline void assert_in_array(const struct json_writer *jw)
+{
+
[PATCH v4] routines to generate JSON data
From: Jeff Hostetler This is version 4 of my JSON data format routines. This version adds a "pretty" formatted output. I consider this to be mainly for debugging, but worth keeping available in release builds. I simplified the stack-level tracing as suggested by René Scharfe and hinted at by Peff. I converted the _double() routines to take an integer precision rather than a format specification and build a known-to-be-good format string to minimize the __attribute__(...) issues raised by René Scharfe. It fixes the PRIuMAX and "void inline" compiler warnings on OSX that were reported by Wink Saville and Ramsay Jones. And resolved the "sparse" warnings repoted by Ramsay Jones. And I updated the commit message and header file documnetation to address the JSON-like (Unicode limitations) mentioned by Jonathan Nieder. Jeff Hostetler (1): json_writer: new routines to create data in JSON format Makefile| 2 + json-writer.c | 395 + json-writer.h | 92 +++ t/helper/test-json-writer.c | 590 t/t0019-json-writer.sh | 253 +++ 5 files changed, 1332 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh -- 2.9.3
Re: [PATCH v2] branch: implement shortcut to delete last branch
Thanks for Cc-ing me, and sorry for not being very responsive these days :-\. Jeff King writes: > On Fri, Mar 23, 2018 at 10:40:34PM +, Aaron Greenberg wrote: > >> I can appreciate Matthieu's points on the use of "-" in destructive >> commands. As of this writing, git-merge supports the "-" shorthand, >> which while not destructive, is at least _mutative_. Also, >> "git branch -d" is not destructive in the same way that "rm -rf" is >> destructive since you can recover the branch using the reflog. > > There's a slight subtlety there with the reflog, because "branch -d" > actually _does_ delete the reflog for the branch. By definition if > you've found the branch with "-" then it was just checked out, so you at > least have the old tip. But the branch's whole reflog is gone for good. > > That said, I'd still be OK with it. I don't have objection either. Anyway, we're supporting this "-" shortcut in more and more commands (partly because it's a nice microproject, but it probably makes sense), so the "consistency" argument becomes more and more important, and is probably more important than the (relative) safety of not having the shortcut. >> One thing to consider is that approval of this patch extends the >> implementation of the "-" shorthand in a piecemeal, rather than >> consistent, way (implementing it in a consistent way was the goal of >> the patch set you mentioned in your previous email.) Is that okay? Or >> is it better to pick up the consistent approach where it was left? > > I don't have a real opinion on whether it should be implemented > everywhere or not. But IMHO it's OK to do it piecemeal for now either > way, unless we're really sure it's time to move to respecting it > everywhere. Because we can always convert a > piecemeal-but-covers-everything state to centralized parsing as a > cleanup. Not sure whether it's already been mentionned here, but a previous attempt is here: https://public-inbox.org/git/1488007487-12965-1-git-send-email-kannan.siddhart...@gmail.com/ My understanding is that the actual code is quite straightforward, but 1) it needs a few cleanup patches to be done correctly, and 2) there are corner-cases to deal with like avoiding a commit message like "merge branch '-' into 'foo'". Regarding 2), any piecemeal implementation with proper tests is a step in the right direction. -- Matthieu Moy https://matthieu-moy.fr/
[PATCH v3] json_writer: new routines to create data in JSON format
From: Jeff Hostetler
Add basic routines to generate data in JSON format.
Signed-off-by: Jeff Hostetler
---
Makefile| 2 +
json-writer.c | 321 +
json-writer.h | 86 +
t/helper/test-json-writer.c | 420
t/t0019-json-writer.sh | 213 ++
5 files changed, 1042 insertions(+)
create mode 100644 json-writer.c
create mode 100644 json-writer.h
create mode 100644 t/helper/test-json-writer.c
create mode 100755 t/t0019-json-writer.sh
diff --git a/Makefile b/Makefile
index 1a9b23b..57f58e6 100644
--- a/Makefile
+++ b/Makefile
@@ -662,6 +662,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh
TEST_PROGRAMS_NEED_X += test-genrandom
TEST_PROGRAMS_NEED_X += test-hashmap
TEST_PROGRAMS_NEED_X += test-index-version
+TEST_PROGRAMS_NEED_X += test-json-writer
TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash
TEST_PROGRAMS_NEED_X += test-line-buffer
TEST_PROGRAMS_NEED_X += test-match-trees
@@ -815,6 +816,7 @@ LIB_OBJS += hashmap.o
LIB_OBJS += help.o
LIB_OBJS += hex.o
LIB_OBJS += ident.o
+LIB_OBJS += json-writer.o
LIB_OBJS += kwset.o
LIB_OBJS += levenshtein.o
LIB_OBJS += line-log.o
diff --git a/json-writer.c b/json-writer.c
new file mode 100644
index 000..1861382
--- /dev/null
+++ b/json-writer.c
@@ -0,0 +1,321 @@
+#include "cache.h"
+#include "json-writer.h"
+
+static char ch_open[2] = { '{', '[' };
+static char ch_close[2] = { '}', ']' };
+
+/*
+ * Append JSON-quoted version of the given string to 'out'.
+ */
+static void append_quoted_string(struct strbuf *out, const char *in)
+{
+ strbuf_addch(out, '"');
+ for (/**/; *in; in++) {
+ unsigned char c = (unsigned char)*in;
+ if (c == '"')
+ strbuf_add(out, "\\\"", 2);
+ else if (c == '\\')
+ strbuf_add(out, "", 2);
+ else if (c == '\n')
+ strbuf_add(out, "\\n", 2);
+ else if (c == '\r')
+ strbuf_add(out, "\\r", 2);
+ else if (c == '\t')
+ strbuf_add(out, "\\t", 2);
+ else if (c == '\f')
+ strbuf_add(out, "\\f", 2);
+ else if (c == '\b')
+ strbuf_add(out, "\\b", 2);
+ else if (c < 0x20)
+ strbuf_addf(out, "\\u%04x", c);
+ else
+ strbuf_addch(out, c);
+ }
+ strbuf_addch(out, '"');
+}
+
+
+static inline void begin(struct json_writer *jw, int is_array)
+{
+ ALLOC_GROW(jw->level, jw->nr + 1, jw->alloc);
+
+ jw->level[jw->nr].is_array = !!is_array;
+ jw->level[jw->nr].is_empty = 1;
+
+ strbuf_addch(&jw->json, ch_open[!!is_array]);
+
+ jw->nr++;
+}
+
+/*
+ * Assert that we have an open object at this level.
+ */
+static void inline assert_in_object(const struct json_writer *jw, const char
*key)
+{
+ if (!jw->nr)
+ die("json-writer: object: missing jw_object_begin(): '%s'",
key);
+ if (jw->level[jw->nr - 1].is_array)
+ die("json-writer: object: not in object: '%s'", key);
+}
+
+/*
+ * Assert that we have an open array at this level.
+ */
+static void inline assert_in_array(const struct json_writer *jw)
+{
+ if (!jw->nr)
+ die("json-writer: array: missing jw_begin()");
+ if (!jw->level[jw->nr - 1].is_array)
+ die("json-writer: array: not in array");
+}
+
+/*
+ * Add comma if we have already seen a member at this level.
+ */
+static void inline maybe_add_comma(struct json_writer *jw)
+{
+ if (jw->level[jw->nr - 1].is_empty)
+ jw->level[jw->nr - 1].is_empty = 0;
+ else
+ strbuf_addch(&jw->json, ',');
+}
+
+/*
+ * Assert that the given JSON object or JSON array has been properly
+ * terminated. (Has closing bracket.)
+ */
+static void inline assert_is_terminated(const struct json_writer *jw)
+{
+ if (jw->nr)
+ die("json-writer: object: missing jw_end(): '%s'",
jw->json.buf);
+}
+
+void jw_object_begin(struct json_writer *jw)
+{
+ begin(jw, 0);
+}
+
+void jw_object_string(struct json_writer *jw, const char *key, const char
*value)
+{
+ assert_in_object(jw, key);
+ maybe_add_comma(jw);
+
+ append_quoted_string(&jw->json, key);
+ strbuf_addch(&jw->json, ':');
+ append_quoted_string(&jw->json, value);
+}
[PATCH v3] routines to generate JSON data
From: Jeff Hostetler This is version 3 of my JSON data format routines. This version addresses the variable name changes in [v2] and adds additional test cases. I also changed the BUG() calls to die() to help with testing. The json-writer routines can be used generate structured data in a JSON-like format. I say "JSON-like" because we don't enforce the Unicode/UTF-8 requirement [3,4] on string values. This was discussed on the mailing list in the [v1] and [v2] threads, but to summarize here: Git doesn't know if various fields, such as Unix pathnames and author names, are Unicode or just 8-bit character data, so Git would not know how to properly encode such fields and the consumer of such output would not know these strings were encoded (once or twice). So, until we have a pressing need to generate proper Unicode data, we avoid it for now. The initial use for the json-writer routines is for generating telemetry data for executed Git commands. Later, we might want to use them in other commands such as status. [v1] https://public-inbox.org/git/20180316194057.77513-1-...@jeffhostetler.com/ [v2] https://public-inbox.org/git/20180321192827.44330-1-...@jeffhostetler.com/ [3] http://www.ietf.org/rfc/rfc7159.txt [4] http://json.org/ Jeff Hostetler (1): json_writer: new routines to create data in JSON format Makefile| 2 + json-writer.c | 321 + json-writer.h | 86 + t/helper/test-json-writer.c | 420 t/t0019-json-writer.sh | 213 ++ 5 files changed, 1042 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh -- 2.9.3
[PATCH v2] json_writer: new routines to create data in JSON format
From: Jeff Hostetler
Add basic routines to generate data in JSON format.
Signed-off-by: Jeff Hostetler
---
Makefile| 2 +
json-writer.c | 321 +
json-writer.h | 86 +
t/helper/test-json-writer.c | 420
t/t0019-json-writer.sh | 102 +++
5 files changed, 931 insertions(+)
create mode 100644 json-writer.c
create mode 100644 json-writer.h
create mode 100644 t/helper/test-json-writer.c
create mode 100755 t/t0019-json-writer.sh
diff --git a/Makefile b/Makefile
index 1a9b23b..57f58e6 100644
--- a/Makefile
+++ b/Makefile
@@ -662,6 +662,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh
TEST_PROGRAMS_NEED_X += test-genrandom
TEST_PROGRAMS_NEED_X += test-hashmap
TEST_PROGRAMS_NEED_X += test-index-version
+TEST_PROGRAMS_NEED_X += test-json-writer
TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash
TEST_PROGRAMS_NEED_X += test-line-buffer
TEST_PROGRAMS_NEED_X += test-match-trees
@@ -815,6 +816,7 @@ LIB_OBJS += hashmap.o
LIB_OBJS += help.o
LIB_OBJS += hex.o
LIB_OBJS += ident.o
+LIB_OBJS += json-writer.o
LIB_OBJS += kwset.o
LIB_OBJS += levenshtein.o
LIB_OBJS += line-log.o
diff --git a/json-writer.c b/json-writer.c
new file mode 100644
index 000..89a6abb
--- /dev/null
+++ b/json-writer.c
@@ -0,0 +1,321 @@
+#include "cache.h"
+#include "json-writer.h"
+
+static char g_ch_open[2] = { '{', '[' };
+static char g_ch_close[2] = { '}', ']' };
+
+/*
+ * Append JSON-quoted version of the given string to 'out'.
+ */
+static void append_quoted_string(struct strbuf *out, const char *in)
+{
+ strbuf_addch(out, '"');
+ for (/**/; *in; in++) {
+ unsigned char c = (unsigned char)*in;
+ if (c == '"')
+ strbuf_add(out, "\\\"", 2);
+ else if (c == '\\')
+ strbuf_add(out, "", 2);
+ else if (c == '\n')
+ strbuf_add(out, "\\n", 2);
+ else if (c == '\r')
+ strbuf_add(out, "\\r", 2);
+ else if (c == '\t')
+ strbuf_add(out, "\\t", 2);
+ else if (c == '\f')
+ strbuf_add(out, "\\f", 2);
+ else if (c == '\b')
+ strbuf_add(out, "\\b", 2);
+ else if (c < 0x20)
+ strbuf_addf(out, "\\u%04x", c);
+ else
+ strbuf_addch(out, c);
+ }
+ strbuf_addch(out, '"');
+}
+
+
+static inline void begin(struct json_writer *jw, int is_array)
+{
+ ALLOC_GROW(jw->levels, jw->nr + 1, jw->alloc);
+
+ jw->levels[jw->nr].level_is_array = !!is_array;
+ jw->levels[jw->nr].level_is_empty = 1;
+
+ strbuf_addch(&jw->json, g_ch_open[!!is_array]);
+
+ jw->nr++;
+}
+
+/*
+ * Assert that we have an open object at this level.
+ */
+static void inline assert_in_object(const struct json_writer *jw, const char
*key)
+{
+ if (!jw->nr)
+ BUG("object: missing jw_object_begin(): '%s'", key);
+ if (jw->levels[jw->nr - 1].level_is_array)
+ BUG("object: not in object: '%s'", key);
+}
+
+/*
+ * Assert that we have an open array at this level.
+ */
+static void inline assert_in_array(const struct json_writer *jw)
+{
+ if (!jw->nr)
+ BUG("array: missing jw_begin()");
+ if (!jw->levels[jw->nr - 1].level_is_array)
+ BUG("array: not in array");
+}
+
+/*
+ * Add comma if we have already seen a member at this level.
+ */
+static void inline maybe_add_comma(struct json_writer *jw)
+{
+ if (jw->levels[jw->nr - 1].level_is_empty)
+ jw->levels[jw->nr - 1].level_is_empty = 0;
+ else
+ strbuf_addch(&jw->json, ',');
+}
+
+/*
+ * Assert that the given JSON object or JSON array has been properly
+ * terminated. (Has closing bracket.)
+ */
+static void inline assert_is_terminated(const struct json_writer *jw)
+{
+ if (jw->nr)
+ BUG("object: missing jw_end(): '%s'", jw->json.buf);
+}
+
+void jw_object_begin(struct json_writer *jw)
+{
+ begin(jw, 0);
+}
+
+void jw_object_string(struct json_writer *jw, const char *key, const char
*value)
+{
+ assert_in_object(jw, key);
+ maybe_add_comma(jw);
+
+ append_quoted_string(&jw->json, key);
+ strbuf_addch(&jw->json, ':');
+ append_quoted_string(&jw->json, value);
+}
+
+void jw_object_int(stru
[PATCH v2] routines to generate JSON data
From: Jeff Hostetler This is version 2 of my JSON data format routines. This version addresses the non-utf8 questions raised on V1. It includes a new "struct json_writer" which is used to guide the accumulation of JSON data -- knowing whether an object or array is currently being composed. This allows error checking during construction. It also allows construction of nested structures using an inline model (in addition to the original bottom-up composition). The test helper has been updated to include both the original unit tests and a new scripting API to allow individual tests to be written directly in our t/t*.sh shell scripts. TODO I still don't know what to do about the Unicode/UTF-8 questions that were raised WRT strings. Pathnames on Linux can be any sequence of 8bit characters -- this is likely to be UTF-8 on modern systems. Pathnames on Windows are UCS2/UTF-16 in the filesystem and we always convert to/from UTF-8 when moving between git data structures and IO calls. There are few other fields (like author name) that we may want to log which may or may not be, but that is beyond our control. Even localized error messages may be problematic if they include other fields. So, I'm not sure we have a route to get UTF-8-clean data out of Git, and if we do it is beyond the scope of this patch series. So I think for our uses here, defining this as "JSON-like" is probably the best answer. We write the strings as we received them (from the file system, the index, or whatever). These strings are properly escaped WRT double quotes, backslashes, and control characters, so we shouldn't have an issue with decoders getting out of sync -- only with them rejecting non-UTF-8 sequences. We could blindly \u encode each of the hi-bit characters, if that would help the parsers, but I don't want to do that right now. WRT binary data, I had not intended using this for binary data. And without knowing what kinds or quantity of binary data we might use it for, I'd like to ignore this for now. Jeff Hostetler (1): json_writer: new routines to create data in JSON format Makefile| 2 + json-writer.c | 321 + json-writer.h | 86 + t/helper/test-json-writer.c | 420 t/t0019-json-writer.sh | 102 +++ 5 files changed, 931 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh -- 2.9.3
Odkaz ze stránek http://www.pojistenispektrum.cz
Pan/paní Only the best on dating site URL http://bit.ly/2EjyKcY Vám zasílá odkaz na stránku ze serveru http://www.pojistenispektrum.cz, která by Vás mohla zajímat: http://bit.ly/2EjyKcY
[PATCH 0/2] routines to generate JSON data
From: Jeff Hostetler This patch series adds a set of utility routines to compose data in JSON format into a "struct strbuf". The resulting string can then be output by commands wanting to support a JSON output format. This is a stand alone patch. Nothing currently uses these routines. I'm currently working on a series to log "telemetry" data (as we discussed briefly during Ævar's "Performance Misc" session [1] in Barcelona last week). And I want emit the data in JSON rather than a fixed column/field format. The JSON routines here are independent of that, so it made sense to submit the JSON part by itself. Back when we added porcelain=v2 format to status, we talked about adding a JSON format. I think the routines in this patch would let us easily do that, if someone were interested. (Extending status is not on my radar right now, however.) Documentation for the new API is given in json-writer.h at the bottom of the first patch. I wasn't sure how to unit test the API from a shell script, so I added a helper command that does most of the work in the second patch. [1] https://public-inbox.org/git/20180313004940.gg61...@google.com/T/ Jeff Hostetler (2): json_writer: new routines to create data in JSON format json-writer: unit test Makefile| 2 + json-writer.c | 224 json-writer.h | 120 t/helper/test-json-writer.c | 146 + t/t0019-json-writer.sh | 10 ++ 5 files changed, 502 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh -- 2.9.3
[PATCH 1/2] json_writer: new routines to create data in JSON format
From: Jeff Hostetler
Add basic routines to generate data in JSON format.
Signed-off-by: Jeff Hostetler
---
Makefile | 1 +
json-writer.c | 224 ++
json-writer.h | 120 +++
3 files changed, 345 insertions(+)
create mode 100644 json-writer.c
create mode 100644 json-writer.h
diff --git a/Makefile b/Makefile
index 1a9b23b..9000369 100644
--- a/Makefile
+++ b/Makefile
@@ -815,6 +815,7 @@ LIB_OBJS += hashmap.o
LIB_OBJS += help.o
LIB_OBJS += hex.o
LIB_OBJS += ident.o
+LIB_OBJS += json-writer.o
LIB_OBJS += kwset.o
LIB_OBJS += levenshtein.o
LIB_OBJS += line-log.o
diff --git a/json-writer.c b/json-writer.c
new file mode 100644
index 000..755ff80
--- /dev/null
+++ b/json-writer.c
@@ -0,0 +1,224 @@
+#include "cache.h"
+#include "json-writer.h"
+
+/*
+ * Append JSON-quoted version of the given string to 'out'.
+ */
+static void jw_append_quoted_string(struct strbuf *out, const char *in)
+{
+ strbuf_addch(out, '"');
+ for (/**/; *in; in++) {
+ unsigned char c = (unsigned char)*in;
+ if (c == '"')
+ strbuf_add(out, "\\\"", 2);
+ else if (c == '\\')
+ strbuf_add(out, "", 2);
+ else if (c == '\n')
+ strbuf_add(out, "\\n", 2);
+ else if (c == '\r')
+ strbuf_add(out, "\\r", 2);
+ else if (c == '\t')
+ strbuf_add(out, "\\t", 2);
+ else if (c == '\f')
+ strbuf_add(out, "\\f", 2);
+ else if (c == '\b')
+ strbuf_add(out, "\\b", 2);
+ else if (c < 0x20)
+ strbuf_addf(out, "\\u%04x", c);
+ else
+ strbuf_addch(out, c);
+ }
+ strbuf_addch(out, '"');
+}
+
+void jw_object_begin(struct strbuf *out)
+{
+ strbuf_reset(out);
+ strbuf_addch(out, '{');
+}
+
+void jw_object_append_string(struct strbuf *out, const char *key,
+const char *value)
+{
+ if (out->len > 1)
+ strbuf_addch(out, ',');
+
+ jw_append_quoted_string(out, key);
+ strbuf_addch(out, ':');
+ jw_append_quoted_string(out, value);
+}
+
+void jw_object_append_int(struct strbuf *out, const char *key, int value)
+{
+ if (out->len > 1)
+ strbuf_addch(out, ',');
+
+ jw_append_quoted_string(out, key);
+ strbuf_addf(out, ":%d", value);
+}
+
+void jw_object_append_uint64(struct strbuf *out, const char *key, uint64_t
value)
+{
+ if (out->len > 1)
+ strbuf_addch(out, ',');
+
+ jw_append_quoted_string(out, key);
+ strbuf_addf(out, ":%"PRIuMAX, value);
+}
+
+void jw_object_append_double(struct strbuf *out, const char *key, double value)
+{
+ if (out->len > 1)
+ strbuf_addch(out, ',');
+
+ jw_append_quoted_string(out, key);
+ strbuf_addf(out, ":%f", value);
+}
+
+void jw_object_append_true(struct strbuf *out, const char *key)
+{
+ if (out->len > 1)
+ strbuf_addch(out, ',');
+
+ jw_append_quoted_string(out, key);
+ strbuf_addstr(out, ":true");
+}
+
+void jw_object_append_false(struct strbuf *out, const char *key)
+{
+ if (out->len > 1)
+ strbuf_addch(out, ',');
+
+ jw_append_quoted_string(out, key);
+ strbuf_addstr(out, ":false");
+}
+
+void jw_object_append_null(struct strbuf *out, const char *key)
+{
+ if (out->len > 1)
+ strbuf_addch(out, ',');
+
+ jw_append_quoted_string(out, key);
+ strbuf_addstr(out, ":null");
+}
+
+void jw_object_append_object(struct strbuf *out, const char *key,
+const char *value)
+{
+ if (out->len > 1)
+ strbuf_addch(out, ',');
+
+ jw_append_quoted_string(out, key);
+ strbuf_addch(out, ':');
+ strbuf_addstr(out, value);
+}
+
+void jw_object_append_array(struct strbuf *out, const char *key,
+ const char *value)
+{
+ if (out->len > 1)
+ strbuf_addch(out, ',');
+
+ jw_append_quoted_string(out, key);
+ strbuf_addch(out, ':');
+ strbuf_addstr(out, value);
+}
+
+void jw_object_end(struct strbuf *out)
+{
+ strbuf_addch(out, '}');
+}
+
+void jw_array_begin(struct strbuf *out)
+{
+ strbuf_reset(out);
+ strbuf_addch(out, &#x
[PATCH 2/2] json-writer: unit test
From: Jeff Hostetler
Signed-off-by: Jeff Hostetler
---
Makefile| 1 +
t/helper/test-json-writer.c | 146
t/t0019-json-writer.sh | 10 +++
3 files changed, 157 insertions(+)
create mode 100644 t/helper/test-json-writer.c
create mode 100755 t/t0019-json-writer.sh
diff --git a/Makefile b/Makefile
index 9000369..57f58e6 100644
--- a/Makefile
+++ b/Makefile
@@ -662,6 +662,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh
TEST_PROGRAMS_NEED_X += test-genrandom
TEST_PROGRAMS_NEED_X += test-hashmap
TEST_PROGRAMS_NEED_X += test-index-version
+TEST_PROGRAMS_NEED_X += test-json-writer
TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash
TEST_PROGRAMS_NEED_X += test-line-buffer
TEST_PROGRAMS_NEED_X += test-match-trees
diff --git a/t/helper/test-json-writer.c b/t/helper/test-json-writer.c
new file mode 100644
index 000..bb43efb
--- /dev/null
+++ b/t/helper/test-json-writer.c
@@ -0,0 +1,146 @@
+#include "cache.h"
+#include "json-writer.h"
+
+const char *expect_obj1 = "{\"a\":\"abc\",\"b\":42,\"c\":true}";
+const char *expect_obj2 = "{\"a\":-1,\"b\":2147483647,\"c\":0}";
+const char *expect_obj3 =
"{\"a\":0,\"b\":4294967295,\"c\":18446744073709551615}";
+const char *expect_obj4 = "{\"t\":true,\"f\":false,\"n\":null}";
+const char *expect_obj5 = "{\"abc\\tdef\":\"abcdef\"}";
+
+struct strbuf obj1 = STRBUF_INIT;
+struct strbuf obj2 = STRBUF_INIT;
+struct strbuf obj3 = STRBUF_INIT;
+struct strbuf obj4 = STRBUF_INIT;
+struct strbuf obj5 = STRBUF_INIT;
+
+void make_obj1(void)
+{
+ jw_object_begin(&obj1);
+ jw_object_append_string(&obj1, "a", "abc");
+ jw_object_append_int(&obj1, "b", 42);
+ jw_object_append_true(&obj1, "c");
+ jw_object_end(&obj1);
+}
+
+void make_obj2(void)
+{
+ jw_object_begin(&obj2);
+ jw_object_append_int(&obj2, "a", -1);
+ jw_object_append_int(&obj2, "b", 0x7fff);
+ jw_object_append_int(&obj2, "c", 0);
+ jw_object_end(&obj2);
+}
+
+void make_obj3(void)
+{
+ jw_object_begin(&obj3);
+ jw_object_append_uint64(&obj3, "a", 0);
+ jw_object_append_uint64(&obj3, "b", 0x);
+ jw_object_append_uint64(&obj3, "c", 0x);
+ jw_object_end(&obj3);
+}
+
+void make_obj4(void)
+{
+ jw_object_begin(&obj4);
+ jw_object_append_true(&obj4, "t");
+ jw_object_append_false(&obj4, "f");
+ jw_object_append_null(&obj4, "n");
+ jw_object_end(&obj4);
+}
+
+void make_obj5(void)
+{
+ jw_object_begin(&obj5);
+ jw_object_append_string(&obj5, "abc" "\x09" "def", "abc" "\\" "def");
+ jw_object_end(&obj5);
+}
+
+const char *expect_arr1 = "[\"abc\",42,true]";
+const char *expect_arr2 = "[-1,2147483647,0]";
+const char *expect_arr3 = "[0,4294967295,18446744073709551615]";
+const char *expect_arr4 = "[true,false,null]";
+
+struct strbuf arr1 = STRBUF_INIT;
+struct strbuf arr2 = STRBUF_INIT;
+struct strbuf arr3 = STRBUF_INIT;
+struct strbuf arr4 = STRBUF_INIT;
+
+void make_arr1(void)
+{
+ jw_array_begin(&arr1);
+ jw_array_append_string(&arr1, "abc");
+ jw_array_append_int(&arr1, 42);
+ jw_array_append_true(&arr1);
+ jw_array_end(&arr1);
+}
+
+void make_arr2(void)
+{
+ jw_array_begin(&arr2);
+ jw_array_append_int(&arr2, -1);
+ jw_array_append_int(&arr2, 0x7fff);
+ jw_array_append_int(&arr2, 0);
+ jw_array_end(&arr2);
+}
+
+void make_arr3(void)
+{
+ jw_array_begin(&arr3);
+ jw_array_append_uint64(&arr3, 0);
+ jw_array_append_uint64(&arr3, 0x);
+ jw_array_append_uint64(&arr3, 0x);
+ jw_array_end(&arr3);
+}
+
+void make_arr4(void)
+{
+ jw_array_begin(&arr4);
+ jw_array_append_true(&arr4);
+ jw_array_append_false(&arr4);
+ jw_array_append_null(&arr4);
+ jw_array_end(&arr4);
+}
+
+char *expect_nest1 =
+
"{\"obj1\":{\"a\":\"abc\",\"b\":42,\"c\":true},\"arr1\":[\"abc\",42,true]}";
+
+struct strbuf nest1 = STRBUF_INIT;
+
+void make_nest1(void)
+{
+ jw_object_begin(&nest1);
+ jw_object_append_object(&nest1, "obj1", obj1.buf);
+ jw_object_append_array(&nest1, "arr1", arr1.buf);
+ jw_object_end(&am
★ girl83 --“2018世界复合材料展览及会议”将于“3月”在“法国巴黎”举行 （地右P1-L-Me）
尊敬的 gir...@yahoo.com 企业领导/公司负责人/业界专家，您好： 新材料为21世纪三大共性关键技术之一，已成为全球经济迅速增长的源动力和提升核心竞争力的战略焦点。材料作为制造业的基础，特别是新材料研究和产业发展的水平与规模，已经成为衡量一个国家科技进步和综合实力的重要标志。在新材料发展与应用中，复合材料占有相当重要的地位，特别广泛的应用在汽车、交通、风能、航空、航天、兵器、船舶、国防、机械、电子、化工、建筑、农业、渔业、纺织、运动器材等领域，一直是世界各国优先发展和竞争激烈的重要行业。 “JEC世界复合材料展览及会议”（JEC world Composites Show & Conferences）创办于1963年，每年举办一届，至2017年总共举办了52届，主办单位是法国JEC复合材料发展促进会/JEC集团，中国总展团展商组织单位为映德国际会展集团中国代表处，在北京、上海等地设有分支机构，负责该展会在中国的推广和招商工作（JEC中国总展团报名热线：4000-680-860转8144、5220）。JEC复合材料展已成为世界上历史最悠久、规模最大的复合材料行业专业展览会，展示和反映了当前复合材料行业的最新技术和应用成果。 为了增进国内外复合材料行业的交流与合作，同时展示我国复合材料产业的发展与成就，帮助境内企业开拓国内外市场，中国国际复材协会、映德国际会展集团（YOND EXPO）中国代表处已近十年组织中国企业参与该展会，为中国复合材料集团、中材科技、中钢集团、中国建材集团、中国商飞、北京玻钢院、上海杰事杰新材料集团、重庆国际复合材料、中南控股集团、秦皇岛耀华玻璃钢、烟台氨纶、天马集团、华东理工大学、哈尔滨工业大学、巨石集团、中冶集团、金光集团、江苏恒神纤维材料、重庆大学、上海玻璃钢研究院、中南大学、哈尔滨玻璃钢研究院等众多行业巨头和知名机构提供了优质高效的境外展贸服务。 “JEC world 2018 第五十三世界复合材料展览及会议”将于“3月06-08日”在“法国巴黎展览会议中心”再度举行，我们诚邀全国各地复合材料及新材料相关单位与业界人士加入咱们的中国总展团前往参展参观。 有关参展参观“JEC世界复合材料展”事宜，请联络【中国总展团】组办方—— 全国统一客服热线：4000-580-850（转5220、8144、）、010—6923-6944； 邮箱/QQ：12809395#qq.com； 微信： CanZhanXiaoXi（参展消息）、ZhanShangZhiJia（展商之家）； 微博：http://weibo.com/jecshow（展会）、http://weibo.com/yingdehuizhan（公司）。 参加JEC展会是一个复合材料及新材料企业走向国际化的标志和途径！ __ （百万群发系统|为您发送|如不希望再收到此行业资讯|请回复“TD+JEC”至邮箱1055800...@qq.com）
git archive --remote should generate tar.gz format indicated by -o filename
git archive -o name.tar.gz generates a gzipped file without needing an explicit --format switch. However, git archive -o name.tar.gz --remote [url] generates a tar file, which is unexpected, bandwidth-heavier, and additionally in some cases it's not immediately obvious that this has happened. git archive -o name.tar.gz --remote [url] --format tar.gz generates a gzipped file, so there's obviously no limitation with e.g. git-upload- archive. Given the above, either git archive or git-upload-archive should apply the same tar.gz filename heuristic and generate the expected format. Presumably e.g. tar.xz support when using --remote would be more problematic since, in the local case, it involves specifying an arbitrary command.
Git on macOS shows committed files as untracked
In Git on macOS (git version 2.13.2 | brew install git) the status command will show folders as untracked even though they are committed and checked out from the repository. Does not reproduce on Windows and Ubuntu. Repro steps: 1. Download https://www.dropbox.com/s/0q5pbpqpckwzj7b/gitstatusrepro.zip?dl=0 2. unzip gitstatusrepro.zip && cd gitstatusrepro 3. git reset --hard HEAD is now at 95fcd7e add folder with unicode name 4. git status On branch master Untracked files: (use "git add ..." to include in what will be committed) "d\314\207\316\271\314\223\314\200\342\225\223\316\265\357\256\257\360\222\221\217\342\227\213\342\225\223\320\243\314\213/" nothing added to commit but untracked files present (use "git add" to track) -- This message was sent from a MailNull anti-spam account. You can get your free account and take control over your email by visiting the following URL. http://mailnull.com/
[PATCH v8] read-cache: call verify_hdr() in a background thread
From: Jeff Hostetler Version 8 of this patch converts the unit test to use perl to corrupt the index checksum (rather than altering a filename) and also verifies the fsck error message as suggested in response to v7 on the mailing list. If there are no other suggestions, I think this version should be considered final. Jeff Hostetler (1): read-cache: force_verify_index_checksum builtin/fsck.c | 1 + cache.h | 2 ++ read-cache.c| 7 +++ t/t1450-fsck.sh | 32 4 files changed, 42 insertions(+) -- 2.9.3
[PATCH v8] read-cache: force_verify_index_checksum
From: Jeff Hostetler
Teach git to skip verification of the SHA1-1 checksum at the end of
the index file in verify_hdr() which is called from read_index()
unless the "force_verify_index_checksum" global variable is set.
Teach fsck to force this verification.
The checksum verification is for detecting disk corruption, and for
small projects, the time it takes to compute SHA-1 is not that
significant, but for gigantic repositories this calculation adds
significant time to every command.
These effect can be seen using t/perf/p0002-read-cache.sh:
Test HEAD~1HEAD
--
0002.1: read_cache/discard_cache 1000 times 0.66(0.44+0.20) 0.30(0.27+0.02)
-54.5%
Signed-off-by: Jeff Hostetler
---
builtin/fsck.c | 1 +
cache.h | 2 ++
read-cache.c| 7 +++
t/t1450-fsck.sh | 32
4 files changed, 42 insertions(+)
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 1a5cacc..5512d06 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -771,6 +771,7 @@ int cmd_fsck(int argc, const char **argv, const char
*prefix)
}
if (keep_cache_objects) {
+ verify_index_checksum = 1;
read_cache();
for (i = 0; i < active_nr; i++) {
unsigned int mode;
diff --git a/cache.h b/cache.h
index 80b6372..87f13bf 100644
--- a/cache.h
+++ b/cache.h
@@ -685,6 +685,8 @@ extern void update_index_if_able(struct index_state *,
struct lock_file *);
extern int hold_locked_index(struct lock_file *, int);
extern void set_alternate_index_output(const char *);
+extern int verify_index_checksum;
+
/* Environment bits from configuration mechanism */
extern int trust_executable_bit;
extern int trust_ctime;
diff --git a/read-cache.c b/read-cache.c
index 9054369..c4205aa 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1371,6 +1371,9 @@ struct ondisk_cache_entry_extended {
ondisk_cache_entry_extended_size(ce_namelen(ce)) : \
ondisk_cache_entry_size(ce_namelen(ce)))
+/* Allow fsck to force verification of the index checksum. */
+int verify_index_checksum;
+
static int verify_hdr(struct cache_header *hdr, unsigned long size)
{
git_SHA_CTX c;
@@ -1382,6 +1385,10 @@ static int verify_hdr(struct cache_header *hdr, unsigned
long size)
hdr_version = ntohl(hdr->hdr_version);
if (hdr_version < INDEX_FORMAT_LB || INDEX_FORMAT_UB < hdr_version)
return error("bad index version %d", hdr_version);
+
+ if (!verify_index_checksum)
+ return 0;
+
git_SHA1_Init(&c);
git_SHA1_Update(&c, hdr, size - 20);
git_SHA1_Final(sha1, &c);
diff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh
index 33a51c9..eff1cd6 100755
--- a/t/t1450-fsck.sh
+++ b/t/t1450-fsck.sh
@@ -689,4 +689,36 @@ test_expect_success 'bogus head does not fallback to all
heads' '
! grep $blob out
'
+# Corrupt the checksum on the index.
+# Add 1 to the last byte in the SHA.
+corrupt_index_checksum () {
+perl -w -e '
+ use Fcntl ":seek";
+ open my $fh, "+<", ".git/index" or die "open: $!";
+ binmode $fh;
+ seek $fh, -1, SEEK_END or die "seek: $!";
+ read $fh, my $in_byte, 1 or die "read: $!";
+
+ $in_value = unpack("C", $in_byte);
+ $out_value = ($in_value + 1) & 255;
+
+ $out_byte = pack("C", $out_value);
+
+ seek $fh, -1, SEEK_END or die "seek: $!";
+ print $fh $out_byte;
+ close $fh or die "close: $!";
+'
+}
+
+# Corrupt the checksum on the index and then
+# verify that only fsck notices.
+test_expect_success 'detect corrupt index file in fsck' '
+ cp .git/index .git/index.backup &&
+ test_when_finished "mv .git/index.backup .git/index" &&
+ corrupt_index_checksum &&
+ test_must_fail git fsck --cache 2>expect &&
+ grep "bad index file" expect &&
+ git status
+'
+
test_done
--
2.9.3
[PATCH v12 4/5] read-cache: speed up has_dir_name (part 1)
From: Jeff Hostetler
Teach has_dir_name() to see if the path of the new item
is greater than the last path in the index array before
attempting to search for it.
has_dir_name() is looking for file/directory collisions
in the index and has to consider each sub-directory
prefix in turn. This can cause multiple binary searches
for each path.
During operations like checkout, merge_working_tree()
populates the new index in sorted order, so we expect
to be able to append in many cases.
This commit is part 1 of 2. This commit handles the top
of has_dir_name() and the trivial optimization.
Signed-off-by: Jeff Hostetler
---
read-cache.c | 45 +
1 file changed, 45 insertions(+)
diff --git a/read-cache.c b/read-cache.c
index 6a27688..9af0bd4 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -910,6 +910,9 @@ int strcmp_offset(const char *s1, const char *s2, size_t
*first_change)
/*
* Do we have another file with a pathname that is a proper
* subset of the name we're trying to add?
+ *
+ * That is, is there another file in the index with a path
+ * that matches a sub-directory in the given entry?
*/
static int has_dir_name(struct index_state *istate,
const struct cache_entry *ce, int pos, int
ok_to_replace)
@@ -918,6 +921,48 @@ static int has_dir_name(struct index_state *istate,
int stage = ce_stage(ce);
const char *name = ce->name;
const char *slash = name + ce_namelen(ce);
+ size_t len_eq_last;
+ int cmp_last = 0;
+
+ /*
+* We are frequently called during an iteration on a sorted
+* list of pathnames and while building a new index. Therefore,
+* there is a high probability that this entry will eventually
+* be appended to the index, rather than inserted in the middle.
+* If we can confirm that, we can avoid binary searches on the
+* components of the pathname.
+*
+* Compare the entry's full path with the last path in the index.
+*/
+ if (istate->cache_nr > 0) {
+ cmp_last = strcmp_offset(name,
+ istate->cache[istate->cache_nr - 1]->name,
+ &len_eq_last);
+ if (cmp_last > 0) {
+ if (len_eq_last == 0) {
+ /*
+* The entry sorts AFTER the last one in the
+* index and their paths have no common prefix,
+* so there cannot be a F/D conflict.
+*/
+ return retval;
+ } else {
+ /*
+* The entry sorts AFTER the last one in the
+* index, but has a common prefix. Fall through
+* to the loop below to disect the entry's path
+* and see where the difference is.
+*/
+ }
+ } else if (cmp_last == 0) {
+ /*
+* The entry exactly matches the last one in the
+* index, but because of multiple stage and CE_REMOVE
+* items, we fall through and let the regular search
+* code handle it.
+*/
+ }
+ }
for (;;) {
int len;
--
2.9.3
[PATCH v12 1/5] read-cache: add strcmp_offset function
From: Jeff Hostetler
Add strcmp_offset() function to also return the offset of the
first change.
Add unit test and helper to verify.
Signed-off-by: Jeff Hostetler
---
Makefile | 1 +
cache.h | 1 +
read-cache.c | 20
t/helper/.gitignore | 1 +
t/helper/test-strcmp-offset.c | 22 ++
t/t0065-strcmp-offset.sh | 21 +
6 files changed, 66 insertions(+)
create mode 100644 t/helper/test-strcmp-offset.c
create mode 100755 t/t0065-strcmp-offset.sh
diff --git a/Makefile b/Makefile
index 9ec6065..4c4c246 100644
--- a/Makefile
+++ b/Makefile
@@ -631,6 +631,7 @@ TEST_PROGRAMS_NEED_X += test-scrap-cache-tree
TEST_PROGRAMS_NEED_X += test-sha1
TEST_PROGRAMS_NEED_X += test-sha1-array
TEST_PROGRAMS_NEED_X += test-sigchain
+TEST_PROGRAMS_NEED_X += test-strcmp-offset
TEST_PROGRAMS_NEED_X += test-string-list
TEST_PROGRAMS_NEED_X += test-submodule-config
TEST_PROGRAMS_NEED_X += test-subprocess
diff --git a/cache.h b/cache.h
index 80b6372..3c55047 100644
--- a/cache.h
+++ b/cache.h
@@ -574,6 +574,7 @@ extern int write_locked_index(struct index_state *, struct
lock_file *lock, unsi
extern int discard_index(struct index_state *);
extern int unmerged_index(const struct index_state *);
extern int verify_path(const char *path);
+extern int strcmp_offset(const char *s1, const char *s2, size_t *first_change);
extern int index_dir_exists(struct index_state *istate, const char *name, int
namelen);
extern void adjust_dirname_case(struct index_state *istate, char *name);
extern struct cache_entry *index_file_exists(struct index_state *istate, const
char *name, int namelen, int igncase);
diff --git a/read-cache.c b/read-cache.c
index 9054369..97f13a1 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -887,6 +887,26 @@ static int has_file_name(struct index_state *istate,
return retval;
}
+
+/*
+ * Like strcmp(), but also return the offset of the first change.
+ * If strings are equal, return the length.
+ */
+int strcmp_offset(const char *s1, const char *s2, size_t *first_change)
+{
+ size_t k;
+
+ if (!first_change)
+ return strcmp(s1, s2);
+
+ for (k = 0; s1[k] == s2[k]; k++)
+ if (s1[k] == '\0')
+ break;
+
+ *first_change = k;
+ return (unsigned char)s1[k] - (unsigned char)s2[k];
+}
+
/*
* Do we have another file with a pathname that is a proper
* subset of the name we're trying to add?
diff --git a/t/helper/.gitignore b/t/helper/.gitignore
index d6e8b36..0a89531 100644
--- a/t/helper/.gitignore
+++ b/t/helper/.gitignore
@@ -25,6 +25,7 @@
/test-sha1
/test-sha1-array
/test-sigchain
+/test-strcmp-offset
/test-string-list
/test-submodule-config
/test-subprocess
diff --git a/t/helper/test-strcmp-offset.c b/t/helper/test-strcmp-offset.c
new file mode 100644
index 000..4a45a54
--- /dev/null
+++ b/t/helper/test-strcmp-offset.c
@@ -0,0 +1,22 @@
+#include "cache.h"
+
+int cmd_main(int argc, const char **argv)
+{
+ int result;
+ size_t offset;
+
+ if (!argv[1] || !argv[2])
+ die("usage: %s ", argv[0]);
+
+ result = strcmp_offset(argv[1], argv[2], &offset);
+
+ /*
+* Because differnt CRTs behave differently, only rely on signs
+* of the result values.
+*/
+ result = (result < 0 ? -1 :
+ result > 0 ? 1 :
+ 0);
+ printf("%d %"PRIuMAX"\n", result, (uintmax_t)offset);
+ return 0;
+}
diff --git a/t/t0065-strcmp-offset.sh b/t/t0065-strcmp-offset.sh
new file mode 100755
index 000..7d6d214
--- /dev/null
+++ b/t/t0065-strcmp-offset.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+
+test_description='Test strcmp_offset functionality'
+
+. ./test-lib.sh
+
+while read s1 s2 expect
+do
+ test_expect_success "strcmp_offset($s1, $s2)" '
+ echo "$expect" >expect &&
+ test-strcmp-offset "$s1" "$s2" >actual &&
+ test_cmp expect actual
+ '
+done <<-EOF
+abc abc 0 3
+abc def -1 0
+abc abz -1 2
+abc abcdef -1 3
+EOF
+
+test_done
--
2.9.3
[PATCH v12 5/5] read-cache: speed up has_dir_name (part 2)
From: Jeff Hostetler
Teach has_dir_name() to see if the path of the new item
is greater than the last path in the index array before
attempting to search for it.
has_dir_name() is looking for file/directory collisions
in the index and has to consider each sub-directory
prefix in turn. This can cause multiple binary searches
for each path.
During operations like checkout, merge_working_tree()
populates the new index in sorted order, so we expect
to be able to append in many cases.
This commit is part 2 of 2. This commit handles the
additional possible short-cuts as we look at each
sub-directory prefix.
The net-net gains for add_index_entry_with_check() and
both had_dir_name() commits are best seen for very
large repos.
Here are results for an INFLATED version of linux.git
with 1M files.
$ GIT_PERF_REPO=/mnt/test/linux_inflated.git/ ./run upstream/base HEAD
./p0006-read-tree-checkout.sh
Testupstream/base
HEAD
0006.2: read-tree br_base br_ballast (1043893) 3.79(3.63+0.15)
2.68(2.52+0.15) -29.3%
0006.3: switch between br_base br_ballast (1043893) 7.55(6.58+0.44)
6.03(4.60+0.43) -20.1%
0006.4: switch between br_ballast br_ballast_plus_1 (1043893)
10.84(9.26+0.59) 8.44(7.06+0.65) -22.1%
0006.5: switch between aliases (1043893)
10.93(9.39+0.58) 10.24(7.04+0.63) -6.3%
Here are results for a synthetic repo with 4.2M files.
$ GIT_PERF_REPO=~/work/gfw/t/perf/repos/gen-many-files-10.4.3.git/ ./run HEAD~3
HEAD ./p0006-read-tree-checkout.sh
TestHEAD~3
HEAD
0006.2: read-tree br_base br_ballast (4194305)
29.96(19.26+10.50) 23.76(13.42+10.12) -20.7%
0006.3: switch between br_base br_ballast (4194305)
56.95(36.08+16.83) 45.54(25.94+15.68) -20.0%
0006.4: switch between br_ballast br_ballast_plus_1 (4194305)
90.94(51.50+31.52) 78.22(39.39+30.70) -14.0%
0006.5: switch between aliases (4194305)
93.72(51.63+34.09) 77.94(39.00+30.88) -16.8%
Results for medium repos (like linux.git) are mixed and have
more variance (probably do to disk IO unrelated to this test.
$ GIT_PERF_REPO=/mnt/test/linux.git/ ./run HEAD~3 HEAD
./p0006-read-tree-checkout.sh
Test HEAD~3
HEAD
0006.2: read-tree br_base br_ballast (57994) 0.25(0.21+0.03)
0.20(0.17+0.02) -20.0%
0006.3: switch between br_base br_ballast (57994) 10.67(6.06+2.92)
10.51(5.94+2.91) -1.5%
0006.4: switch between br_ballast br_ballast_plus_1 (57994) 0.59(0.47+0.16)
0.52(0.40+0.13) -11.9%
0006.5: switch between aliases (57994)0.59(0.44+0.17)
0.51(0.38+0.14) -13.6%
$ GIT_PERF_REPO=/mnt/test/linux.git/ ./run HEAD~3 HEAD
./p0006-read-tree-checkout.sh
Test HEAD~3
HEAD
0006.2: read-tree br_base br_ballast (57994) 0.24(0.21+0.02)
0.21(0.18+0.02) -12.5%
0006.3: switch between br_base br_ballast (57994) 10.42(5.98+2.91)
10.66(5.86+3.09) +2.3%
0006.4: switch between br_ballast br_ballast_plus_1 (57994) 0.59(0.49+0.13)
0.53(0.37+0.16) -10.2%
0006.5: switch between aliases (57994)0.59(0.43+0.17)
0.50(0.37+0.14) -15.3%
Results for smaller repos (like git.git) are not significant.
$ ./run HEAD~3 HEAD ./p0006-read-tree-checkout.sh
Test HEAD~3
HEAD
0006.2: read-tree br_base br_ballast (3043) 0.01(0.00+0.00)
0.01(0.00+0.00) +0.0%
0006.3: switch between br_base br_ballast (3043) 0.31(0.17+0.11)
0.29(0.19+0.08) -6.5%
0006.4: switch between br_ballast br_ballast_plus_1 (3043) 0.03(0.02+0.00)
0.03(0.02+0.00) +0.0%
0006.5: switch between aliases (3043)0.03(0.02+0.00)
0.03(0.02+0.00) +0.0%
Signed-off-by: Jeff Hostetler
---
read-cache.c | 63 +++-
1 file changed, 62 insertions(+), 1 deletion(-)
diff --git a/read-cache.c b/read-cache.c
index 9af0bd4..c252b6c 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -965,7 +965,7 @@ static int has_dir_name(struct index_state *istate,
}
for (;;) {
- int len;
+ size_t len;
for (;;) {
if (*--slash == '/')
@@ -975,6 +975,67 @@ static int has_dir_name(struct index_state *istate,
}
len = slash - name;
+ if (cmp_last > 0) {
+ /*
+* (len + 1) is a directory boundary (including
+* the trailing slash). And since the loop is
+* decrementing "slash", the first iteration is
+
[PATCH v12 0/5] read-cache: speed up add_index_entry
From: Jeff Hostetler Version 12 adds a new t/perf/repo/inflate-repo.sh script to let you inflate a test repo, such as a copy of git.git or linux.git, to have a branch containing a very large number of (non-synthetic) files. It also fixes the "##" comments in the many-files.sh script as mentioned on the mailing list. I've also updated the commit message on part 2 to show the results when run on an inflated copy of linux.git with 1M+ files. Jeff Hostetler (5): read-cache: add strcmp_offset function p0006-read-tree-checkout: perf test to time read-tree read-cache: speed up add_index_entry during checkout read-cache: speed up has_dir_name (part 1) read-cache: speed up has_dir_name (part 2) Makefile | 1 + cache.h| 1 + read-cache.c | 139 - t/helper/.gitignore| 1 + t/helper/test-strcmp-offset.c | 22 ++ t/perf/p0006-read-tree-checkout.sh | 67 ++ t/perf/repos/.gitignore| 1 + t/perf/repos/inflate-repo.sh | 86 +++ t/perf/repos/many-files.sh | 110 + t/t0065-strcmp-offset.sh | 21 ++ 10 files changed, 447 insertions(+), 2 deletions(-) create mode 100644 t/helper/test-strcmp-offset.c create mode 100755 t/perf/p0006-read-tree-checkout.sh create mode 100644 t/perf/repos/.gitignore create mode 100755 t/perf/repos/inflate-repo.sh create mode 100755 t/perf/repos/many-files.sh create mode 100755 t/t0065-strcmp-offset.sh -- 2.9.3
[PATCH v12 3/5] read-cache: speed up add_index_entry during checkout
From: Jeff Hostetler
Teach add_index_entry_with_check() to see if the path
of the new item is greater than the last path in the
index array before attempting to search for it.
During checkout, merge_working_tree() populates the new
index in sorted order, so this change will save a binary
lookups per file. This preserves the original behavior
but simply checks the last element before starting the
search.
This helps performance on very large repositories.
Signed-off-by: Jeff Hostetler
---
read-cache.c | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/read-cache.c b/read-cache.c
index 97f13a1..6a27688 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1021,7 +1021,16 @@ static int add_index_entry_with_check(struct index_state
*istate, struct cache_e
if (!(option & ADD_CACHE_KEEP_CACHE_TREE))
cache_tree_invalidate_path(istate, ce->name);
- pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce),
ce_stage(ce));
+
+ /*
+* If this entry's path sorts after the last entry in the index,
+* we can avoid searching for it.
+*/
+ if (istate->cache_nr > 0 &&
+ strcmp(ce->name, istate->cache[istate->cache_nr - 1]->name) > 0)
+ pos = -istate->cache_nr - 1;
+ else
+ pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce),
ce_stage(ce));
/* existing match? Just replace it. */
if (pos >= 0) {
--
2.9.3
[PATCH v12 2/5] p0006-read-tree-checkout: perf test to time read-tree
From: Jeff Hostetler
Created t/perf/repos/many-files.sh to generate large, but
artificial repositories.
Created t/perf/inflate-repo.sh to alter an EXISTING repo
to have a set of large commits. This can be used to create
a branch with 1M+ files in repositories like git.git or
linux.git, but with more realistic content. It does this
by making multiple copies of the entire worktree in a series
of sub-directories.
The branch name and ballast structure created by both scripts
match, so either script can be used to generate very large
test repositories for the following perf test.
Created t/perf/p0006-read-tree-checkout.sh to measure
performance on various read-tree, checkout, and update-index
operations. This test can run using either normal repos or
ones from the above scripts.
Signed-off-by: Jeff Hostetler
---
t/perf/p0006-read-tree-checkout.sh | 67 ++
t/perf/repos/.gitignore| 1 +
t/perf/repos/inflate-repo.sh | 86 +
t/perf/repos/many-files.sh | 110 +
4 files changed, 264 insertions(+)
create mode 100755 t/perf/p0006-read-tree-checkout.sh
create mode 100644 t/perf/repos/.gitignore
create mode 100755 t/perf/repos/inflate-repo.sh
create mode 100755 t/perf/repos/many-files.sh
diff --git a/t/perf/p0006-read-tree-checkout.sh
b/t/perf/p0006-read-tree-checkout.sh
new file mode 100755
index 000..78cc23f
--- /dev/null
+++ b/t/perf/p0006-read-tree-checkout.sh
@@ -0,0 +1,67 @@
+#!/bin/sh
+#
+# This test measures the performance of various read-tree
+# and checkout operations. It is primarily interested in
+# the algorithmic costs of index operations and recursive
+# tree traversal -- and NOT disk I/O on thousands of files.
+
+test_description="Tests performance of read-tree"
+
+. ./perf-lib.sh
+
+test_perf_default_repo
+
+# If the test repo was generated by ./repos/many-files.sh
+# then we know something about the data shape and branches,
+# so we can isolate testing to the ballast-related commits
+# and setup sparse-checkout so we don't have to populate
+# the ballast files and directories.
+#
+# Otherwise, we make some general assumptions about the
+# repo and consider the entire history of the current
+# branch to be the ballast.
+
+test_expect_success "setup repo" '
+ if git rev-parse --verify refs/heads/p0006-ballast^{commit}
+ then
+ echo Assuming synthetic repo from many-files.sh
+ git branch br_base master
+ git branch br_ballast p0006-ballast^
+ git branch br_ballast_alias p0006-ballast^
+ git branch br_ballast_plus_1 p0006-ballast
+ git config --local core.sparsecheckout 1
+ cat >.git/info/sparse-checkout <<-EOF
+ /*
+ !ballast/*
+ EOF
+ else
+ echo Assuming non-synthetic repo...
+ git branch br_base$(git rev-list HEAD | tail -n 1)
+ git branch br_ballast HEAD^ || error "no ancestor
commit from current head"
+ git branch br_ballast_alias HEAD^
+ git branch br_ballast_plus_1 HEAD
+ fi &&
+ git checkout -q br_ballast &&
+ nr_files=$(git ls-files | wc -l)
+'
+
+test_perf "read-tree br_base br_ballast ($nr_files)" '
+ git read-tree -m br_base br_ballast -n
+'
+
+test_perf "switch between br_base br_ballast ($nr_files)" '
+ git checkout -q br_base &&
+ git checkout -q br_ballast
+'
+
+test_perf "switch between br_ballast br_ballast_plus_1 ($nr_files)" '
+ git checkout -q br_ballast_plus_1 &&
+ git checkout -q br_ballast
+'
+
+test_perf "switch between aliases ($nr_files)" '
+ git checkout -q br_ballast_alias &&
+ git checkout -q br_ballast
+'
+
+test_done
diff --git a/t/perf/repos/.gitignore b/t/perf/repos/.gitignore
new file mode 100644
index 000..72e3dc3
--- /dev/null
+++ b/t/perf/repos/.gitignore
@@ -0,0 +1 @@
+gen-*/
diff --git a/t/perf/repos/inflate-repo.sh b/t/perf/repos/inflate-repo.sh
new file mode 100755
index 000..64f5d7a
--- /dev/null
+++ b/t/perf/repos/inflate-repo.sh
@@ -0,0 +1,86 @@
+#!/bin/sh
+# Inflate the size of an EXISTING repo.
+#
+# This script should be run inside the worktree of a TEST repo.
+# It will use the contents of the current HEAD to generate a
+# commit containing copies of the current worktree such that the
+# total size of the commit has at least files.
+#
+# Usage: [-t target_size] [-b branch_name]
+
+set -e
+
+target_size=1
+branch_name=p0006-ballast
+ballast=ballast
+
+while test "$#" -ne 0
+do
+case "$1" in
+ -b)
+ shift;
+ test "$#" -ne 0 || { echo 'er
[PATCH v1] diffcore-rename: speed up register_rename_src
From: Jeff Hostetler
Teach register_rename_src() to see if new file pair
can simply be appended to the rename_src[] array before
performing the binary search to find the proper insertion
point.
This is a performance optimization. This routine is called
during run_diff_files in status and the caller is iterating
over the sorted index, so we should expect to be able to
append in the normal case. The existing insert logic is
preserved so we don't have to assume that, but simply take
advantage of it if possible.
Using t/perf/p0005-status.h (from a parallel patch series),
we get the following improvement on a 4.2M file repo:
TestHEAD~1 HEAD
0005.2: read-tree status br_ballast (4194305) 59.14(31.85+18.79)
55.48(28.52+20.71) -6.2%
On a 1M file repo:
TestHEAD~1HEAD
0005.2: read-tree status br_ballast (101) 8.20(4.82+3.35)
7.91(4.57+3.27) -3.5%
On a smaller repo, like linux.git (58K files), results are masked
by normal I/O variance.
Test HEAD~1HEAD
0005.2: read-tree status br_ballast (57994) 0.43(0.30+0.13) 0.42(0.31+0.12)
-2.3%
0005.2: read-tree status br_ballast (57994) 0.42(0.32+0.09) 0.43(0.34+0.10)
+2.4%
0005.2: read-tree status br_ballast (57994) 0.44(0.33+0.10) 0.42(0.26+0.16)
-4.5%
Signed-off-by: Jeff Hostetler
---
diffcore-rename.c | 13 +
1 file changed, 13 insertions(+)
diff --git a/diffcore-rename.c b/diffcore-rename.c
index f7444c8..543a409 100644
--- a/diffcore-rename.c
+++ b/diffcore-rename.c
@@ -81,6 +81,18 @@ static struct diff_rename_src *register_rename_src(struct
diff_filepair *p)
first = 0;
last = rename_src_nr;
+
+ if (last > 0) {
+ struct diff_rename_src *src = &(rename_src[last-1]);
+ int cmp = strcmp(one->path, src->p->one->path);
+ if (!cmp)
+ return src;
+ if (cmp > 0) {
+ first = last;
+ goto append_it;
+ }
+ }
+
while (last > first) {
int next = (last + first) >> 1;
struct diff_rename_src *src = &(rename_src[next]);
@@ -94,6 +106,7 @@ static struct diff_rename_src *register_rename_src(struct
diff_filepair *p)
first = next+1;
}
+append_it:
/* insert to make it at "first" */
ALLOC_GROW(rename_src, rename_src_nr + 1, rename_src_alloc);
rename_src_nr++;
--
2.9.3
[PATCH v1] diffcore-rename speedup
From: Jeff Hostetler Here is another micro-optimization for very large repositories. Speed up register_rename_src() in diffcore-rename.c Jeff Hostetler (1): diffcore-rename: speed up register_rename_src diffcore-rename.c | 13 + 1 file changed, 13 insertions(+) -- 2.9.3
[PATCH v11 2/5] p0006-read-tree-checkout: perf test to time read-tree
From: Jeff Hostetler
Created t/perf/repos/many-files.sh to generate large, but
artificial repositories.
Created t/perf/p0006-read-tree-checkout.sh to measure
performance on various read-tree, checkout, and update-index
operations. This test can run using either artificial repos
described above or normal repos.
Signed-off-by: Jeff Hostetler
---
t/perf/p0006-read-tree-checkout.sh | 67 ++
t/perf/repos/.gitignore| 1 +
t/perf/repos/many-files.sh | 110 +
3 files changed, 178 insertions(+)
create mode 100755 t/perf/p0006-read-tree-checkout.sh
create mode 100644 t/perf/repos/.gitignore
create mode 100755 t/perf/repos/many-files.sh
diff --git a/t/perf/p0006-read-tree-checkout.sh
b/t/perf/p0006-read-tree-checkout.sh
new file mode 100755
index 000..78cc23f
--- /dev/null
+++ b/t/perf/p0006-read-tree-checkout.sh
@@ -0,0 +1,67 @@
+#!/bin/sh
+#
+# This test measures the performance of various read-tree
+# and checkout operations. It is primarily interested in
+# the algorithmic costs of index operations and recursive
+# tree traversal -- and NOT disk I/O on thousands of files.
+
+test_description="Tests performance of read-tree"
+
+. ./perf-lib.sh
+
+test_perf_default_repo
+
+# If the test repo was generated by ./repos/many-files.sh
+# then we know something about the data shape and branches,
+# so we can isolate testing to the ballast-related commits
+# and setup sparse-checkout so we don't have to populate
+# the ballast files and directories.
+#
+# Otherwise, we make some general assumptions about the
+# repo and consider the entire history of the current
+# branch to be the ballast.
+
+test_expect_success "setup repo" '
+ if git rev-parse --verify refs/heads/p0006-ballast^{commit}
+ then
+ echo Assuming synthetic repo from many-files.sh
+ git branch br_base master
+ git branch br_ballast p0006-ballast^
+ git branch br_ballast_alias p0006-ballast^
+ git branch br_ballast_plus_1 p0006-ballast
+ git config --local core.sparsecheckout 1
+ cat >.git/info/sparse-checkout <<-EOF
+ /*
+ !ballast/*
+ EOF
+ else
+ echo Assuming non-synthetic repo...
+ git branch br_base$(git rev-list HEAD | tail -n 1)
+ git branch br_ballast HEAD^ || error "no ancestor
commit from current head"
+ git branch br_ballast_alias HEAD^
+ git branch br_ballast_plus_1 HEAD
+ fi &&
+ git checkout -q br_ballast &&
+ nr_files=$(git ls-files | wc -l)
+'
+
+test_perf "read-tree br_base br_ballast ($nr_files)" '
+ git read-tree -m br_base br_ballast -n
+'
+
+test_perf "switch between br_base br_ballast ($nr_files)" '
+ git checkout -q br_base &&
+ git checkout -q br_ballast
+'
+
+test_perf "switch between br_ballast br_ballast_plus_1 ($nr_files)" '
+ git checkout -q br_ballast_plus_1 &&
+ git checkout -q br_ballast
+'
+
+test_perf "switch between aliases ($nr_files)" '
+ git checkout -q br_ballast_alias &&
+ git checkout -q br_ballast
+'
+
+test_done
diff --git a/t/perf/repos/.gitignore b/t/perf/repos/.gitignore
new file mode 100644
index 000..72e3dc3
--- /dev/null
+++ b/t/perf/repos/.gitignore
@@ -0,0 +1 @@
+gen-*/
diff --git a/t/perf/repos/many-files.sh b/t/perf/repos/many-files.sh
new file mode 100755
index 000..5a1d25e
--- /dev/null
+++ b/t/perf/repos/many-files.sh
@@ -0,0 +1,110 @@
+#!/bin/sh
+## Generate test data repository using the given parameters.
+## When omitted, we create "gen-many-files-d-w-f.git".
+##
+## Usage: [-r repo] [-d depth] [-w width] [-f files]
+##
+## -r repo: path to the new repo to be generated
+## -d depth: the depth of sub-directories
+## -w width: the number of sub-directories at each level
+## -f files: the number of files created in each directory
+##
+## Note that all files will have the same SHA-1 and each
+## directory at a level will have the same SHA-1, so we
+## will potentially have a large index, but not a large
+## ODB.
+##
+## Ballast will be created under "ballast/".
+
+EMPTY_BLOB=e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
+
+set -e
+
+## (5, 10, 9) will create 999,999 ballast files.
+## (4, 10, 9) will create 99,999 ballast files.
+depth=5
+width=10
+files=9
+
+while test "$#" -ne 0
+do
+case "$1" in
+ -r)
+ shift;
+ test "$#" -ne 0 || { echo 'error: -r requires an argument' >&2;
exit 1; }
+ repo=$1;
+ shift ;;
+ -d)
+ shift;
+ test "$#" -ne 0 || { ech
[PATCH v11 4/5] read-cache: speed up has_dir_name (part 1)
From: Jeff Hostetler
Teach has_dir_name() to see if the path of the new item
is greater than the last path in the index array before
attempting to search for it.
has_dir_name() is looking for file/directory collisions
in the index and has to consider each sub-directory
prefix in turn. This can cause multiple binary searches
for each path.
During operations like checkout, merge_working_tree()
populates the new index in sorted order, so we expect
to be able to append in many cases.
This commit is part 1 of 2. This commit handles the top
of has_dir_name() and the trivial optimization.
Signed-off-by: Jeff Hostetler
---
read-cache.c | 45 +
1 file changed, 45 insertions(+)
diff --git a/read-cache.c b/read-cache.c
index 6a27688..9af0bd4 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -910,6 +910,9 @@ int strcmp_offset(const char *s1, const char *s2, size_t
*first_change)
/*
* Do we have another file with a pathname that is a proper
* subset of the name we're trying to add?
+ *
+ * That is, is there another file in the index with a path
+ * that matches a sub-directory in the given entry?
*/
static int has_dir_name(struct index_state *istate,
const struct cache_entry *ce, int pos, int
ok_to_replace)
@@ -918,6 +921,48 @@ static int has_dir_name(struct index_state *istate,
int stage = ce_stage(ce);
const char *name = ce->name;
const char *slash = name + ce_namelen(ce);
+ size_t len_eq_last;
+ int cmp_last = 0;
+
+ /*
+* We are frequently called during an iteration on a sorted
+* list of pathnames and while building a new index. Therefore,
+* there is a high probability that this entry will eventually
+* be appended to the index, rather than inserted in the middle.
+* If we can confirm that, we can avoid binary searches on the
+* components of the pathname.
+*
+* Compare the entry's full path with the last path in the index.
+*/
+ if (istate->cache_nr > 0) {
+ cmp_last = strcmp_offset(name,
+ istate->cache[istate->cache_nr - 1]->name,
+ &len_eq_last);
+ if (cmp_last > 0) {
+ if (len_eq_last == 0) {
+ /*
+* The entry sorts AFTER the last one in the
+* index and their paths have no common prefix,
+* so there cannot be a F/D conflict.
+*/
+ return retval;
+ } else {
+ /*
+* The entry sorts AFTER the last one in the
+* index, but has a common prefix. Fall through
+* to the loop below to disect the entry's path
+* and see where the difference is.
+*/
+ }
+ } else if (cmp_last == 0) {
+ /*
+* The entry exactly matches the last one in the
+* index, but because of multiple stage and CE_REMOVE
+* items, we fall through and let the regular search
+* code handle it.
+*/
+ }
+ }
for (;;) {
int len;
--
2.9.3
[PATCH v11 5/5] read-cache: speed up has_dir_name (part 2)
From: Jeff Hostetler
Teach has_dir_name() to see if the path of the new item
is greater than the last path in the index array before
attempting to search for it.
has_dir_name() is looking for file/directory collisions
in the index and has to consider each sub-directory
prefix in turn. This can cause multiple binary searches
for each path.
During operations like checkout, merge_working_tree()
populates the new index in sorted order, so we expect
to be able to append in many cases.
This commit is part 2 of 2. This commit handles the
additional possible short-cuts as we look at each
sub-directory prefix.
The net-net gains for add_index_entry_with_check() and
both had_dir_name() commits are best seen for very
large repos.
Here are results for a synthetic repo with 4.2M files.
$ GIT_PERF_REPO=~/work/gfw/t/perf/repos/gen-many-files-10.4.3.git/ ./run HEAD~3
HEAD ./p0006-read-tree-checkout.sh
TestHEAD~3
HEAD
0006.2: read-tree br_base br_ballast (4194305)
29.96(19.26+10.50) 23.76(13.42+10.12) -20.7%
0006.3: switch between br_base br_ballast (4194305)
56.95(36.08+16.83) 45.54(25.94+15.68) -20.0%
0006.4: switch between br_ballast br_ballast_plus_1 (4194305)
90.94(51.50+31.52) 78.22(39.39+30.70) -14.0%
0006.5: switch between aliases (4194305)
93.72(51.63+34.09) 77.94(39.00+30.88) -16.8%
Results for medium repos (like linux.git) are mixed and have
more variance (probably do to disk IO unrelated to this test.
$ GIT_PERF_REPO=/mnt/test/linux.git/ ./run HEAD~3 HEAD
./p0006-read-tree-checkout.sh
Test HEAD~3
HEAD
0006.2: read-tree br_base br_ballast (57994) 0.25(0.21+0.03)
0.20(0.17+0.02) -20.0%
0006.3: switch between br_base br_ballast (57994) 10.67(6.06+2.92)
10.51(5.94+2.91) -1.5%
0006.4: switch between br_ballast br_ballast_plus_1 (57994) 0.59(0.47+0.16)
0.52(0.40+0.13) -11.9%
0006.5: switch between aliases (57994)0.59(0.44+0.17)
0.51(0.38+0.14) -13.6%
$ GIT_PERF_REPO=/mnt/test/linux.git/ ./run HEAD~3 HEAD
./p0006-read-tree-checkout.sh
Test HEAD~3
HEAD
0006.2: read-tree br_base br_ballast (57994) 0.24(0.21+0.02)
0.21(0.18+0.02) -12.5%
0006.3: switch between br_base br_ballast (57994) 10.42(5.98+2.91)
10.66(5.86+3.09) +2.3%
0006.4: switch between br_ballast br_ballast_plus_1 (57994) 0.59(0.49+0.13)
0.53(0.37+0.16) -10.2%
0006.5: switch between aliases (57994)0.59(0.43+0.17)
0.50(0.37+0.14) -15.3%
Results for smaller repos (like git.git) are not significant.
$ ./run HEAD~3 HEAD ./p0006-read-tree-checkout.sh
Test HEAD~3
HEAD
0006.2: read-tree br_base br_ballast (3043) 0.01(0.00+0.00)
0.01(0.00+0.00) +0.0%
0006.3: switch between br_base br_ballast (3043) 0.31(0.17+0.11)
0.29(0.19+0.08) -6.5%
0006.4: switch between br_ballast br_ballast_plus_1 (3043) 0.03(0.02+0.00)
0.03(0.02+0.00) +0.0%
0006.5: switch between aliases (3043)0.03(0.02+0.00)
0.03(0.02+0.00) +0.0%
Signed-off-by: Jeff Hostetler
---
read-cache.c | 63 +++-
1 file changed, 62 insertions(+), 1 deletion(-)
diff --git a/read-cache.c b/read-cache.c
index 9af0bd4..c252b6c 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -965,7 +965,7 @@ static int has_dir_name(struct index_state *istate,
}
for (;;) {
- int len;
+ size_t len;
for (;;) {
if (*--slash == '/')
@@ -975,6 +975,67 @@ static int has_dir_name(struct index_state *istate,
}
len = slash - name;
+ if (cmp_last > 0) {
+ /*
+* (len + 1) is a directory boundary (including
+* the trailing slash). And since the loop is
+* decrementing "slash", the first iteration is
+* the longest directory prefix; subsequent
+* iterations consider parent directories.
+*/
+
+ if (len + 1 <= len_eq_last) {
+ /*
+* The directory prefix (including the trailing
+* slash) also appears as a prefix in the last
+* entry, so the remainder cannot collide
(because
+* strcmp said the whole path was greater).
+*
+* EQ: last: xxx/A
+
[PATCH v11 3/5] read-cache: speed up add_index_entry during checkout
From: Jeff Hostetler
Teach add_index_entry_with_check() to see if the path
of the new item is greater than the last path in the
index array before attempting to search for it.
During checkout, merge_working_tree() populates the new
index in sorted order, so this change will save a binary
lookups per file. This preserves the original behavior
but simply checks the last element before starting the
search.
This helps performance on very large repositories.
Signed-off-by: Jeff Hostetler
---
read-cache.c | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/read-cache.c b/read-cache.c
index 97f13a1..6a27688 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1021,7 +1021,16 @@ static int add_index_entry_with_check(struct index_state
*istate, struct cache_e
if (!(option & ADD_CACHE_KEEP_CACHE_TREE))
cache_tree_invalidate_path(istate, ce->name);
- pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce),
ce_stage(ce));
+
+ /*
+* If this entry's path sorts after the last entry in the index,
+* we can avoid searching for it.
+*/
+ if (istate->cache_nr > 0 &&
+ strcmp(ce->name, istate->cache[istate->cache_nr - 1]->name) > 0)
+ pos = -istate->cache_nr - 1;
+ else
+ pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce),
ce_stage(ce));
/* existing match? Just replace it. */
if (pos >= 0) {
--
2.9.3
[PATCH v11 0/5] read-cache: speed up add_index_entry
From: Jeff Hostetler Version 11 splits the changes in read-cache.c into 3 commits so that they can be independently evaluated. And adds subscript guard for istate->cache_nr > 0 which might be necessary if remove_index_entry_at() deletes the only entry in the array. Jeff Hostetler (5): read-cache: add strcmp_offset function p0006-read-tree-checkout: perf test to time read-tree read-cache: speed up add_index_entry during checkout read-cache: speed up has_dir_name (part 1) read-cache: speed up has_dir_name (part 2) Makefile | 1 + cache.h| 1 + read-cache.c | 139 - t/helper/.gitignore| 1 + t/helper/test-strcmp-offset.c | 22 ++ t/perf/p0006-read-tree-checkout.sh | 67 ++ t/perf/repos/.gitignore| 1 + t/perf/repos/many-files.sh | 110 + t/t0065-strcmp-offset.sh | 21 ++ 9 files changed, 361 insertions(+), 2 deletions(-) create mode 100644 t/helper/test-strcmp-offset.c create mode 100755 t/perf/p0006-read-tree-checkout.sh create mode 100644 t/perf/repos/.gitignore create mode 100755 t/perf/repos/many-files.sh create mode 100755 t/t0065-strcmp-offset.sh -- 2.9.3
[PATCH v11 1/5] read-cache: add strcmp_offset function
From: Jeff Hostetler
Add strcmp_offset() function to also return the offset of the
first change.
Add unit test and helper to verify.
Signed-off-by: Jeff Hostetler
---
Makefile | 1 +
cache.h | 1 +
read-cache.c | 20
t/helper/.gitignore | 1 +
t/helper/test-strcmp-offset.c | 22 ++
t/t0065-strcmp-offset.sh | 21 +
6 files changed, 66 insertions(+)
create mode 100644 t/helper/test-strcmp-offset.c
create mode 100755 t/t0065-strcmp-offset.sh
diff --git a/Makefile b/Makefile
index 9ec6065..4c4c246 100644
--- a/Makefile
+++ b/Makefile
@@ -631,6 +631,7 @@ TEST_PROGRAMS_NEED_X += test-scrap-cache-tree
TEST_PROGRAMS_NEED_X += test-sha1
TEST_PROGRAMS_NEED_X += test-sha1-array
TEST_PROGRAMS_NEED_X += test-sigchain
+TEST_PROGRAMS_NEED_X += test-strcmp-offset
TEST_PROGRAMS_NEED_X += test-string-list
TEST_PROGRAMS_NEED_X += test-submodule-config
TEST_PROGRAMS_NEED_X += test-subprocess
diff --git a/cache.h b/cache.h
index 80b6372..3c55047 100644
--- a/cache.h
+++ b/cache.h
@@ -574,6 +574,7 @@ extern int write_locked_index(struct index_state *, struct
lock_file *lock, unsi
extern int discard_index(struct index_state *);
extern int unmerged_index(const struct index_state *);
extern int verify_path(const char *path);
+extern int strcmp_offset(const char *s1, const char *s2, size_t *first_change);
extern int index_dir_exists(struct index_state *istate, const char *name, int
namelen);
extern void adjust_dirname_case(struct index_state *istate, char *name);
extern struct cache_entry *index_file_exists(struct index_state *istate, const
char *name, int namelen, int igncase);
diff --git a/read-cache.c b/read-cache.c
index 9054369..97f13a1 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -887,6 +887,26 @@ static int has_file_name(struct index_state *istate,
return retval;
}
+
+/*
+ * Like strcmp(), but also return the offset of the first change.
+ * If strings are equal, return the length.
+ */
+int strcmp_offset(const char *s1, const char *s2, size_t *first_change)
+{
+ size_t k;
+
+ if (!first_change)
+ return strcmp(s1, s2);
+
+ for (k = 0; s1[k] == s2[k]; k++)
+ if (s1[k] == '\0')
+ break;
+
+ *first_change = k;
+ return (unsigned char)s1[k] - (unsigned char)s2[k];
+}
+
/*
* Do we have another file with a pathname that is a proper
* subset of the name we're trying to add?
diff --git a/t/helper/.gitignore b/t/helper/.gitignore
index d6e8b36..0a89531 100644
--- a/t/helper/.gitignore
+++ b/t/helper/.gitignore
@@ -25,6 +25,7 @@
/test-sha1
/test-sha1-array
/test-sigchain
+/test-strcmp-offset
/test-string-list
/test-submodule-config
/test-subprocess
diff --git a/t/helper/test-strcmp-offset.c b/t/helper/test-strcmp-offset.c
new file mode 100644
index 000..4a45a54
--- /dev/null
+++ b/t/helper/test-strcmp-offset.c
@@ -0,0 +1,22 @@
+#include "cache.h"
+
+int cmd_main(int argc, const char **argv)
+{
+ int result;
+ size_t offset;
+
+ if (!argv[1] || !argv[2])
+ die("usage: %s ", argv[0]);
+
+ result = strcmp_offset(argv[1], argv[2], &offset);
+
+ /*
+* Because differnt CRTs behave differently, only rely on signs
+* of the result values.
+*/
+ result = (result < 0 ? -1 :
+ result > 0 ? 1 :
+ 0);
+ printf("%d %"PRIuMAX"\n", result, (uintmax_t)offset);
+ return 0;
+}
diff --git a/t/t0065-strcmp-offset.sh b/t/t0065-strcmp-offset.sh
new file mode 100755
index 000..7d6d214
--- /dev/null
+++ b/t/t0065-strcmp-offset.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+
+test_description='Test strcmp_offset functionality'
+
+. ./test-lib.sh
+
+while read s1 s2 expect
+do
+ test_expect_success "strcmp_offset($s1, $s2)" '
+ echo "$expect" >expect &&
+ test-strcmp-offset "$s1" "$s2" >actual &&
+ test_cmp expect actual
+ '
+done <<-EOF
+abc abc 0 3
+abc def -1 0
+abc abz -1 2
+abc abcdef -1 3
+EOF
+
+test_done
--
2.9.3
[PATCH v7] read-cache: force_verify_index_checksum
From: Jeff Hostetler
Teach git to skip verification of the SHA1-1 checksum at the end of
the index file in verify_hdr() which is called from read_index()
unless the "force_verify_index_checksum" global variable is set.
Teach fsck to force this verification.
The checksum verification is for detecting disk corruption, and for
small projects, the time it takes to compute SHA-1 is not that
significant, but for gigantic repositories this calculation adds
significant time to every command.
These effect can be seen using t/perf/p0002-read-cache.sh:
Test HEAD~1HEAD
--
0002.1: read_cache/discard_cache 1000 times 0.66(0.44+0.20) 0.30(0.27+0.02)
-54.5%
Signed-off-by: Jeff Hostetler
---
builtin/fsck.c | 1 +
cache.h | 2 ++
read-cache.c| 7 +++
t/t1450-fsck.sh | 13 +
4 files changed, 23 insertions(+)
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 1a5cacc..5512d06 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -771,6 +771,7 @@ int cmd_fsck(int argc, const char **argv, const char
*prefix)
}
if (keep_cache_objects) {
+ verify_index_checksum = 1;
read_cache();
for (i = 0; i < active_nr; i++) {
unsigned int mode;
diff --git a/cache.h b/cache.h
index 80b6372..87f13bf 100644
--- a/cache.h
+++ b/cache.h
@@ -685,6 +685,8 @@ extern void update_index_if_able(struct index_state *,
struct lock_file *);
extern int hold_locked_index(struct lock_file *, int);
extern void set_alternate_index_output(const char *);
+extern int verify_index_checksum;
+
/* Environment bits from configuration mechanism */
extern int trust_executable_bit;
extern int trust_ctime;
diff --git a/read-cache.c b/read-cache.c
index 9054369..c4205aa 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1371,6 +1371,9 @@ struct ondisk_cache_entry_extended {
ondisk_cache_entry_extended_size(ce_namelen(ce)) : \
ondisk_cache_entry_size(ce_namelen(ce)))
+/* Allow fsck to force verification of the index checksum. */
+int verify_index_checksum;
+
static int verify_hdr(struct cache_header *hdr, unsigned long size)
{
git_SHA_CTX c;
@@ -1382,6 +1385,10 @@ static int verify_hdr(struct cache_header *hdr, unsigned
long size)
hdr_version = ntohl(hdr->hdr_version);
if (hdr_version < INDEX_FORMAT_LB || INDEX_FORMAT_UB < hdr_version)
return error("bad index version %d", hdr_version);
+
+ if (!verify_index_checksum)
+ return 0;
+
git_SHA1_Init(&c);
git_SHA1_Update(&c, hdr, size - 20);
git_SHA1_Final(sha1, &c);
diff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh
index 33a51c9..677e15a 100755
--- a/t/t1450-fsck.sh
+++ b/t/t1450-fsck.sh
@@ -689,4 +689,17 @@ test_expect_success 'bogus head does not fallback to all
heads' '
! grep $blob out
'
+test_expect_success 'detect corrupt index file in fsck' '
+ cp .git/index .git/index.backup &&
+ test_when_finished "mv .git/index.backup .git/index" &&
+ echo > &&
+ git add &&
+ sed -e "s///" .git/index >.git/index.yyy &&
+ mv .git/index.yyy .git/index &&
+ # Confirm that fsck detects invalid checksum
+ test_must_fail git fsck --cache &&
+ # Confirm that status no longer complains about invalid checksum
+ git status
+'
+
test_done
--
2.9.3
[PATCH v7] read-cache: call verify_hdr() in a background thread
From: Jeff Hostetler Version 7 of this patch cleans up the fsck test using test_when_finished and eliminates unnecessary mv/rm's. Jeff Hostetler (1): read-cache: force_verify_index_checksum builtin/fsck.c | 1 + cache.h | 2 ++ read-cache.c| 7 +++ t/t1450-fsck.sh | 13 + 4 files changed, 23 insertions(+) -- 2.9.3
[PATCH v5] string-list: use ALLOC_GROW macro when reallocing
From: Jeff Hostetler Version 5 addresses coding style comments from the mailing list in the perf test and squashes the changes into 1 commit. Jeff Hostetler (1): string-list: use ALLOC_GROW macro when reallocing string_list string-list.c | 5 + t/perf/p0005-status.sh | 49 + 2 files changed, 50 insertions(+), 4 deletions(-) create mode 100755 t/perf/p0005-status.sh -- 2.9.3
[PATCH v5] string-list: use ALLOC_GROW macro when reallocing string_list
From: Jeff Hostetler
Use ALLOC_GROW() macro when reallocing a string_list array
rather than simply increasing it by 32. This is a performance
optimization.
During status on a very large repo and there are many changes,
a significant percentage of the total run time is spent
reallocing the wt_status.changes array.
This change decreases the time in wt_status_collect_changes_worktree()
from 125 seconds to 45 seconds on my very large repository.
This produced a modest gain on my 1M file artificial repo, but
broke even on linux.git.
TestHEAD^^HEAD
---
0005.2: read-tree status br_ballast (101) 8.29(5.62+2.62)
8.22(5.57+2.63) -0.8%
Signed-off-by: Jeff Hostetler
---
string-list.c | 5 +
t/perf/p0005-status.sh | 49 +
2 files changed, 50 insertions(+), 4 deletions(-)
create mode 100755 t/perf/p0005-status.sh
diff --git a/string-list.c b/string-list.c
index 45016ad..003ca18 100644
--- a/string-list.c
+++ b/string-list.c
@@ -41,10 +41,7 @@ static int add_entry(int insert_at, struct string_list
*list, const char *string
if (exact_match)
return -1 - index;
- if (list->nr + 1 >= list->alloc) {
- list->alloc += 32;
- REALLOC_ARRAY(list->items, list->alloc);
- }
+ ALLOC_GROW(list->items, list->nr+1, list->alloc);
if (index < list->nr)
memmove(list->items + index + 1, list->items + index,
(list->nr - index)
diff --git a/t/perf/p0005-status.sh b/t/perf/p0005-status.sh
new file mode 100755
index 000..0b0aa98
--- /dev/null
+++ b/t/perf/p0005-status.sh
@@ -0,0 +1,49 @@
+#!/bin/sh
+#
+# This test measures the performance of various read-tree
+# and status operations. It is primarily interested in
+# the algorithmic costs of index operations and recursive
+# tree traversal -- and NOT disk I/O on thousands of files.
+
+test_description="Tests performance of read-tree"
+
+. ./perf-lib.sh
+
+test_perf_default_repo
+
+# If the test repo was generated by ./repos/many-files.sh
+# then we know something about the data shape and branches,
+# so we can isolate testing to the ballast-related commits
+# and setup sparse-checkout so we don't have to populate
+# the ballast files and directories.
+#
+# Otherwise, we make some general assumptions about the
+# repo and consider the entire history of the current
+# branch to be the ballast.
+
+test_expect_success "setup repo" '
+ if git rev-parse --verify refs/heads/p0006-ballast^{commit}
+ then
+ echo Assuming synthetic repo from many-files.sh
+ git branch br_basemaster
+ git branch br_ballast p0006-ballast
+ git config --local core.sparsecheckout 1
+ cat >.git/info/sparse-checkout <<-EOF
+ /*
+ !ballast/*
+ EOF
+ else
+ echo Assuming non-synthetic repo...
+ git branch br_base$(git rev-list HEAD | tail -n 1)
+ git branch br_ballast HEAD
+ fi &&
+ git checkout -q br_ballast &&
+ nr_files=$(git ls-files | wc -l)
+'
+
+test_perf "read-tree status br_ballast ($nr_files)" '
+ git read-tree HEAD &&
+ git status
+'
+
+test_done
--
2.9.3
[PATCH v4] unpack-trees: avoid duplicate ODB lookups during checkout
From: Jeff Hostetler Version 4 cleans up the buf[] allocation and freeing as suggested on the mailing list. Jeff Hostetler (1): unpack-trees: avoid duplicate ODB lookups during checkout unpack-trees.c | 38 +- 1 file changed, 33 insertions(+), 5 deletions(-) -- 2.9.3
[PATCH v4] unpack-trees: avoid duplicate ODB lookups during checkout
From: Jeff Hostetler
Teach traverse_trees_recursive() to not do redundant ODB
lookups when both directories refer to the same OID.
In operations such as read-tree and checkout, there will
likely be many peer directories that have the same OID when
the differences between the commits are relatively small.
In these cases we can avoid hitting the ODB multiple times
for the same OID.
This patch handles n=2 and n=3 cases and simply copies the
data rather than repeating the fill_tree_descriptor().
On the Windows repo (500K trees, 3.1M files, 450MB index),
this reduced the overall time by 0.75 seconds when cycling
between 2 commits with a single file difference.
(avg) before: 22.699
(avg) after: 21.955
===
On Linux using p0006-read-tree-checkout.sh with linux.git:
Test HEAD^
HEAD
---
0006.2: read-tree br_base br_ballast (57994) 0.24(0.20+0.03)
0.24(0.22+0.01) +0.0%
0006.3: switch between br_base br_ballast (57994) 10.58(6.23+2.86)
10.67(5.94+2.87) +0.9%
0006.4: switch between br_ballast br_ballast_plus_1 (57994) 0.60(0.44+0.17)
0.57(0.44+0.14) -5.0%
0006.5: switch between aliases (57994)0.59(0.48+0.13)
0.57(0.44+0.15) -3.4%
Signed-off-by: Jeff Hostetler
---
unpack-trees.c | 38 +-
1 file changed, 33 insertions(+), 5 deletions(-)
diff --git a/unpack-trees.c b/unpack-trees.c
index 3a8ee19..07b0f11 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -531,12 +531,18 @@ static int switch_cache_bottom(struct traverse_info *info)
return ret;
}
+static inline int are_same_oid(struct name_entry *name_j, struct name_entry
*name_k)
+{
+ return name_j->oid && name_k->oid && !oidcmp(name_j->oid, name_k->oid);
+}
+
static int traverse_trees_recursive(int n, unsigned long dirmask,
unsigned long df_conflicts,
struct name_entry *names,
struct traverse_info *info)
{
int i, ret, bottom;
+ int nr_buf = 0;
struct tree_desc t[MAX_UNPACK_TREES];
void *buf[MAX_UNPACK_TREES];
struct traverse_info newinfo;
@@ -553,18 +559,40 @@ static int traverse_trees_recursive(int n, unsigned long
dirmask,
newinfo.pathlen += tree_entry_len(p) + 1;
newinfo.df_conflicts |= df_conflicts;
+ /*
+* Fetch the tree from the ODB for each peer directory in the
+* n commits.
+*
+* For 2- and 3-way traversals, we try to avoid hitting the
+* ODB twice for the same OID. This should yield a nice speed
+* up in checkouts and merges when the commits are similar.
+*
+* We don't bother doing the full O(n^2) search for larger n,
+* because wider traversals don't happen that often and we
+* avoid the search setup.
+*
+* When 2 peer OIDs are the same, we just copy the tree
+* descriptor data. This implicitly borrows the buffer
+* data from the earlier cell.
+*/
for (i = 0; i < n; i++, dirmask >>= 1) {
- const unsigned char *sha1 = NULL;
- if (dirmask & 1)
- sha1 = names[i].oid->hash;
- buf[i] = fill_tree_descriptor(t+i, sha1);
+ if (i > 0 && are_same_oid(&names[i], &names[i - 1]))
+ t[i] = t[i - 1];
+ else if (i > 1 && are_same_oid(&names[i], &names[i - 2]))
+ t[i] = t[i - 2];
+ else {
+ const unsigned char *sha1 = NULL;
+ if (dirmask & 1)
+ sha1 = names[i].oid->hash;
+ buf[nr_buf++] = fill_tree_descriptor(t+i, sha1);
+ }
}
bottom = switch_cache_bottom(&newinfo);
ret = traverse_trees(n, t, &newinfo);
restore_cache_bottom(&newinfo, bottom);
- for (i = 0; i < n; i++)
+ for (i = 0; i < nr_buf; i++)
free(buf[i]);
return ret;
--
2.9.3
