state of the htrace
Hi all, Happy (belated) New Year. I've been a bit quiet since I went on vacation for a few weeks recently, and since I have some other projects going on. But today I wanted to highlight some really cool stuff that's been happening in the HTrace and broader tracing community recently. Last weekend, I gave an "intro to htrace" talk this weekend at the Scale 14x Linux conference. See https://www.socallinuxexpo.org/scale/14x/presentations/introducing-apache-htrace . Also there is a video here: https://www.youtube.com/watch?v=t-TwCLwYIGE It starts at 6:50, since the first few minutes of the recording are just me setting up the VGA connection and microphone :P I thought this one went really well (especially the demo), and hopefully will get the word out to even more people. In HDFS, we've been exploring adding annotations to certain trace spans to collect even more useful information. For example, Zhe Zhang posted a patch to HDFS-9576 to add "position" and "length" annotations to DFSInputStream#byteArrayRead, etc. spans. This should give us data on things like the average and median read length in a set of HDFS requests. Similarly, I posted HDFS-9674 which adds the "maximum write latency" to OpWriteBlock spans generated by the DataNode. This is very handy when analyzing HDFS write pipelines. I think we will see more of these really helpful annotations, and they will expand our ideas about what HTrace can do. A few weeks back, I wrote a blog post for my employer, Cloudera. It talks about setting up HTrace and htraced on a CDH5.5 cluster. See http://blog.cloudera.com/blog/2015/12/new-in-cloudera-labs-apache-htrace-incubating/ Hopefully HTrace can "bridge the chasm" between being a developer tool, and being a trusted ops tool. We have some ways to go, but having these precompiled packages for CDH5.5 is a big step forward. (OK, I'll shut up about vendor stuff now...) Another really cool thing is that Sean Busbey and others in the YCSB community are working on integrating HTrace. YCSB is a very popular benchmark for big data / Hadoopy workloads. The github issue is here: https://github.com/brianfrankcooper/YCSB/issues/415 cheers, Colin
Re: git tag and branch naming
Sounds good. So: RC tags are named like 4.1.0RC1, 4.1.0RC2, etc. release tags are named like rel/4.1.0 branches are named like branch-4.1, branch-4.1.1, etc. cheers, Colin On Wed, Jan 27, 2016 at 10:45 AM, Sean Busbey wrote: > nope, the rel/ prefix is reserved for things that have been approved by a > PMC. RCs that require a tag should still stay in normal tags. > > On Wed, Jan 27, 2016 at 12:43 PM, Colin P. McCabe > wrote: > >> Thanks, Sean. That sounds like a good idea. I guess we can drop the >> "-release" suffix then. "rel/4.0" and "rel/4.0.1", etc. seem pretty >> self-explanatory. My main goal was just to make branches look >> different than tags. I would prefer to keep the "-branch" suffix on >> branches just to make that clear as well. >> >> Sean, would RCs also receive the "rel/" prefix, or not? I'm guessing >> not, since we don't need to preserve them forever. >> >> best, >> Colin >> >> >> On Tue, Jan 26, 2016 at 9:26 PM, Sean Busbey wrote: >> > with the new ASF release tag policy, this would make our release tags >> look >> > like 'rel/4.0-release' and 'rel/4.0.1-release'. >> > >> > the 'rel' prefix makes the distinction between branches and tagged >> releases >> > clear to me. what do others think? >> > >> > On Tue, Jan 26, 2016 at 10:41 PM, Masatake Iwasaki < >> > iwasak...@oss.nttdata.co.jp> wrote: >> > >> >> Sorry for late reply. >> >> >> >> I agree with the proposed naming conversion for branches and tags. >> >> If there is no objection further, we should close HTRACE-331 and >> >> prepare for the next release. >> >> >> >> Thanks, >> >> Masatake Iwasaki >> >> >> >> >> >> On 12/15/15 04:53, Colin P. McCabe wrote: >> >> >> >>> As part of our release process, we create git tags for each release >> >>> candidate (RC)... for example, 3.1.0RC9 and 4.0.1RC1. We also often >> >>> use release branches-- for example, the "4.0" branch. >> >>> >> >>> As Sean Busbey pointed out, we should also be creating "release" tags, >> >>> so that people who want to check out the release can do so without >> >>> having to figure out which RC was anointed as the release. I also >> >>> think we should adopt a naming convention for release branches and >> >>> tags so that people attempting to check out tags don't accidentally >> >>> check out branches, and vice versa. >> >>> >> >>> The branch and tag naming is confusing right now. For example, >> >>> someone running "git checkout 4.0" might be surprised to learn that >> >>> this checks out a branch currently containing 4.0.1, not the git tag >> >>> for the 4.0 release. >> >>> >> >>> I'm thinking we should adopt the following convention: >> >>> * release tags should have "release" in the name. So the tag for >> >>> htrace 4.1 should be "4.1-release" >> >>> * RC tags continue to be "4.1-RC1" and so forth. >> >>> * release branches should have "branch" in the name. So the branch for >> >>> 4.1 should be "branch-4.1". In general, branches should not include >> >>> "RC[0-9]" or "release" in the names, to avoid confusion with the tags. >> >>> >> >>> Let me know what you think. If you guys agree, I will also create >> >>> 4.0-release and 4.0.1-release tags corresponding to those releases. >> >>> >> >>> best, >> >>> Colin >> >>> >> >> >> >> >> > >> > >> > -- >> > Sean >> > > > > -- > Sean
Re: Time for a new release?
Ah, sorry. I did some preliminary work (like cutting the branch) but I haven't made the RC yet. I'll try to get it out there soon. best, Colin On Tue, Jan 26, 2016 at 8:47 PM, Masatake Iwasaki wrote: > Hi Colin, > > I can see branch-4.1 created in the repo. > Have you already started to create RC? > > Regards, > Masatake Iwasaki > > On 1/15/16 10:18, Masatake Iwasaki wrote: >>> >>> I think it's time to cut the 4.1 release. >> >> >> +1. >> >> I volunteer to do release management work, if you like. >> >> Thanks, >> Masatake Iwasaki >> >> >> On 1/14/16 07:52, Colin P. McCabe wrote: >>> >>> Hi all, >>> >>> Happy new year! >>> >>> I think it's time to cut the 4.1 release. We've fixed up a lot of >>> bugs, and made a lot of progress since 4.0.1. >>> >>> If everyone agrees, I'll post something in the next few days. >>> >>> best, >>> Colin >> >> >
Re: git tag and branch naming
nope, the rel/ prefix is reserved for things that have been approved by a PMC. RCs that require a tag should still stay in normal tags. On Wed, Jan 27, 2016 at 12:43 PM, Colin P. McCabe wrote: > Thanks, Sean. That sounds like a good idea. I guess we can drop the > "-release" suffix then. "rel/4.0" and "rel/4.0.1", etc. seem pretty > self-explanatory. My main goal was just to make branches look > different than tags. I would prefer to keep the "-branch" suffix on > branches just to make that clear as well. > > Sean, would RCs also receive the "rel/" prefix, or not? I'm guessing > not, since we don't need to preserve them forever. > > best, > Colin > > > On Tue, Jan 26, 2016 at 9:26 PM, Sean Busbey wrote: > > with the new ASF release tag policy, this would make our release tags > look > > like 'rel/4.0-release' and 'rel/4.0.1-release'. > > > > the 'rel' prefix makes the distinction between branches and tagged > releases > > clear to me. what do others think? > > > > On Tue, Jan 26, 2016 at 10:41 PM, Masatake Iwasaki < > > iwasak...@oss.nttdata.co.jp> wrote: > > > >> Sorry for late reply. > >> > >> I agree with the proposed naming conversion for branches and tags. > >> If there is no objection further, we should close HTRACE-331 and > >> prepare for the next release. > >> > >> Thanks, > >> Masatake Iwasaki > >> > >> > >> On 12/15/15 04:53, Colin P. McCabe wrote: > >> > >>> As part of our release process, we create git tags for each release > >>> candidate (RC)... for example, 3.1.0RC9 and 4.0.1RC1. We also often > >>> use release branches-- for example, the "4.0" branch. > >>> > >>> As Sean Busbey pointed out, we should also be creating "release" tags, > >>> so that people who want to check out the release can do so without > >>> having to figure out which RC was anointed as the release. I also > >>> think we should adopt a naming convention for release branches and > >>> tags so that people attempting to check out tags don't accidentally > >>> check out branches, and vice versa. > >>> > >>> The branch and tag naming is confusing right now. For example, > >>> someone running "git checkout 4.0" might be surprised to learn that > >>> this checks out a branch currently containing 4.0.1, not the git tag > >>> for the 4.0 release. > >>> > >>> I'm thinking we should adopt the following convention: > >>> * release tags should have "release" in the name. So the tag for > >>> htrace 4.1 should be "4.1-release" > >>> * RC tags continue to be "4.1-RC1" and so forth. > >>> * release branches should have "branch" in the name. So the branch for > >>> 4.1 should be "branch-4.1". In general, branches should not include > >>> "RC[0-9]" or "release" in the names, to avoid confusion with the tags. > >>> > >>> Let me know what you think. If you guys agree, I will also create > >>> 4.0-release and 4.0.1-release tags corresponding to those releases. > >>> > >>> best, > >>> Colin > >>> > >> > >> > > > > > > -- > > Sean > -- Sean
Re: git tag and branch naming
Thanks, Sean. That sounds like a good idea. I guess we can drop the "-release" suffix then. "rel/4.0" and "rel/4.0.1", etc. seem pretty self-explanatory. My main goal was just to make branches look different than tags. I would prefer to keep the "-branch" suffix on branches just to make that clear as well. Sean, would RCs also receive the "rel/" prefix, or not? I'm guessing not, since we don't need to preserve them forever. best, Colin On Tue, Jan 26, 2016 at 9:26 PM, Sean Busbey wrote: > with the new ASF release tag policy, this would make our release tags look > like 'rel/4.0-release' and 'rel/4.0.1-release'. > > the 'rel' prefix makes the distinction between branches and tagged releases > clear to me. what do others think? > > On Tue, Jan 26, 2016 at 10:41 PM, Masatake Iwasaki < > iwasak...@oss.nttdata.co.jp> wrote: > >> Sorry for late reply. >> >> I agree with the proposed naming conversion for branches and tags. >> If there is no objection further, we should close HTRACE-331 and >> prepare for the next release. >> >> Thanks, >> Masatake Iwasaki >> >> >> On 12/15/15 04:53, Colin P. McCabe wrote: >> >>> As part of our release process, we create git tags for each release >>> candidate (RC)... for example, 3.1.0RC9 and 4.0.1RC1. We also often >>> use release branches-- for example, the "4.0" branch. >>> >>> As Sean Busbey pointed out, we should also be creating "release" tags, >>> so that people who want to check out the release can do so without >>> having to figure out which RC was anointed as the release. I also >>> think we should adopt a naming convention for release branches and >>> tags so that people attempting to check out tags don't accidentally >>> check out branches, and vice versa. >>> >>> The branch and tag naming is confusing right now. For example, >>> someone running "git checkout 4.0" might be surprised to learn that >>> this checks out a branch currently containing 4.0.1, not the git tag >>> for the 4.0 release. >>> >>> I'm thinking we should adopt the following convention: >>> * release tags should have "release" in the name. So the tag for >>> htrace 4.1 should be "4.1-release" >>> * RC tags continue to be "4.1-RC1" and so forth. >>> * release branches should have "branch" in the name. So the branch for >>> 4.1 should be "branch-4.1". In general, branches should not include >>> "RC[0-9]" or "release" in the names, to avoid confusion with the tags. >>> >>> Let me know what you think. If you guys agree, I will also create >>> 4.0-release and 4.0.1-release tags corresponding to those releases. >>> >>> best, >>> Colin >>> >> >> > > > -- > Sean
