[
https://issues.apache.org/jira/browse/AVRO-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712858#action_12712858
]
Scott Carey commented on AVRO-27:
---------------------------------
Performance results using COBSPerfTest on some JVM / OS / Hardware combinations.
First, an overview:
The 64 bit JRE on MacOS X has roughly similar performance characteristics in
these tests to the Linux Sun JRE 1.6.0_12. The Mac OSX 32 bit 1.5 JRE is
vastly different.
A 32 bit JVM is slightly faster than a 64 bit JVM on the byte-by-byte work,
roghly the same at 4 byte at a time work, and slower at 8 byte at a time work.
This is mostly expected.
Variations in VM from Sun 1.6.0_12 through a few early access versions of
1.6.0_14 have roghly the same performance. That is, the performance
improvements in the latest JRE (of which, there are many) don't seem to have an
impact here.
Larger byte chunks help decoding only a little unless zero words dominate, and
then it helps a lot.
Larger chunks help encoding significantly across the board. COLS -- working
with 8 byte chunks -- is about 4x faster than COBS.
The results below could use some formatting work -- it is very verbose.
Al results with Centos 5.3
Xeon 5335 is 2.0Ghz, 4MB cache per pair of cores, 2x quad core
Xeon E5440 is 2.83Ghz, 6MB cache per pair of cores, 2x quad core
Results have the following headers if you wish to search:
* 1.6.0_13 Xeon 5335 defaults
* 1.6.0_14b03 Xeon 5335 defaults
* 1.6.0_14b03 Xeon 5335 compressed pointers, escape analysis
* 1.6.0_14b06 32 bit Xeon 5335 defaults
* 1.6.0_12 Xeon E5440 defaults
Results:
* 1.6.0_13 Xeon 5335 defaults
$ /usr/java/jdk1.6.0_13/bin/java -server -Xmx512m -jar COBSPerfTest.jar
COBSCodec, one zero word every 1 words
Encoding at 89.13659 MB/sec
Decoding at 57.16702 MB/sec
COBSCodec, one zero word every 10 words
Encoding at 69.81948 MB/sec
Decoding at 208.78065 MB/sec
COBSCodec, one zero word every 100 words
Encoding at 144.56085 MB/sec
Decoding at 925.55365 MB/sec
COBSCodec, one zero word every 1000 words
Encoding at 155.95493 MB/sec
Decoding at 1033.0511 MB/sec
COBSCodec, one zero word every 10000 words
Encoding at 157.28098 MB/sec
Decoding at 1038.535 MB/sec
COWSCodec variant 1, one zero word every 1 words
Encoding at 248.32587 MB/sec
Decoding at 272.1762 MB/sec
COWSCodec variant 1, one zero word every 10 words
Encoding at 158.03842 MB/sec
Decoding at 244.39633 MB/sec
COWSCodec variant 1, one zero word every 100 words
Encoding at 178.13342 MB/sec
Decoding at 314.68652 MB/sec
COWSCodec variant 1, one zero word every 1000 words
Encoding at 179.34563 MB/sec
Decoding at 319.3074 MB/sec
COWSCodec variant 1, one zero word every 10000 words
Encoding at 179.1614 MB/sec
Decoding at 317.2071 MB/sec
COWSCodec variant 1, one zero word every 100000 words
Encoding at 179.04832 MB/sec
Decoding at 318.77673 MB/sec
COWSCodec variant 2, one zero word every 1 words
Encoding at 212.54866 MB/sec
Decoding at 239.11534 MB/sec
COWSCodec variant 2, one zero word every 10 words
Encoding at 225.34329 MB/sec
Decoding at 466.0846 MB/sec
COWSCodec variant 2, one zero word every 100 words
Encoding at 323.6535 MB/sec
Decoding at 1133.9556 MB/sec
COWSCodec variant 2, one zero word every 1000 words
Encoding at 329.2162 MB/sec
Decoding at 1204.8298 MB/sec
COWSCodec variant 2, one zero word every 10000 words
Encoding at 328.59866 MB/sec
Decoding at 1212.3876 MB/sec
COWSCodec variant 2, one zero word every 100000 words
Encoding at 328.2159 MB/sec
Decoding at 1205.3972 MB/sec
COWSCodec variant 3, one zero word every 1 words
Encoding at 224.24924 MB/sec
Decoding at 252.21263 MB/sec
COWSCodec variant 3, one zero word every 10 words
Encoding at 235.53137 MB/sec
Decoding at 506.0203 MB/sec
COWSCodec variant 3, one zero word every 100 words
Encoding at 320.96054 MB/sec
Decoding at 1094.6545 MB/sec
COWSCodec variant 3, one zero word every 1000 words
Encoding at 328.45444 MB/sec
Decoding at 1213.4741 MB/sec
COWSCodec variant 3, one zero word every 10000 words
Encoding at 328.3331 MB/sec
Decoding at 1231.2334 MB/sec
COWSCodec variant 3, one zero word every 100000 words
Encoding at 328.1387 MB/sec
Decoding at 1217.2268 MB/sec
COLSCodec, one zero word every 1 words
Encoding at 291.29678 MB/sec
Decoding at 343.89276 MB/sec
COLSCodec, one zero word every 10 words
Encoding at 354.09015 MB/sec
Decoding at 812.4928 MB/sec
Original array was modified!
COLSCodec, one zero word every 100 words
Encoding at 433.4998 MB/sec
Decoding at 1204.3855 MB/sec
COLSCodec, one zero word every 1000 words
Encoding at 423.8553 MB/sec
Decoding at 1237.3381 MB/sec
COLSCodec, one zero word every 10000 words
Encoding at 421.88364 MB/sec
Decoding at 1238.6761 MB/sec
COLSCodec, one zero word every 100000 words
Encoding at 419.33118 MB/sec
Decoding at 1239.0199 MB/sec
COLSCodec, one zero word every 1000000 words
Encoding at 420.57434 MB/sec
Decoding at 1218.9686 MB/sec
* 1.6.0_14b03 Xeon 5335 defaults
$ java -server -Xmx512m -jar COBSPerfTest.jar
COBSCodec, one zero word every 1 words
Encoding at 91.95477 MB/sec
Decoding at 57.01143 MB/sec
COBSCodec, one zero word every 10 words
Encoding at 73.45933 MB/sec
Decoding at 207.67142 MB/sec
COBSCodec, one zero word every 100 words
Encoding at 144.0236 MB/sec
Decoding at 913.1517 MB/sec
COBSCodec, one zero word every 1000 words
Encoding at 155.32053 MB/sec
Decoding at 1032.9912 MB/sec
COBSCodec, one zero word every 10000 words
Encoding at 156.60835 MB/sec
Decoding at 1024.763 MB/sec
COWSCodec variant 1, one zero word every 1 words
Encoding at 271.9177 MB/sec
Decoding at 276.57822 MB/sec
COWSCodec variant 1, one zero word every 10 words
Encoding at 152.21716 MB/sec
Decoding at 191.38951 MB/sec
COWSCodec variant 1, one zero word every 100 words
Encoding at 171.42383 MB/sec
Decoding at 224.81892 MB/sec
COWSCodec variant 1, one zero word every 1000 words
Encoding at 173.3674 MB/sec
Decoding at 228.82373 MB/sec
COWSCodec variant 1, one zero word every 10000 words
Encoding at 173.56622 MB/sec
Decoding at 228.35956 MB/sec
COWSCodec variant 1, one zero word every 100000 words
Encoding at 173.60176 MB/sec
Decoding at 229.12196 MB/sec
COWSCodec variant 2, one zero word every 1 words
Encoding at 214.48987 MB/sec
Decoding at 241.93507 MB/sec
COWSCodec variant 2, one zero word every 10 words
Encoding at 244.36378 MB/sec
Decoding at 473.0567 MB/sec
COWSCodec variant 2, one zero word every 100 words
Encoding at 345.88748 MB/sec
Decoding at 1003.376 MB/sec
COWSCodec variant 2, one zero word every 1000 words
Encoding at 349.1008 MB/sec
Decoding at 1026.7786 MB/sec
COWSCodec variant 2, one zero word every 10000 words
Encoding at 347.61612 MB/sec
Decoding at 1028.8761 MB/sec
COWSCodec variant 2, one zero word every 100000 words
Encoding at 346.9563 MB/sec
Decoding at 1061.9762 MB/sec
COWSCodec variant 3, one zero word every 1 words
Encoding at 210.84114 MB/sec
Decoding at 258.27982 MB/sec
COWSCodec variant 3, one zero word every 10 words
Encoding at 252.59242 MB/sec
Decoding at 507.0884 MB/sec
COWSCodec variant 3, one zero word every 100 words
Encoding at 353.3254 MB/sec
Decoding at 1150.1593 MB/sec
COWSCodec variant 3, one zero word every 1000 words
Encoding at 358.27298 MB/sec
Decoding at 1208.8944 MB/sec
COWSCodec variant 3, one zero word every 10000 words
Encoding at 357.32245 MB/sec
Decoding at 1215.5607 MB/sec
COWSCodec variant 3, one zero word every 100000 words
Encoding at 356.93134 MB/sec
Decoding at 1210.7133 MB/sec
COLSCodec, one zero word every 1 words
Encoding at 287.6796 MB/sec
Decoding at 362.4284 MB/sec
COLSCodec, one zero word every 10 words
Encoding at 349.48486 MB/sec
Decoding at 817.0665 MB/sec
Original array was modified!
COLSCodec, one zero word every 100 words
Encoding at 418.7336 MB/sec
Decoding at 1214.5057 MB/sec
COLSCodec, one zero word every 1000 words
Encoding at 410.76407 MB/sec
Decoding at 1239.2533 MB/sec
COLSCodec, one zero word every 10000 words
Encoding at 408.02432 MB/sec
Decoding at 1245.9232 MB/sec
COLSCodec, one zero word every 100000 words
Encoding at 406.2959 MB/sec
Decoding at 1252.01 MB/sec
COLSCodec, one zero word every 1000000 words
Encoding at 405.99057 MB/sec
Decoding at 1252.2338 MB/sec
* 1.6.0_14b03 Xeon 5335 compressed pointers, escape analysis
[cand...@britney COBSPerfTest]$ java -server -Xmx512m -XX:+DoEscapeAnalysis
-XX:+UseCompressedOops -jar COBSPerfTest.jar
COBSCodec, one zero word every 1 words
Encoding at 91.98761 MB/sec
Decoding at 53.635868 MB/sec
COBSCodec, one zero word every 10 words
Encoding at 72.98973 MB/sec
Decoding at 205.35959 MB/sec
COBSCodec, one zero word every 100 words
Encoding at 144.04861 MB/sec
Decoding at 918.5997 MB/sec
COBSCodec, one zero word every 1000 words
Encoding at 154.981 MB/sec
Decoding at 1018.9709 MB/sec
COBSCodec, one zero word every 10000 words
Encoding at 156.41275 MB/sec
Decoding at 1032.3058 MB/sec
COWSCodec variant 1, one zero word every 1 words
Encoding at 252.68245 MB/sec
Decoding at 307.39664 MB/sec
COWSCodec variant 1, one zero word every 10 words
Encoding at 163.27182 MB/sec
Decoding at 209.55176 MB/sec
COWSCodec variant 1, one zero word every 100 words
Encoding at 189.6774 MB/sec
Decoding at 263.66977 MB/sec
COWSCodec variant 1, one zero word every 1000 words
Encoding at 193.37485 MB/sec
Decoding at 270.99658 MB/sec
COWSCodec variant 1, one zero word every 10000 words
Encoding at 193.74573 MB/sec
Decoding at 271.46988 MB/sec
COWSCodec variant 1, one zero word every 100000 words
Encoding at 194.11456 MB/sec
Decoding at 270.73804 MB/sec
COWSCodec variant 2, one zero word every 1 words
Encoding at 216.82019 MB/sec
Decoding at 243.21117 MB/sec
COWSCodec variant 2, one zero word every 10 words
Encoding at 242.51544 MB/sec
Decoding at 465.20282 MB/sec
COWSCodec variant 2, one zero word every 100 words
Encoding at 344.99945 MB/sec
Decoding at 1157.6014 MB/sec
COWSCodec variant 2, one zero word every 1000 words
Encoding at 351.1931 MB/sec
Decoding at 1211.4054 MB/sec
COWSCodec variant 2, one zero word every 10000 words
Encoding at 349.90894 MB/sec
Decoding at 1217.9989 MB/sec
COWSCodec variant 2, one zero word every 100000 words
Encoding at 349.40396 MB/sec
Decoding at 1210.6339 MB/sec
COWSCodec variant 3, one zero word every 1 words
Encoding at 240.06367 MB/sec
Decoding at 228.17952 MB/sec
COWSCodec variant 3, one zero word every 10 words
Encoding at 255.28317 MB/sec
Decoding at 496.779 MB/sec
COWSCodec variant 3, one zero word every 100 words
Encoding at 360.55945 MB/sec
Decoding at 1142.717 MB/sec
COWSCodec variant 3, one zero word every 1000 words
Encoding at 365.1012 MB/sec
Decoding at 1205.8257 MB/sec
COWSCodec variant 3, one zero word every 10000 words
Encoding at 363.70743 MB/sec
Decoding at 1213.5723 MB/sec
COWSCodec variant 3, one zero word every 100000 words
Encoding at 363.2405 MB/sec
Decoding at 1208.7316 MB/sec
COLSCodec, one zero word every 1 words
Encoding at 298.33194 MB/sec
Decoding at 318.14648 MB/sec
COLSCodec, one zero word every 10 words
Encoding at 368.6357 MB/sec
Decoding at 825.8583 MB/sec
Original array was modified!
COLSCodec, one zero word every 100 words
Encoding at 449.0997 MB/sec
Decoding at 1191.9662 MB/sec
COLSCodec, one zero word every 1000 words
Encoding at 441.75586 MB/sec
Decoding at 1223.806 MB/sec
COLSCodec, one zero word every 10000 words
Encoding at 439.18317 MB/sec
Decoding at 1227.0127 MB/sec
COLSCodec, one zero word every 100000 words
Encoding at 438.62714 MB/sec
Decoding at 1224.557 MB/sec
COLSCodec, one zero word every 1000000 words
Encoding at 438.62115 MB/sec
Decoding at 1224.6772 MB/sec
* 1.6.0_14b06 32 bit Xeon 5335 defaults
$ /usr/java/jdk1.6.0_14ea6_32bit/bin/java -server -Xmx512m -jar
COBSPerfTest.jar COBSCodec, one zero word every 1 words
Encoding at 101.488785 MB/sec
Decoding at 44.9381 MB/sec
COBSCodec, one zero word every 10 words
Encoding at 76.98102 MB/sec
Decoding at 186.26143 MB/sec
COBSCodec, one zero word every 100 words
Encoding at 154.48914 MB/sec
Decoding at 926.46204 MB/sec
COBSCodec, one zero word every 1000 words
Encoding at 169.65015 MB/sec
Decoding at 996.02625 MB/sec
COBSCodec, one zero word every 10000 words
Encoding at 171.83167 MB/sec
Decoding at 1069.7236 MB/sec
COWSCodec variant 1, one zero word every 1 words
Encoding at 229.62816 MB/sec
Decoding at 347.4478 MB/sec
COWSCodec variant 1, one zero word every 10 words
Encoding at 137.4511 MB/sec
Decoding at 181.96013 MB/sec
COWSCodec variant 1, one zero word every 100 words
Encoding at 170.84563 MB/sec
Decoding at 246.61394 MB/sec
COWSCodec variant 1, one zero word every 1000 words
Encoding at 175.59972 MB/sec
Decoding at 255.19583 MB/sec
COWSCodec variant 1, one zero word every 10000 words
Encoding at 176.94963 MB/sec
Decoding at 257.768 MB/sec
COWSCodec variant 1, one zero word every 100000 words
Encoding at 175.58342 MB/sec
Decoding at 255.8668 MB/sec
COWSCodec variant 2, one zero word every 1 words
Encoding at 212.1405 MB/sec
Decoding at 257.78635 MB/sec
COWSCodec variant 2, one zero word every 10 words
Encoding at 231.08081 MB/sec
Decoding at 421.9081 MB/sec
COWSCodec variant 2, one zero word every 100 words
Encoding at 348.02103 MB/sec
Decoding at 1133.5847 MB/sec
COWSCodec variant 2, one zero word every 1000 words
Encoding at 358.29077 MB/sec
Decoding at 1170.7545 MB/sec
COWSCodec variant 2, one zero word every 10000 words
Encoding at 360.5535 MB/sec
Decoding at 1223.9012 MB/sec
COWSCodec variant 2, one zero word every 100000 words
Encoding at 358.03394 MB/sec
Decoding at 1216.9368 MB/sec
COWSCodec variant 3, one zero word every 1 words
Encoding at 226.55222 MB/sec
Decoding at 275.24838 MB/sec
COWSCodec variant 3, one zero word every 10 words
Encoding at 243.09453 MB/sec
Decoding at 469.97775 MB/sec
COWSCodec variant 3, one zero word every 100 words
Encoding at 351.21555 MB/sec
Decoding at 1129.5447 MB/sec
COWSCodec variant 3, one zero word every 1000 words
Encoding at 358.14252 MB/sec
Decoding at 1196.7433 MB/sec
COWSCodec variant 3, one zero word every 10000 words
Encoding at 360.71323 MB/sec
Decoding at 1199.4408 MB/sec
COWSCodec variant 3, one zero word every 100000 words
Encoding at 358.2802 MB/sec
Decoding at 1224.6678 MB/sec
COLSCodec, one zero word every 1 words
Encoding at 208.82603 MB/sec
Decoding at 275.9128 MB/sec
COLSCodec, one zero word every 10 words
Encoding at 265.03033 MB/sec
Decoding at 730.78546 MB/sec
Original array was modified!
COLSCodec, one zero word every 100 words
Encoding at 310.9054 MB/sec
Decoding at 1157.1534 MB/sec
COLSCodec, one zero word every 1000 words
Encoding at 308.7317 MB/sec
Decoding at 1238.4891 MB/sec
COLSCodec, one zero word every 10000 words
Encoding at 306.90793 MB/sec
Decoding at 1220.6907 MB/sec
COLSCodec, one zero word every 100000 words
Encoding at 305.49704 MB/sec
Decoding at 1205.0568 MB/sec
COLSCodec, one zero word every 1000000 words
Encoding at 305.3674 MB/sec
Decoding at 1234.8855 MB/sec
* 1.6.0_12 Xeon E5440 defaults
$ java -server -Xmx512m -jar COBSPerfTest.jar
COBSCodec, one zero word every 1 words
Encoding at 124.19903 MB/sec
Decoding at 80.51218 MB/sec
COBSCodec, one zero word every 10 words
Encoding at 97.80887 MB/sec
Decoding at 293.82983 MB/sec
COBSCodec, one zero word every 100 words
Encoding at 203.51627 MB/sec
Decoding at 1299.7317 MB/sec
COBSCodec, one zero word every 1000 words
Encoding at 219.41322 MB/sec
Decoding at 1422.3486 MB/sec
COBSCodec, one zero word every 10000 words
Encoding at 220.89801 MB/sec
Decoding at 1420.3978 MB/sec
COWSCodec variant 1, one zero word every 1 words
Encoding at 344.6565 MB/sec
Decoding at 390.2233 MB/sec
COWSCodec variant 1, one zero word every 10 words
Encoding at 220.47774 MB/sec
Decoding at 360.3579 MB/sec
COWSCodec variant 1, one zero word every 100 words
Encoding at 250.53049 MB/sec
Decoding at 447.04602 MB/sec
COWSCodec variant 1, one zero word every 1000 words
Encoding at 253.66922 MB/sec
Decoding at 450.34372 MB/sec
COWSCodec variant 1, one zero word every 10000 words
Encoding at 253.64081 MB/sec
Decoding at 447.10074 MB/sec
COWSCodec variant 1, one zero word every 100000 words
Encoding at 252.63756 MB/sec
Decoding at 447.50485 MB/sec
COWSCodec variant 2, one zero word every 1 words
Encoding at 275.47418 MB/sec
Decoding at 332.2978 MB/sec
COWSCodec variant 2, one zero word every 10 words
Encoding at 316.82657 MB/sec
Decoding at 657.1525 MB/sec
COWSCodec variant 2, one zero word every 100 words
Encoding at 449.77597 MB/sec
Decoding at 1545.4358 MB/sec
COWSCodec variant 2, one zero word every 1000 words
Encoding at 457.52542 MB/sec
Decoding at 1653.704 MB/sec
COWSCodec variant 2, one zero word every 10000 words
Encoding at 456.66467 MB/sec
Decoding at 1658.9537 MB/sec
COWSCodec variant 2, one zero word every 100000 words
Encoding at 455.9669 MB/sec
Decoding at 1655.1809 MB/sec
COWSCodec variant 3, one zero word every 1 words
Encoding at 315.7178 MB/sec
Decoding at 360.02884 MB/sec
COWSCodec variant 3, one zero word every 10 words
Encoding at 331.1007 MB/sec
Decoding at 723.18024 MB/sec
COWSCodec variant 3, one zero word every 100 words
Encoding at 443.8783 MB/sec
Decoding at 1560.0219 MB/sec
COWSCodec variant 3, one zero word every 1000 words
Encoding at 447.92645 MB/sec
Decoding at 1541.4951 MB/sec
COWSCodec variant 3, one zero word every 10000 words
Encoding at 449.71402 MB/sec
Decoding at 1394.1431 MB/sec
COWSCodec variant 3, one zero word every 100000 words
Encoding at 441.31396 MB/sec
Decoding at 1361.6113 MB/sec
COLSCodec, one zero word every 1 words
Encoding at 405.91794 MB/sec
Decoding at 482.06976 MB/sec
COLSCodec, one zero word every 10 words
Encoding at 491.71738 MB/sec
Decoding at 1079.3405 MB/sec
Original array was modified!
COLSCodec, one zero word every 100 words
Encoding at 598.31836 MB/sec
Decoding at 1616.031 MB/sec
COLSCodec, one zero word every 1000 words
Encoding at 586.9973 MB/sec
Decoding at 1666.9445 MB/sec
COLSCodec, one zero word every 10000 words
Encoding at 585.7841 MB/sec
Decoding at 1674.5248 MB/sec
COLSCodec, one zero word every 100000 words
Encoding at 585.28375 MB/sec
Decoding at 1664.8573 MB/sec
COLSCodec, one zero word every 1000000 words
Encoding at 585.0993 MB/sec
Decoding at 1662.1304 MB/sec
> Consistent Overhead Byte Stuffing (COBS) encoded block format for Object
> Container Files
> ----------------------------------------------------------------------------------------
>
> Key: AVRO-27
> URL: https://issues.apache.org/jira/browse/AVRO-27
> Project: Avro
> Issue Type: New Feature
> Components: spec
> Reporter: Matt Massie
> Attachments: COBSCodec.java, COBSCodec2.java, COBSPerfTest.java,
> COLSCodec.java, COWSCodec.java, COWSCodec2.java, COWSCodec3.java
>
>
> Object Container Files could use a 1 byte sync marker (set to zero) using
> zig-zag and COBS encoding within blocks to efficiently escape zeros from the
> record data.
> h4. Zig-Zag encoding
> With zig-zag encoding only the value of 0 (zero) gets encoded into a value
> with a single zero byte. This property means that we can write any non-zero
> zig-zag long inside a block within concern for creating an unintentional sync
> byte.
> h4. COBS encoding
> We'll use COBS encoding to ensure that all zeros are escaped inside the block
> payload. You can read http://www.sigcomm.org/sigcomm97/papers/p062.pdf for
> the details about COBS encoding.
> h1. Block Format
> All blocks start and end with a sync byte (set to zero) with a
> type-length-value format internally as follows:
> || name || format || length in bytes || value || meaning ||
> | sync | byte | 1 | always 0 (zero) | The sync byte serves as a clear marker
> for the start of a block |
> | type | zig-zag long | variable | must be non-zero | The type field
> expresses whether the block is for _metadata_ or _normal_ data. |
> | length | zig-zag long | variable | must be non-zero | The length field
> expresses the number of bytes until the next record (including the cobs code
> and sync byte). Useful for skipping ahead to the next block. |
> | cobs_code | byte | 1 | see COBS code table below | Used in escaping zeros
> from the block payload |
> | payload | cobs-encoded | Greater than or equal to zero | all non-zero bytes
> | The payload of the block |
> | sync | byte | 1 | always 0 (zero) | The sync byte serves as a clear marker
> for the end of the block |
> h2. COBS code table
> || Code || Followed by || Meaning |
> | 0x00 | (not applicable) | (not allowed ) |
> | 0x01 | nothing | Empty payload followed by the closing sync byte |
> | 0x02 | one data byte | The single data byte, followed by the closing sync
> byte |
> | 0x03 | two data bytes | The pair of data bytes, followed by the closing
> sync byte |
> | 0x04 | three data bytes | The three data bytes, followed by the closing
> sync byte |
> | n | (n-1) data bytes | The (n-1) data bytes, followed by the closing sync
> byte |
> | 0xFD | 252 data bytes | The 252 data bytes, followed by the closing sync
> byte |
> | 0xFE | 253 data bytes | The 253 data bytes, followed by the closing sync
> byte |
> | 0xFF | 254 data bytes | The 254 data bytes *not* followed by a zero. |
> (taken from http://www.sigcomm.org/sigcomm97/papers/p062.pdf)
> h1. Encoding
> Only the block writer needs to perform byte-by-byte processing to encode the
> block. The overhead for COBS encoding is very small in terms of the
> in-memory state required.
> h1. Decoding
> Block readers are not required to do as much byte-by-byte processing as a
> writer. The reader could (for example) find a _metadata_ block by doing the
> following:
> # Search for a zero byte in the file which marks the start of a record
> # Read and zig-zag decode the _type_ of the block
> #* If the block is _normal_ data, read the _length_, seek ahead to the next
> block and goto step #2 again
> #* If the block is a _metadata_ block, cobs decode the data
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
