[
https://issues.apache.org/jira/browse/AVRO-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780734#action_12780734
]
Doug Cutting commented on AVRO-230:
-----------------------------------
> generate a random number and compare to the encoders stored value?
What I'm thinking of is that we provide somewhere a spec for a random data
generator. This can be based on, e.g., a simple 32-bit Linear Congruential
random number generator:
http://en.wikipedia.org/wiki/Linear_congruential_generator
This can be implemented in a few simple lines of code as the "raw" generator.
Since small values should often be more common, we can define a biased
generator by using the high bit (high bits are more random) of the next raw
random number as a coin. Toss it until false, counting the tosses, then use
that many high-order bits of the next raw random number as the biased value.
Then we specify for each kind of schema, how random data should be generated.
- int: take the next biased value. subtract it from 0 if the next raw
value's high-bit is one.
- long: multiply the next two biased values, then compute sign as above.
- float/double: divide the next two biased values, compute sign as above.
- bytes: use next biased value as length, up to, e.g., 64k, then fill with
high-byte of next raw value.
- string: use next biased value as length, up to, e.g., 64k, then fill with
randomly selected [a-z].
- union: select branch next_raw % branch_count
- etc.
The remaining question is how to seed the LCG generator. My instinct is not to
always start with a fixed seed, but rather should select a seed based on, e.g.,
the system time when a test file is created, and store that in each file's
metadata.
> Create a shared schema test directory structure
> -----------------------------------------------
>
> Key: AVRO-230
> URL: https://issues.apache.org/jira/browse/AVRO-230
> Project: Avro
> Issue Type: New Feature
> Components: c, c++, java, python
> Reporter: Matt Massie
> Assignee: Matt Massie
> Fix For: 1.3.0
>
>
> This is an example of my proposed directory structure:
> * invalid_schemas/
> ** broken.json
> ** wrong.json
> * valid_data/
> ** foo_test/
> *** schema.json
> *** json_data/
> **** valid_json_test_data.json
> **** more_valid_json_test_data.json
> *** binary_data/
> **** valid_binary_test_data.bin
> **** more_test_data.bin
> ** bar_test/
> *** schema.json
> *** json_data/
> **** ...
> * invalid_data/
> ** baz_test/
> *** schema.json
> *** json_data/
> **** ...
> *** binary_data/
> **** ...
> This structure supports positive and negative tests for avro schemas, json
> data and binary data.
> * The "invalid_schema" directory holds a number of invalid schemas that
> should fail to parse.
> * The "valid_data" directory has a number of self-contained tests in separate
> directories. Each test directory is required to have a "schema.json" file
> that valid avro schema. The "json_data" and "binary_data" directories are
> optional for each test.
> * The "invalid_data" directory has the same rules as the "valid_data"
> directory. The data files should fail during tests (negative testing).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.