[ 
https://issues.apache.org/jira/browse/AVRO-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780734#action_12780734
 ] 

Doug Cutting commented on AVRO-230:
-----------------------------------

> generate a random number and compare to the encoders stored value?

What I'm thinking of is that we provide somewhere a spec for a random data 
generator.  This can be based on, e.g., a simple 32-bit Linear Congruential 
random number generator:

http://en.wikipedia.org/wiki/Linear_congruential_generator

This can be implemented in a few simple lines of code as the "raw" generator.  
Since small values should often be more common, we can define a biased 
generator by using the high bit (high bits are more random) of the next raw 
random number as a coin.  Toss it until false, counting the tosses, then use 
that many high-order bits of the next raw random number as the biased value.

Then we specify for each kind of schema, how random data should be generated.
  - int: take the next biased value.  subtract it from 0 if the next raw 
value's high-bit is one.
  - long: multiply the next two biased values, then compute sign as above.
  - float/double: divide the next two biased values, compute sign as above.
  - bytes: use next biased value as length, up to, e.g., 64k, then fill with 
high-byte of next raw value.
  - string: use next biased value as length, up to, e.g., 64k, then fill with 
randomly selected [a-z].
  - union: select branch next_raw % branch_count
  - etc.

The remaining question is how to seed the LCG generator.  My instinct is not to 
always start with a fixed seed, but rather should select a seed based on, e.g., 
the system time when a test file is created, and store that in each file's 
metadata.


> Create a shared schema test directory structure
> -----------------------------------------------
>
>                 Key: AVRO-230
>                 URL: https://issues.apache.org/jira/browse/AVRO-230
>             Project: Avro
>          Issue Type: New Feature
>          Components: c, c++, java, python
>            Reporter: Matt Massie
>            Assignee: Matt Massie
>             Fix For: 1.3.0
>
>
> This is an example of my proposed directory structure:
> * invalid_schemas/
> ** broken.json
> ** wrong.json
> * valid_data/
> ** foo_test/
> *** schema.json
> *** json_data/
> **** valid_json_test_data.json
> **** more_valid_json_test_data.json
> *** binary_data/
> **** valid_binary_test_data.bin
> **** more_test_data.bin
> ** bar_test/
> *** schema.json
> *** json_data/
> **** ...
> * invalid_data/
> ** baz_test/
> *** schema.json
> *** json_data/
> **** ...
> *** binary_data/
> **** ...
> This structure supports positive and negative tests for avro schemas, json 
> data and binary data.
> * The "invalid_schema" directory holds a number of invalid schemas that 
> should fail to parse.
> * The "valid_data" directory has a number of self-contained tests in separate 
> directories.  Each test directory is required to have a "schema.json" file 
> that valid avro schema.  The "json_data" and "binary_data" directories are 
> optional for each test.
> * The "invalid_data" directory has the same rules as the "valid_data" 
> directory.  The data files should fail during tests (negative testing).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to