(Thanks for the [Discuss] tip.) Recursion is primarily a code-maintenance problem. I'm not sure how to quantify the complexity, but certainly Schema.java itself has a lot of logic in it to deal with recursion, as do all the "Grammar Generator" classes plus Generic Data -- the classes that are performance sensitive.
I don't think that recursion is inherently expensive performance-wise, but by making the encoding/resolution/decoding logic unnecessarily complicated, it makes it difficult to implement more aggressive strategies for higher-performance (e.g., dynamic code generation). On a similar note, the complexity of recursion could make it hard to add new features to Avro. I've yet to see compelling uses of recursion surface in this thread. Perhaps we deprecate recursion in 1.9, with the goal of eliminating it in 1.10? (Specifically, we write error messages to stderr when we parse recursive types -- with a flag to silence those message in case they get in someone's way.) If this deprecation creates howls of complaint because the feature is more useful than this thread seems to suggest, then we can keep it in. On Tue, Dec 4, 2018 at 9:25 AM Sean Busbey <bus...@cloudera.com.invalid> wrote: > > In the future please use "[DISCUSS]" at the start of your subject line > for these kinds of proposals. that'll get more folks to see the > discussion, e.g. when they filter this list. > > as a point of clarification, 1.9.0 is a major version change for the > Avro project. the "1" is a file format version. that's why API > incompatibilities are allowed in a new 1.y version. As Doug mentioned, > "would not be able to read binary files" means you're talking about a > file format incompatibility, which I don't think we should do. > > Are you interested in removing this feature to improve code > maintenance or to improve performance? both? > > Can you quantify the amount of complexity you're referring to? > On Fri, Nov 30, 2018 at 12:22 PM Raymie Stata <ray...@gmail.com> wrote: > > > > I understand we've been willing to introduce backward-incompatible API > > changes (not file-format changes) into minor release versions. If so, > > here's an idea for consideration: > > > > Let's eliminate recursive records from Avro 1.9.x. Recursion > > introduces a _lot_ of complexity into many parts of the Avro code > > base. We could vastly simplify the code base, and probably speed > > things up, by getting rid of this feature. > > > > The specific proposal would be that Avro 1.9.x would refuse to accept > > recursive records, and thus would not be able to read binary files > > written by older versions of Avro. I haven't heard of anyone actually > > using them, so I don't think this would be a problem. > > > > -- > busbey
