[ https://issues.apache.org/jira/browse/AVRO-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17508480#comment-17508480 ]
Kevin commented on AVRO-3451: ----------------------------- Ok, so testing the correct branch with [~Klamer] 's work I get most of the performance benefit that I see in my approach. One additional thing I did that measurably helped – surprisingly – was to use {{&str}} and not {{String}} for the HashMap key. Until the Resolved Schema can be reused across every encoding I think it would be a worthwhile to take the String clone-and-drop out of the tight loop that is the encoder in many applications. Note that I see about 80% of my performance gains when I use PR 1602's code in my application, so real-world it seems good enough for us and of course getting namespace to work correctly is important. On the other hand, I note that the Schema Write benchmarks that {{cargo bench}} runs are quite a bit more performant with my change. Part if that is changing the key from {{String}} to {{&str }}but there may be something else going on. To summarize: I think PR 1602 is the way to go. I'd suggest making the Hash Key a {{&str}} and call it done. Follow-on work to allow the Resolved Schema to be reused across calls to the encoder would likely be a good performance enhancement for many applications. > fix poor Avro write performance > ------------------------------- > > Key: AVRO-3451 > URL: https://issues.apache.org/jira/browse/AVRO-3451 > Project: Apache Avro > Issue Type: Improvement > Components: rust > Affects Versions: 1.11.0 > Environment: Mac OS X Big Sur > {code:java} > installed toolchains > -------------------- > stable-x86_64-apple-darwin (default) > nightly-x86_64-apple-darwin > active toolchain > ---------------- > stable-x86_64-apple-darwin (default) > rustc 1.56.1 (59eed8a2a 2021-11-01) {code} > Reporter: Kevin > Priority: Major > Attachments: Screen Shot 2022-03-14 at 7.30.24 PM.png > > Original Estimate: 1h > Remaining Estimate: 1h > > Rust implementation of Apache Avro library – apache-avro (née avro-rs) – > demonstrates poor write performance when serializing Rust structures to Avro. > Profiling indicates that this implementation spends an inordinate amount of > time in the function {{encode::encode_ref}} performing {{clone()}} and > {{drop}} operations related to a HashMap<String, Schema> type. > We modified the function {{encode_ref0}} as follows: > {code:java} > -pub fn encode_ref(value: &Value, schema: &Schema, buffer: &mut Vec<u8>) { > - fn encode_ref0( > +pub fn encode_ref<'a>(value: &Value, schema: &'a Schema, buffer: &mut > Vec<u8>) { > + fn encode_ref0<'a>( > value: &Value, > - schema: &Schema, > + schema: &'a Schema, > buffer: &mut Vec<u8>, > - schemas_by_name: &mut HashMap<String, Schema>, > + schemas_by_name: &mut HashMap<&'a str, &'a Schema>, > ) { > match &schema { > Schema::Ref { ref name } => { > - let resolved = > schemas_by_name.get(name.name.as_str()).unwrap(); > + let resolved = schemas_by_name.get(&name.name as > &str).unwrap(); > return encode_ref0(value, resolved, buffer, &mut > schemas_by_name.clone()); > } > Schema::Record { ref name, .. } > | Schema::Enum { ref name, .. } > | Schema::Fixed { ref name, .. } => { > - schemas_by_name.insert(name.name.clone(), schema.clone()); > + schemas_by_name.insert(&name.name, &schema); > } > _ => (), > }{code} > to remove any need for Clone in the {{schemas_by_name}} cache and see a > notable improvement (factor of 4 to 5) in our application with this change. > After this change, all Cargo Tests still pass and Benchmarks display a very > significant improvement in Write performance across the board. Attached below > is one example benchmark for {{big schema, write 10k records}} with Before on > the Left and After on the Right. -- This message was sent by Atlassian Jira (v8.20.1#820001)