AngersZhuuuu commented on pull request #30957: URL: https://github.com/apache/spark/pull/30957#issuecomment-754173132
> > In spark now I just make it support default 7 level. > > The `7` looks a bit arbitrary to me. What's the behavior of hive? In hive, default separators's size is 8, and the first three one is `field delimit`, 'collection delimit', and 'keyvalue delimit' so when we use separator, it start with pos 1, means we can support 7 level nested complex struct. When we set `hive.serialization.extend.nesting.levels` to true, the separators size is 24, when we set `hive.serialization.extend.additional.nesting.levels` to true then the size will be 3 +4 + 1 + 13 + 14 + 128 you can see the code here https://github.com/apache/hive/blob/5171037313f0fc370f502f06c10c58766a6483d1/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySerDeParameters.java#L250-L299 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
