jiangjiangtian commented on issue #10892: URL: https://github.com/apache/incubator-gluten/issues/10892#issuecomment-3410001769
> I am curious why the resizer occupies this much of memory. How complex the batch schema is? I try to output some statistics about the row vector and its children vectors: ``` estimated byte size 1998751236, numRows: 904 encoding: FLAT, isLazy: 0, type: INTEGER child 0 estimated byte size 4000 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 1 estimated byte size 57856 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 2 estimated byte size 28928 encoding: DICTIONARY, isLazy: 0, type: VARCHAR child 3 estimated byte size 45858 encoding: DICTIONARY, isLazy: 0, type: VARCHAR child 4 estimated byte size 105160 encoding: DICTIONARY, isLazy: 0, type: VARCHAR child 5 estimated byte size 31520 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 6 estimated byte size 25120 encoding: LAZY, isLazy: 1, type: MAP<VARCHAR,VARCHAR> isLoaded: 1, MAP child 7 estimated byte size 828928 encoding: LAZY, isLazy: 1, type: INTEGER isLoaded: 1, FLAT child 8 estimated byte size 4000 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 9 estimated byte size 26715 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 10 estimated byte size 33749 encoding: LAZY, isLazy: 1, type: ARRAY<VARCHAR> isLoaded: 1, ARRAY child 11 estimated byte size 121546 encoding: LAZY, isLazy: 1, type: MAP<VARCHAR,VARCHAR> isLoaded: 1, MAP child 12 estimated byte size 217793 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 13 estimated byte size 28113 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 14 estimated byte size 28928 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 15 estimated byte size 57448 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 16 estimated byte size 57856 encoding: LAZY, isLazy: 1, type: ARRAY<VARCHAR> isLoaded: 1, ARRAY child 17 estimated byte size 1171856976 encoding: LAZY, isLazy: 1, type: ARRAY<VARCHAR> isLoaded: 1, ARRAY child 18 estimated byte size 61148981 encoding: LAZY, isLazy: 1, type: ARRAY<VARCHAR> isLoaded: 1, ARRAY child 19 estimated byte size 195852983 encoding: LAZY, isLazy: 1, type: ARRAY<VARCHAR> isLoaded: 1, ARRAY child 20 estimated byte size 537728511 encoding: LAZY, isLazy: 1, type: ARRAY<VARCHAR> isLoaded: 1, ARRAY child 21 estimated byte size 5651640 encoding: LAZY, isLazy: 1, type: ARRAY<VARCHAR> isLoaded: 1, ARRAY child 22 estimated byte size 194352 encoding: LAZY, isLazy: 1, type: ARRAY<VARCHAR> isLoaded: 1, ARRAY child 23 estimated byte size 6800770 encoding: LAZY, isLazy: 1, type: ARRAY<VARCHAR> isLoaded: 1, ARRAY child 24 estimated byte size 136902 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 25 estimated byte size 57856 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 26 estimated byte size 57856 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 27 estimated byte size 33060 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 28 estimated byte size 33060 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 29 estimated byte size 24795 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 30 estimated byte size 28928 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 31 estimated byte size 22873 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 32 estimated byte size 33637 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 33 estimated byte size 19285 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 34 estimated byte size 25375 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 35 estimated byte size 31557 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 36 estimated byte size 36817 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 37 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 38 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 39 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 40 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 41 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 42 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 43 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 44 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 45 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 46 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 47 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 48 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 49 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 50 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 51 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 52 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 53 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 54 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 55 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 56 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 57 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 58 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 59 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 60 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 61 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 62 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 63 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: ARRAY<INTEGER> isLoaded: 1, ARRAY child 64 estimated byte size 61824 encoding: LAZY, isLazy: 1, type: VARCHAR isLoaded: 1, DICTIONARY child 65 estimated byte size 30305 encoding: LAZY, isLazy: 1, type: ARRAY<VARCHAR> isLoaded: 1, ARRAY child 66 estimated byte size 51605 encoding: LAZY, isLazy: 1, type: MAP<VARCHAR,VARCHAR> isLoaded: 1, MAP child 67 estimated byte size 15458474 encoding: CONSTANT, isLazy: 0, type: VARCHAR child 68 estimated byte size 16 encoding: CONSTANT, isLazy: 0, type: VARCHAR child 69 estimated byte size 16 encoding: CONSTANT, isLazy: 0, type: VARCHAR child 70 estimated byte size 16 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
