Hi Parquet Developers,
Am using Parquet for analytics on AWS EC2. A INSERT OVERWRITE from one
small hive to another tables works fine with parquet with snappy. But when
the file is having 1.2+ Billion records, the insert overwrite fails due to
Java Heap issue.
Few other stats about the data :
Parquet version =1.2.5(CDH5.1.3)
Total files in the HDFS directory : 555
Attached PrintFooter Parquet.txt with the PrintFooter Information. (Note:-
Changes the column names to random characters)
HDFS block size : 256MB
*Yarn configurations :*
yarn.nodemanager.resource.memory-mb = 24GB
yarn.scheduler.minimum-allocation-mb = 2GB
mapreduce.map.memory.mb = 6GB
mapreduce.reduce.memory.mb = 12GB
mapreduce.map.java.opts = 4.5GB
mapreduce.reduce.java.opts = 9GB
yarn.nodemanager.vmem-pmem-ratio = 2.1
Can anyone help me debug or help me fix this issue ? Is this fixed in any
newer Parquet version (>v1.2.5)?
Let me know in case if some one needs more information.
Pasting the task attempt log below :
2014-10-27 13:10:41,556 FATAL [main]
org.apache.hadoop.mapred.YarnChild: Error running child :
java.lang.OutOfMemoryError: Java heap space
at parquet.column.values.dictionary.IntList.initSlab(IntList.java:87)
at parquet.column.values.dictionary.IntList.<init>(IntList.java:83)
at
parquet.column.values.dictionary.DictionaryValuesWriter.<init>(DictionaryValuesWriter.java:85)
at
parquet.column.values.dictionary.DictionaryValuesWriter$PlainDoubleDictionaryValuesWriter.<init>(DictionaryValuesWriter.java:471)
at parquet.column.impl.ColumnWriterImpl.<init>(ColumnWriterImpl.java:91)
at
parquet.column.impl.ColumnWriteStoreImpl.newMemColumn(ColumnWriteStoreImpl.java:60)
at
parquet.column.impl.ColumnWriteStoreImpl.getColumnWriter(ColumnWriteStoreImpl.java:52)
at
parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.<init>(MessageColumnIO.java:98)
at parquet.io.MessageColumnIO.getRecordWriter(MessageColumnIO.java:283)
at
parquet.hadoop.InternalParquetRecordWriter.initStore(InternalParquetRecordWriter.java:102)
at
parquet.hadoop.InternalParquetRecordWriter.<init>(InternalParquetRecordWriter.java:88)
at
parquet.hadoop.ParquetRecordWriter.<init>(ParquetRecordWriter.java:61)
at
parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:267)
at
parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:245)
at
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.<init>(ParquetRecordWriterWrapper.java:54)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getParquerRecordWriterWrapper(MapredParquetOutputFormat.java:122)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getHiveRecordWriter(MapredParquetOutputFormat.java:113)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:296)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:283)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:516)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createNewPaths(FileSinkOperator.java:692)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:764)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:609)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:501)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:501)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:501)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519)
--
Thanks
Suraj Nayak M
[abc] BINARY 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min: 144
max: 1.493K average: 933 total: 758.81K (raw data: 855.77K saving 11%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 136 max: 1.941K average: 1.052K total: 855.77K
column values statistics: no stats for this column
[def] BINARY 7.7% of all space [PLAIN, RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
506.306K max: 7.931M average: 4.659M total: 3.788G (raw data: 24.664G saving
84%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 1.149M max: 52.453M average: 30.337M total: 24.664G
column values statistics: no stats for this column
[ddeeedfv] INT64 0.7% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
51.404K max: 822.797K average: 478.413K total: 388.949M (raw data: 443.303M
saving 12%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 60.986K max: 926.346K average: 545.269K total: 443.303M
column values statistics: no stats for this column
[vdfcvd] BINARY 0.7% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
52.119K max: 824.66K average: 479.82K total: 390.094M (raw data: 444.737M
saving 12%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 62.503K max: 927.963K average: 547.032K total: 444.737M
column values statistics: no stats for this column
[dfvdfvdf] INT32 1.2% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
64.652K max: 1.37M average: 778.881K total: 633.23M (raw data: 796.524M saving
20%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 76.04K max: 1.93M average: 979.734K total: 796.524M
column values statistics: no stats for this column
[fvdfvdfv] INT32 0.6% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
38.063K max: 648.005K average: 383.759K total: 311.996M (raw data: 317.203M
saving 1%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 38.607K max: 656.965K average: 390.164K total: 317.203M
column values statistics: no stats for this column
[dvdfvdfvdwa] INT32 3.5% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
195.245K max: 3.587M average: 2.135M total: 1.736G (raw data: 1.744G saving 0%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 195.209K max: 3.586M average: 2.145M total: 1.744G
column values statistics: no stats for this column
[gfbvdfvd] BINARY 0.5% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
33.471K max: 598.523K average: 346.761K total: 281.917M (raw data: 410.681M
saving 31%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 48.958K max: 860.85K average: 505.143K total: 410.681M
column values statistics: no stats for this column
[saferdfsd] INT32 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min: 52
max: 349 average: 224 total: 182.878K (raw data: 171.188K saving -6%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 48 max: 327 average: 210 total: 171.188K
column values statistics: no stats for this column
[sdcsdfcsdf] BINARY 0.6% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
37.462K max: 639.083K average: 377.339K total: 306.777M (raw data: 317.441M
saving 3%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 38.688K max: 657.738K average: 390.456K total: 317.441M
column values statistics: no stats for this column
[scsdfvcsdf] INT32 1.1% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
67.301K max: 1.268M average: 701.107K total: 570.0M (raw data: 897.724M saving
36%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 104.326K max: 2.02M average: 1.104M total: 897.724M
column values statistics: no stats for this column
[sdfcvdsffdd] INT32 0.6% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
40.302K max: 729.122K average: 410.228K total: 333.515M (raw data: 532.673M
saving 37%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 65.398K max: 1.132M average: 655.195K total: 532.673M
column values statistics: no stats for this column
[sxfvcsdfvs] INT32 0.0% of all space [PLAIN, RLE, BIT_PACKED] min: 29 max: 31
average: 30 total: 24.673K (raw data: 23.047K saving -7%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 29 average: 28 total: 23.047K
column values statistics: no stats for this column
[asfcvsdff] INT32 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
6.043K max: 101.985K average: 55.766K total: 45.338M (raw data: 86.084M saving
47%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 10.182K max: 208.534K average: 105.884K total: 86.084M
column values statistics: no stats for this column
[sdfvcsdf] BINARY 0.1% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
10.979K max: 182.444K average: 106.428K total: 86.526M (raw data: 103.678M
saving 16%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 13.033K max: 218.883K average: 127.525K total: 103.678M
column values statistics: no stats for this column
[shdsfdffs] DOUBLE 0.1% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
9.391K max: 152.421K average: 84.503K total: 68.701M (raw data: 133.508M saving
48%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 16.943K max: 310.815K average: 164.216K total: 133.508M
column values statistics: no stats for this column
[csdcsdf] DOUBLE 0.1% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
7.878K max: 128.93K average: 71.916K total: 58.468M (raw data: 104.738M saving
44%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 13.331K max: 246.794K average: 128.829K total: 104.738M
column values statistics: no stats for this column
[dsfsdre] DOUBLE 2.8% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
162.577K max: 2.915M average: 1.715M total: 1.394G (raw data: 1.575G saving 11%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 176.103K max: 3.262M average: 1.937M total: 1.575G
column values statistics: no stats for this column
[jhsfddsjhd] DOUBLE 2.3% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
136.649K max: 2.513M average: 1.399M total: 1.137G (raw data: 1.259G saving 9%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 151.755K max: 2.715M average: 1.549M total: 1.259G
column values statistics: no stats for this column
[sdfsed] DOUBLE 2.9% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
168.901K max: 3.078M average: 1.763M total: 1.433G (raw data: 1.639G saving 12%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 195.326K max: 3.448M average: 2.016M total: 1.639G
column values statistics: no stats for this column
[asedawe] DOUBLE 2.2% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
135.216K max: 2.489M average: 1.383M total: 1.124G (raw data: 1.254G saving 10%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 151.372K max: 2.706M average: 1.543M total: 1.254G
column values statistics: no stats for this column
[asfsds] BINARY 0.7% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
41.896K max: 722.958K average: 423.498K total: 344.304M (raw data: 398.268M
saving 13%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 48.76K max: 835.591K average: 489.875K total: 398.268M
column values statistics: no stats for this column
[fsdfds] BINARY 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
3.578K max: 83.864K average: 45.878K total: 37.299M (raw data: 51.26M saving
27%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 4.567K max: 115.645K average: 63.051K total: 51.26M
column values statistics: no stats for this column
[awedwew] INT32 1.2% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
76.892K max: 1.331M average: 772.915K total: 628.38M (raw data: 724.401M saving
13%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 89.268K max: 1.521M average: 891.023K total: 724.401M
column values statistics: no stats for this column
[fasasdrf] BINARY 0.1% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
7.55K max: 135.118K average: 75.399K total: 61.299M (raw data: 104.572M saving
41%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 12.61K max: 225.773K average: 128.625K total: 104.572M
column values statistics: no stats for this column
[asrftdghy] BINARY 1.2% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
76.555K max: 1.299M average: 767.686K total: 624.129M (raw data: 631.193M
saving 1%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 76.89K max: 1.309M average: 776.376K total: 631.193M
column values statistics: no stats for this column
[saefdrfer] INT32 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
4.726K max: 90.738K average: 52.183K total: 42.424M (raw data: 62.22M saving
31%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 6.232K max: 134.0K average: 76.532K total: 62.22M
column values statistics: no stats for this column
[ferfcerfcer] DOUBLE 0.3% of all space [PLAIN, RLE, BIT_PACKED,
PLAIN_DICTIONARY] min: 29 max: 1.358M average: 186.554K total: 151.668M (raw
data: 163.615M saving 7%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 1.507M average: 201.248K total: 163.615M
column values statistics: no stats for this column
[ferfcswwef] DOUBLE 0.2% of all space [PLAIN, RLE, BIT_PACKED,
PLAIN_DICTIONARY] min: 29 max: 1.094M average: 153.897K total: 125.118M (raw
data: 136.055M saving 8%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 1.209M average: 167.349K total: 136.055M
column values statistics: no stats for this column
[sfdsdcsdcs] BINARY 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
1.608K max: 34.364K average: 18.027K total: 14.656M (raw data: 17.471M saving
16%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 1.923K max: 42.145K average: 21.489K total: 17.471M
column values statistics: no stats for this column
[jshdsdsscsfgr] BINARY 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 435 max: 12.191K average: 5.6K total: 4.553M (raw data: 4.776M saving 4%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 424 max: 13.095K average: 5.874K total: 4.776M
column values statistics: no stats for this column
[vsdef8sd] BINARY 8.6% of all space [PLAIN, RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 461.197K max: 8.98M average: 5.201M total: 4.228G (raw data: 26.562G
saving 84%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 1.087M max: 56.927M average: 32.671M total: 26.562G
column values statistics: no stats for this column
[kjsfkd] BINARY 6.5% of all space [PLAIN, RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 465.615K max: 6.718M average: 3.957M total: 3.217G (raw data: 16.091G
saving 80%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 865.573K max: 34.528M average: 19.792M total: 16.091G
column values statistics: no stats for this column
[vsdfcsdfsedr] BINARY 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 2.719K max: 57.189K average: 32.735K total: 26.613M (raw data: 31.447M
saving 15%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 3.162K max: 67.622K average: 38.681K total: 31.447M
column values statistics: no stats for this column
[pos_mfy_side_cd] BINARY 0.6% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 33.24K max: 651.828K average: 369.965K total: 300.781M (raw data: 585.973M
saving 48%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 54.051K max: 1.321M average: 720.754K total: 585.973M
column values statistics: no stats for this column
[sdfdrfsedrf] BINARY 0.1% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
5.567K max: 131.157K average: 75.359K total: 61.267M (raw data: 64.852M saving
5%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 5.841K max: 138.501K average: 79.769K total: 64.852M
column values statistics: no stats for this column
[jhsrfjedr] DOUBLE 2.5% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
168.29K max: 2.666M average: 1.564M total: 1.272G (raw data: 1.736G saving 26%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 211.049K max: 3.656M average: 2.136M total: 1.736G
column values statistics: no stats for this column
[fkjnsfsdfs] DOUBLE 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
1.717K max: 37.121K average: 16.724K total: 13.596M (raw data: 19.695M saving
30%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 2.103K max: 56.995K average: 24.225K total: 19.695M
column values statistics: no stats for this column
[kjsfkdfjvfd] DOUBLE 2.1% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
129.996K max: 2.207M average: 1.281M total: 1.042G (raw data: 1.295G saving 19%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 161.704K max: 2.721M average: 1.593M total: 1.295G
column values statistics: no stats for this column
[asfdrfdstgfrtd] DOUBLE 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 572 max: 14.51K average: 6.247K total: 5.078M (raw data: 5.949M saving 14%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 596 max: 17.823K average: 7.317K total: 5.949M
column values statistics: no stats for this column
[sfderferfa] INT32 0.9% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
47.996K max: 971.094K average: 546.739K total: 444.499M (raw data: 671.462M
saving 33%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 69.727K max: 1.455M average: 825.906K total: 671.462M
column values statistics: no stats for this column
[sdfsddvdfv] BINARY 8.6% of all space [PLAIN, RLE, BIT_PACKED,
PLAIN_DICTIONARY] min: 464.184K max: 8.981M average: 5.201M total: 4.228G (raw
data: 26.562G saving 84%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 1.087M max: 56.928M average: 32.672M total: 26.562G
column values statistics: no stats for this column
[dsfdffddf] BINARY 7.5% of all space [PLAIN, RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 460.227K max: 8.881M average: 4.519M total: 3.674G (raw data: 22.455G
saving 83%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 1.067M max: 55.814M average: 27.621M total: 22.455G
column values statistics: no stats for this column
[sdfcdvdv] BINARY 0.0% of all space [PLAIN, RLE, BIT_PACKED] min: 29 max: 31
average: 30 total: 24.673K (raw data: 23.047K saving -7%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 29 average: 28 total: 23.047K
column values statistics: no stats for this column
[dfvdfvfdvf] BINARY 0.0% of all space [PLAIN, RLE, BIT_PACKED,
PLAIN_DICTIONARY] min: 29 max: 4.273K average: 476 total: 387.551K (raw data:
526.74K saving 26%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 5.914K average: 647 total: 526.74K
column values statistics: no stats for this column
[dvdfvfdgvd] BINARY 0.0% of all space [PLAIN, RLE, BIT_PACKED,
PLAIN_DICTIONARY] min: 29 max: 465.095K average: 49.494K total: 40.239M (raw
data: 88.817M saving 54%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 1.025M average: 109.246K total: 88.817M
column values statistics: no stats for this column
[mhdfjhfdjhdf] BINARY 0.0% of all space [PLAIN, RLE, BIT_PACKED,
PLAIN_DICTIONARY] min: 29 max: 4.289K average: 479 total: 389.698K (raw data:
529.427K saving 26%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 5.92K average: 651 total: 529.427K
column values statistics: no stats for this column
[jhsfjhvfd] BINARY 0.0% of all space [PLAIN, RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 29 max: 466.681K average: 49.517K total: 40.258M (raw data: 88.822M saving
54%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 1.026M average: 109.252K total: 88.822M
column values statistics: no stats for this column
[kjdnfvkjdfv] DOUBLE 3.6% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
316.335K max: 3.692M average: 2.211M total: 1.798G (raw data: 2.426G saving 25%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 393.457K max: 5.127M average: 2.985M total: 2.426G
column values statistics: no stats for this column
[khdbfvjhdf] DOUBLE 4.1% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
267.46K max: 4.329M average: 2.503M total: 2.035G (raw data: 2.22G saving 8%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 321.934K max: 4.658M average: 2.731M total: 2.22G
column values statistics: no stats for this column
[kjdfvjbhdf] DOUBLE 3.7% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
315.815K max: 3.688M average: 2.262M total: 1.839G (raw data: 2.457G saving 25%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 391.193K max: 5.123M average: 3.022M total: 2.457G
column values statistics: no stats for this column
[vjndfjhdfj] DOUBLE 3.6% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
353.409K max: 3.624M average: 2.228M total: 1.811G (raw data: 2.507G saving 27%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 425.049K max: 5.173M average: 3.084M total: 2.507G
column values statistics: no stats for this column
[jndkfjnvfd] DOUBLE 3.6% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
350.507K max: 3.649M average: 2.226M total: 1.81G (raw data: 2.497G saving 27%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 420.385K max: 5.165M average: 3.072M total: 2.497G
column values statistics: no stats for this column
[asfdcdsfgdf] DOUBLE 4.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
381.806K max: 4.382M average: 2.453M total: 1.994G (raw data: 2.682G saving 25%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 470.019K max: 5.514M average: 3.299M total: 2.682G
column values statistics: no stats for this column
[hjsfjjsdsd] DOUBLE 4.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
381.751K max: 4.383M average: 2.455M total: 1.996G (raw data: 2.682G saving 25%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 469.561K max: 5.515M average: 3.299M total: 2.682G
column values statistics: no stats for this column
[ljdfjhdff] INT32 0.2% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
15.157K max: 265.209K average: 150.661K total: 122.487M (raw data: 139.965M
saving 12%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 17.189K max: 300.896K average: 172.159K total: 139.965M
column values statistics: no stats for this column
[dfjdfhdfd] INT32 0.2% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
16.158K max: 289.711K average: 165.335K total: 134.417M (raw data: 150.908M
saving 10%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 18.094K max: 323.347K average: 185.619K total: 150.908M
column values statistics: no stats for this column
[hbdsjfdsjsfd] INT32 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
3.563K max: 72.02K average: 39.783K total: 32.343M (raw data: 40.55M saving 20%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 4.314K max: 90.816K average: 49.877K total: 40.55M
column values statistics: no stats for this column
[hbsdjhsds] INT32 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
3.563K max: 72.02K average: 39.915K total: 32.45M (raw data: 40.687M saving 20%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 4.314K max: 90.816K average: 50.046K total: 40.687M
column values statistics: no stats for this column
[anhsdl] INT32 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
3.563K max: 72.02K average: 40.256K total: 32.728M (raw data: 41.013M saving
20%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 4.314K max: 90.816K average: 50.447K total: 41.013M
column values statistics: no stats for this column
[sdfdrgfre] INT32 0.1% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
7.999K max: 153.948K average: 85.915K total: 69.849M (raw data: 84.863M saving
17%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 9.688K max: 186.586K average: 104.383K total: 84.863M
column values statistics: no stats for this column
[dmhdjfdf] INT32 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
2.485K max: 48.511K average: 25.753K total: 20.937M (raw data: 28.633M saving
26%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 3.001K max: 66.01K average: 35.219K total: 28.633M
column values statistics: no stats for this column
[kjhdfjhdfd] DOUBLE 0.2% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
16.621K max: 337.219K average: 174.251K total: 141.666M (raw data: 185.447M
saving 23%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 22.335K max: 415.024K average: 228.102K total: 185.447M
column values statistics: no stats for this column
[hbdfjdfjd] INT32 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
4.078K max: 70.783K average: 39.507K total: 32.119M (raw data: 39.332M saving
18%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 4.773K max: 86.906K average: 48.379K total: 39.332M
column values statistics: no stats for this column
[jhdjdd] INT32 0.0% of all space [PLAIN, RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 29 max: 431 average: 91 total: 74.017K (raw data: 71.129K saving -4%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 423 average: 87 total: 71.129K
column values statistics: no stats for this column
[dfvdfvdf] INT32 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min: 52
max: 564 average: 256 total: 208.711K (raw data: 197.024K saving -5%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 48 max: 542 average: 242 total: 197.024K
column values statistics: no stats for this column
[dyjnyyuu] INT32 0.7% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min:
41.862K max: 723.544K average: 423.527K total: 344.327M (raw data: 397.954M
saving 13%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 48.684K max: 834.978K average: 489.488K total: 397.954M
column values statistics: no stats for this column
[oyuyuyr] BINARY 0.0% of all space [PLAIN, RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 29 max: 933 average: 165 total: 134.471K (raw data: 151.23K saving 11%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 1.188K average: 186 total: 151.23K
column values statistics: no stats for this column
[hryhtyh] BINARY 0.0% of all space [RLE, BIT_PACKED, PLAIN_DICTIONARY] min: 53
max: 580 average: 305 total: 248.005K (raw data: 233.462K saving -6%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 49 max: 552 average: 287 total: 233.462K
column values statistics: no stats for this column
[hgnbghn] INT32 0.0% of all space [PLAIN, RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 29 max: 292 average: 70 total: 57.38K (raw data: 54.742K saving -4%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 285 average: 67 total: 54.742K
column values statistics: no stats for this column
[ghnghght] BINARY 0.0% of all space [PLAIN, RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 29 max: 59 average: 30 total: 24.757K (raw data: 23.125K saving -7%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 55 average: 28 total: 23.125K
column values statistics: no stats for this column
[ghngnghgg] INT32 0.0% of all space [PLAIN, RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 29 max: 58 average: 30 total: 24.754K (raw data: 23.122K saving -7%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 54 average: 28 total: 23.122K
column values statistics: no stats for this column
[ghnghghthhh] BINARY 0.0% of all space [PLAIN, RLE, BIT_PACKED,
PLAIN_DICTIONARY] min: 29 max: 58 average: 30 total: 24.754K (raw data: 23.122K
saving -7%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 54 average: 28 total: 23.122K
column values statistics: no stats for this column
[hnghngh] DOUBLE 0.0% of all space [PLAIN, RLE, BIT_PACKED] min: 29 max: 31
average: 30 total: 24.673K (raw data: 23.047K saving -7%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 29 average: 28 total: 23.047K
column values statistics: no stats for this column
[ghnghngh] BINARY 0.0% of all space [PLAIN, RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 29 max: 172 average: 60 total: 49.264K (raw data: 46.683K saving -5%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 168 average: 57 total: 46.683K
column values statistics: no stats for this column
[nghhngnhg] DOUBLE 0.0% of all space [PLAIN, RLE, BIT_PACKED, PLAIN_DICTIONARY]
min: 29 max: 254 average: 71 total: 58.415K (raw data: 56.25K saving -3%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 255 average: 69 total: 56.25K
column values statistics: no stats for this column
[hgsdjhssssf] DOUBLE 0.0% of all space [PLAIN, RLE, BIT_PACKED,
PLAIN_DICTIONARY] min: 29 max: 254 average: 71 total: 58.415K (raw data: 56.25K
saving -3%)
values: min: 152.662K max: 2.606M average: 1.545M total: 1.256G
uncompressed: min: 27 max: 255 average: 69 total: 56.25K
column values statistics: no stats for this column
number of blocks: 813
total data size: 48.972G (raw 153.908G)
total record: 1.256G
average block size: 60.237M (raw 189.309M)
average record count: 1.545M
