[ https://issues.apache.org/jira/browse/HIVE-8292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mostafa Mokhtar updated HIVE-8292: ---------------------------------- Description: Reading from bucketed partitioned tables has significantly higher overhead compared to non-bucketed non-partitioned files. 50% of the profile is spent in MapOperator.cleanUpInputFileChangedOp 5% the CPU in {code} Path onepath = normalizePath(onefile); {code} And 45% the CPU in {code} onepath.toUri().relativize(fpath.toUri()).equals(fpath.toUri()); {code} >From the profiler {code} Stack Trace Sample Count Percentage(%) org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(Object) 978 28.613 org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(Writable) 978 28.613 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged() 866 25.336 org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp() 866 25.336 java.net.URI.relativize(URI) 655 19.163 java.net.URI.relativize(URI, URI) 655 19.163 java.net.URI.normalize(String) 517 15.126 java.net.URI.needsNormalization(String) 372 10.884 java.lang.String.charAt(int) 235 6.875 java.net.URI.equal(String, String) 27 0.79 java.lang.StringBuilder.toString() 1 0.029 java.lang.StringBuilder.<init>() 1 0.029 java.lang.StringBuilder.append(String) 1 0.029 org.apache.hadoop.hive.ql.exec.MapOperator.normalizePath(String) 167 4.886 org.apache.hadoop.fs.Path.<init>(String) 162 4.74 org.apache.hadoop.fs.Path.initialize(String, String, String, String) 162 4.74 org.apache.hadoop.fs.Path.normalizePath(String, String) 97 2.838 org.apache.commons.lang.StringUtils.replace(String, String, String) 97 2.838 org.apache.commons.lang.StringUtils.replace(String, String, String, int) 97 2.838 java.lang.String.indexOf(String, int) 97 2.838 java.net.URI.<init>(String, String, String, String, String) 65 1.902 {code} was: Reading from bucketed partitioned tables has significantly higher overhead compared to non-bucketed non-partitioned files. 20% of the profile is spent in MapOperator.cleanUpInputFileChangedOp 5% the CPU in {code} Path onepath = normalizePath(onefile); {code} And 15% the CPU in {code} onepath.toUri().relativize(fpath.toUri()).equals(fpath.toUri()); {code} >From the profiler {code} Stack Trace Sample Count Percentage(%) org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(Object) 978 28.613 org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(Writable) 978 28.613 org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged() 866 25.336 org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp() 866 25.336 java.net.URI.relativize(URI) 655 19.163 java.net.URI.relativize(URI, URI) 655 19.163 java.net.URI.normalize(String) 517 15.126 java.net.URI.needsNormalization(String) 372 10.884 java.lang.String.charAt(int) 235 6.875 java.net.URI.equal(String, String) 27 0.79 java.lang.StringBuilder.toString() 1 0.029 java.lang.StringBuilder.<init>() 1 0.029 java.lang.StringBuilder.append(String) 1 0.029 org.apache.hadoop.hive.ql.exec.MapOperator.normalizePath(String) 167 4.886 org.apache.hadoop.fs.Path.<init>(String) 162 4.74 org.apache.hadoop.fs.Path.initialize(String, String, String, String) 162 4.74 org.apache.hadoop.fs.Path.normalizePath(String, String) 97 2.838 org.apache.commons.lang.StringUtils.replace(String, String, String) 97 2.838 org.apache.commons.lang.StringUtils.replace(String, String, String, int) 97 2.838 java.lang.String.indexOf(String, int) 97 2.838 java.net.URI.<init>(String, String, String, String, String) 65 1.902 {code} > Reading from partitioned bucketed tables has high overhead in > MapOperator.cleanUpInputFileChangedOp > --------------------------------------------------------------------------------------------------- > > Key: HIVE-8292 > URL: https://issues.apache.org/jira/browse/HIVE-8292 > Project: Hive > Issue Type: Bug > Components: Query Processor > Affects Versions: 0.14.0 > Environment: cn105 > Reporter: Mostafa Mokhtar > Assignee: Prasanth J > Fix For: 0.14.0 > > Attachments: 2014_09_29_14_46_04.jfr > > > Reading from bucketed partitioned tables has significantly higher overhead > compared to non-bucketed non-partitioned files. > 50% of the profile is spent in MapOperator.cleanUpInputFileChangedOp > 5% the CPU in > {code} > Path onepath = normalizePath(onefile); > {code} > And > 45% the CPU in > {code} > onepath.toUri().relativize(fpath.toUri()).equals(fpath.toUri()); > {code} > From the profiler > {code} > Stack Trace Sample Count Percentage(%) > org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(Object) 978 > 28.613 > org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(Writable) > 978 28.613 > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged() > 866 25.336 > > org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp() > 866 25.336 > java.net.URI.relativize(URI) 655 19.163 > java.net.URI.relativize(URI, URI) 655 19.163 > java.net.URI.normalize(String) 517 15.126 > java.net.URI.needsNormalization(String) > 372 10.884 > java.lang.String.charAt(int) 235 > 6.875 > > java.net.URI.equal(String, String) 27 0.79 > > java.lang.StringBuilder.toString() 1 0.029 > > java.lang.StringBuilder.<init>() 1 0.029 > > java.lang.StringBuilder.append(String) 1 0.029 > > org.apache.hadoop.hive.ql.exec.MapOperator.normalizePath(String) 167 > 4.886 > > org.apache.hadoop.fs.Path.<init>(String) 162 4.74 > > org.apache.hadoop.fs.Path.initialize(String, String, String, String) 162 > 4.74 > org.apache.hadoop.fs.Path.normalizePath(String, String) 97 2.838 > org.apache.commons.lang.StringUtils.replace(String, String, String) > 97 2.838 > org.apache.commons.lang.StringUtils.replace(String, String, > String, int) 97 2.838 > java.lang.String.indexOf(String, int) 97 2.838 > java.net.URI.<init>(String, String, String, String, String) > 65 1.902 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)