[jira] [Commented] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
[ https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163143#comment-15163143 ] Alan Gates commented on HIVE-13132: --- Several comments: # It would be good to test whether HIVE-2573 solves the issue, since there's no point in making further changes if it does. # I see how this code prevents the system from repeatedly downloading the functions (since it tracks whether the metastore has been searched) but I don't see how it prevents pre-fetching all the functions at startup. # I don't think using statics in the FunctionRegistry will work. This will cause HiveServer2 to share the function names across sessions, which we don't want because there won't be a way to force new functions to be downloaded. That is, HS2 will download the set of functions when it first starts, and not do so again because the static haveSearchedMetastore will be true. cc [~jdere] and [~sershe] since both of you have done work in this area recently. > Hive should lazily load and cache metastore (permanent) functions > - > > Key: HIVE-13132 > URL: https://issues.apache.org/jira/browse/HIVE-13132 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.13.1 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > Attachments: HIVE-13132.1.patch > > > In Hive 0.13.1, we have noticed that as the number of databases increases, > the start-up time of the Hive interactive shell increases. This is because > during start-up, all databases are iterated over to fetch the permanent > functions to display in the {{SHOW FUNCTIONS}} output. > {noformat:title=FunctionRegistry.java} > private static Set getFunctionNames(boolean searchMetastore) { > Set functionNames = mFunctions.keySet(); > if (searchMetastore) { > functionNames = new HashSet(functionNames); > try { > Hive db = getHive(); > List dbNames = db.getAllDatabases(); > for (String dbName : dbNames) { > List funcNames = db.getFunctions(dbName, "*"); > for (String funcName : funcNames) { > functionNames.add(FunctionUtils.qualifyFunctionName(funcName, > dbName)); > } > } > } catch (Exception e) { > LOG.error(e); > // Continue on, we can still return the functions we've gotten to > this point. > } > } > return functionNames; > } > {noformat} > Instead of eagerly loading all metastore functions, we should only load them > the first time {{SHOW FUNCTIONS}} is invoked. We should also cache the > results. > Note that this issue may have been fixed by HIVE-2573, though I haven't > verified this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
[ https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163542#comment-15163542 ] Hive QA commented on HIVE-13132: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12789357/HIVE-13132.1.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7081/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7081/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7081/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-7081/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive 74c2f06..f8f2014 branch-2.0 -> origin/branch-2.0 402fabe..2f73233 master -> origin/master + git reset --hard HEAD HEAD is now at 402fabe HIVE-13064 : Fix passing serde properties from insert overwrite directory (Rajat Khandelwal, reviewed by Amareshwari Sriramadasu) + git clean -f -d + git checkout master Already on 'master' Your branch is behind 'origin/master' by 3 commits, and can be fast-forwarded. + git reset --hard origin/master HEAD is now at 2f73233 HIVE-13128. NullScan fails on a secure setup. (Siddharth Seth, reviewed by Sergey Shelukhin) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12789357 - PreCommit-HIVE-TRUNK-Build > Hive should lazily load and cache metastore (permanent) functions > - > > Key: HIVE-13132 > URL: https://issues.apache.org/jira/browse/HIVE-13132 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.13.1 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > Attachments: HIVE-13132.1.patch > > > In Hive 0.13.1, we have noticed that as the number of databases increases, > the start-up time of the Hive interactive shell increases. This is because > during start-up, all databases are iterated over to fetch the permanent > functions to display in the {{SHOW FUNCTIONS}} output. > {noformat:title=FunctionRegistry.java} > private static Set getFunctionNames(boolean searchMetastore) { > Set functionNames = mFunctions.keySet(); > if (searchMetastore) { > functionNames = new HashSet(functionNames); > try { > Hive db = getHive(); > List dbNames = db.getAllDatabases(); > for (String dbName : dbNames) { > List funcNames = db.getFunctions(dbName, "*"); > for (String funcName : funcNames) { > functionNames.add(FunctionUtils.qualifyFunctionName(funcName, > dbName)); > } > } > } catch (Exception e) { > LOG.error(e); > // Continue on, we can still return the functions we've gotten to > this poin
[jira] [Commented] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
[ https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167639#comment-15167639 ] Anthony Hsu commented on HIVE-13132: Thanks for the review, [~alangates]. # I tested and unfortunately, HIVE-2573 does NOT solve this issue. A stacktrace in jdb shows all functions are loaded during CliDriver start-up: {noformat} main[1] where [1] org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions (Hive.java:173) [2] org.apache.hadoop.hive.ql.metadata.Hive. (Hive.java:166) [3] org.apache.hadoop.hive.ql.session.SessionState.start (SessionState.java:503) [4] org.apache.hadoop.hive.cli.CliDriver.run (CliDriver.java:677) [5] org.apache.hadoop.hive.cli.CliDriver.main (CliDriver.java:621) [6] sun.reflect.NativeMethodAccessorImpl.invoke0 (native method) [7] sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) [8] sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) [9] java.lang.reflect.Method.invoke (Method.java:483) [10] org.apache.hadoop.util.RunJar.run (RunJar.java:221) [11] org.apache.hadoop.util.RunJar.main (RunJar.java:136) {noformat} # My fix is a bit hacky and only works on the Hive 0.13.1 branch. I basically changed the CliDriver initialization code to use {{FunctionRegistry.getFunctionNames(String funcPatternStr)}} instead of {{FunctionRegistry.getFunctionNames()}}. In the Hive 0.13.1 branch, the former does NOT search the metastore while the latter does. This is no longer the case in trunk. # I don't have much experience with HS2 but I will take a look. I'll work on a cleaner solution that removes the pre-loading of metastore functions from the CliDriver initialization code path. Suggestions welcome. > Hive should lazily load and cache metastore (permanent) functions > - > > Key: HIVE-13132 > URL: https://issues.apache.org/jira/browse/HIVE-13132 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.13.1 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > Attachments: HIVE-13132.1.patch > > > In Hive 0.13.1, we have noticed that as the number of databases increases, > the start-up time of the Hive interactive shell increases. This is because > during start-up, all databases are iterated over to fetch the permanent > functions to display in the {{SHOW FUNCTIONS}} output. > {noformat:title=FunctionRegistry.java} > private static Set getFunctionNames(boolean searchMetastore) { > Set functionNames = mFunctions.keySet(); > if (searchMetastore) { > functionNames = new HashSet(functionNames); > try { > Hive db = getHive(); > List dbNames = db.getAllDatabases(); > for (String dbName : dbNames) { > List funcNames = db.getFunctions(dbName, "*"); > for (String funcName : funcNames) { > functionNames.add(FunctionUtils.qualifyFunctionName(funcName, > dbName)); > } > } > } catch (Exception e) { > LOG.error(e); > // Continue on, we can still return the functions we've gotten to > this point. > } > } > return functionNames; > } > {noformat} > Instead of eagerly loading all metastore functions, we should only load them > the first time {{SHOW FUNCTIONS}} is invoked. We should also cache the > results. > Note that this issue may have been fixed by HIVE-2573, though I haven't > verified this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
[ https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174819#comment-15174819 ] Jason Dere commented on HIVE-13132: --- It is true that HIVE-2573 still loads all permanent functions from all databases during startup. One thing that might improve things is HIVE-10319, which rather than calling once for every database, only makes a single call to the metastore to retrieve all permanent functions. Can you check if this helps at all? > Hive should lazily load and cache metastore (permanent) functions > - > > Key: HIVE-13132 > URL: https://issues.apache.org/jira/browse/HIVE-13132 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.13.1 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > Attachments: HIVE-13132.1.patch > > > In Hive 0.13.1, we have noticed that as the number of databases increases, > the start-up time of the Hive interactive shell increases. This is because > during start-up, all databases are iterated over to fetch the permanent > functions to display in the {{SHOW FUNCTIONS}} output. > {noformat:title=FunctionRegistry.java} > private static Set getFunctionNames(boolean searchMetastore) { > Set functionNames = mFunctions.keySet(); > if (searchMetastore) { > functionNames = new HashSet(functionNames); > try { > Hive db = getHive(); > List dbNames = db.getAllDatabases(); > for (String dbName : dbNames) { > List funcNames = db.getFunctions(dbName, "*"); > for (String funcName : funcNames) { > functionNames.add(FunctionUtils.qualifyFunctionName(funcName, > dbName)); > } > } > } catch (Exception e) { > LOG.error(e); > // Continue on, we can still return the functions we've gotten to > this point. > } > } > return functionNames; > } > {noformat} > Instead of eagerly loading all metastore functions, we should only load them > the first time {{SHOW FUNCTIONS}} is invoked. We should also cache the > results. > Note that this issue may have been fixed by HIVE-2573, though I haven't > verified this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
[ https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181545#comment-15181545 ] Anthony Hsu commented on HIVE-13132: [~jdere], thanks for pointing out HIVE-10319! I backported it to Hive 0.13.1 and it reduced my CLI start-up time from 20+ seconds to < 1 second. {{SHOW FUNCTIONS}} also now takes < 1 second rather than 20+ seconds. This ticket is no longer needed, so I will resolve it. > Hive should lazily load and cache metastore (permanent) functions > - > > Key: HIVE-13132 > URL: https://issues.apache.org/jira/browse/HIVE-13132 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.13.1 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > Attachments: HIVE-13132.1.patch > > > In Hive 0.13.1, we have noticed that as the number of databases increases, > the start-up time of the Hive interactive shell increases. This is because > during start-up, all databases are iterated over to fetch the permanent > functions to display in the {{SHOW FUNCTIONS}} output. > {noformat:title=FunctionRegistry.java} > private static Set getFunctionNames(boolean searchMetastore) { > Set functionNames = mFunctions.keySet(); > if (searchMetastore) { > functionNames = new HashSet(functionNames); > try { > Hive db = getHive(); > List dbNames = db.getAllDatabases(); > for (String dbName : dbNames) { > List funcNames = db.getFunctions(dbName, "*"); > for (String funcName : funcNames) { > functionNames.add(FunctionUtils.qualifyFunctionName(funcName, > dbName)); > } > } > } catch (Exception e) { > LOG.error(e); > // Continue on, we can still return the functions we've gotten to > this point. > } > } > return functionNames; > } > {noformat} > Instead of eagerly loading all metastore functions, we should only load them > the first time {{SHOW FUNCTIONS}} is invoked. We should also cache the > results. > Note that this issue may have been fixed by HIVE-2573, though I haven't > verified this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
[ https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181558#comment-15181558 ] Anthony Hsu commented on HIVE-13132: Correction: CLI start-up time was reduce from 20+ seconds to < 5 seconds. > Hive should lazily load and cache metastore (permanent) functions > - > > Key: HIVE-13132 > URL: https://issues.apache.org/jira/browse/HIVE-13132 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.13.1 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > Attachments: HIVE-13132.1.patch > > > In Hive 0.13.1, we have noticed that as the number of databases increases, > the start-up time of the Hive interactive shell increases. This is because > during start-up, all databases are iterated over to fetch the permanent > functions to display in the {{SHOW FUNCTIONS}} output. > {noformat:title=FunctionRegistry.java} > private static Set getFunctionNames(boolean searchMetastore) { > Set functionNames = mFunctions.keySet(); > if (searchMetastore) { > functionNames = new HashSet(functionNames); > try { > Hive db = getHive(); > List dbNames = db.getAllDatabases(); > for (String dbName : dbNames) { > List funcNames = db.getFunctions(dbName, "*"); > for (String funcName : funcNames) { > functionNames.add(FunctionUtils.qualifyFunctionName(funcName, > dbName)); > } > } > } catch (Exception e) { > LOG.error(e); > // Continue on, we can still return the functions we've gotten to > this point. > } > } > return functionNames; > } > {noformat} > Instead of eagerly loading all metastore functions, we should only load them > the first time {{SHOW FUNCTIONS}} is invoked. We should also cache the > results. > Note that this issue may have been fixed by HIVE-2573, though I haven't > verified this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)