[jira] [Commented] (HIVE-7762) Enhancement while getting partitions via webhcat client
[ https://issues.apache.org/jira/browse/HIVE-7762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143083#comment-14143083 ] Hive QA commented on HIVE-7762: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12670381/HIVE-7762.3.patch {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6301 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.ql.parse.TestParse.testParse_union {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/920/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/920/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-920/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12670381 Enhancement while getting partitions via webhcat client --- Key: HIVE-7762 URL: https://issues.apache.org/jira/browse/HIVE-7762 Project: Hive Issue Type: Improvement Components: WebHCat Reporter: Suhas Vasu Priority: Minor Attachments: HIVE-7762.2.patch, HIVE-7762.3.patch, HIVE-7762.patch Hcatalog creates partitions in lower case, whereas getting partitions from hcatalog via webhcat client doesn't handle this. So the client starts throwing exceptions. Ex: CREATE EXTERNAL TABLE in_table (word STRING, cnt INT) PARTITIONED BY (Year STRING, Month STRING, Date STRING, Hour STRING, Minute STRING) STORED AS TEXTFILE LOCATION '/user/suhas/hcat-data/in/'; Then i try to get partitions by: {noformat} String inputTableName = in_table; String database = default; MapString, String partitionSpec = new HashMapString, String(); partitionSpec.put(Year, 2014); partitionSpec.put(Month, 08); partitionSpec.put(Date, 11); partitionSpec.put(Hour, 00); partitionSpec.put(Minute, 00); HCatClient client = get(catalogUrl); HCatPartition hCatPartition = client.getPartition(database, inputTableName, partitionSpec); {noformat} This throws up saying: {noformat} Exception in thread main org.apache.hcatalog.common.HCatException : 9001 : Exception occurred while processing HCat request : Invalid partition-key specified: year at org.apache.hcatalog.api.HCatClientHMSImpl.getPartition(HCatClientHMSImpl.java:366) at com.inmobi.demo.HcatPartitions.main(HcatPartitions.java:34) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) {noformat} The same code works if i do {noformat} partitionSpec.put(year, 2014); partitionSpec.put(month, 08); partitionSpec.put(date, 11); partitionSpec.put(hour, 00); partitionSpec.put(minute, 00); {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7762) Enhancement while getting partitions via webhcat client
[ https://issues.apache.org/jira/browse/HIVE-7762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14137021#comment-14137021 ] Suhas Vasu commented on HIVE-7762: -- Sure, i'' add the test-cases. makes sense to have them. Guess {noformat}HCatClient.getPartitions(){noformat} behaves as expected but i'll check it once. I'll update the jira with the findings Enhancement while getting partitions via webhcat client --- Key: HIVE-7762 URL: https://issues.apache.org/jira/browse/HIVE-7762 Project: Hive Issue Type: Improvement Components: WebHCat Reporter: Suhas Vasu Priority: Minor Attachments: HIVE-7762.2.patch, HIVE-7762.patch Hcatalog creates partitions in lower case, whereas getting partitions from hcatalog via webhcat client doesn't handle this. So the client starts throwing exceptions. Ex: CREATE EXTERNAL TABLE in_table (word STRING, cnt INT) PARTITIONED BY (Year STRING, Month STRING, Date STRING, Hour STRING, Minute STRING) STORED AS TEXTFILE LOCATION '/user/suhas/hcat-data/in/'; Then i try to get partitions by: {noformat} String inputTableName = in_table; String database = default; MapString, String partitionSpec = new HashMapString, String(); partitionSpec.put(Year, 2014); partitionSpec.put(Month, 08); partitionSpec.put(Date, 11); partitionSpec.put(Hour, 00); partitionSpec.put(Minute, 00); HCatClient client = get(catalogUrl); HCatPartition hCatPartition = client.getPartition(database, inputTableName, partitionSpec); {noformat} This throws up saying: {noformat} Exception in thread main org.apache.hcatalog.common.HCatException : 9001 : Exception occurred while processing HCat request : Invalid partition-key specified: year at org.apache.hcatalog.api.HCatClientHMSImpl.getPartition(HCatClientHMSImpl.java:366) at com.inmobi.demo.HcatPartitions.main(HcatPartitions.java:34) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) {noformat} The same code works if i do {noformat} partitionSpec.put(year, 2014); partitionSpec.put(month, 08); partitionSpec.put(date, 11); partitionSpec.put(hour, 00); partitionSpec.put(minute, 00); {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7762) Enhancement while getting partitions via webhcat client
[ https://issues.apache.org/jira/browse/HIVE-7762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135954#comment-14135954 ] Mithun Radhakrishnan commented on HIVE-7762: Hello, Suhas. Thanks for working on fixing this inconsistency. Generally, this is a good fix. I'd encourage you to add a test-case to TestHCatClient, to create a table with uppercase partition columns, and then querying it with a lowercase partition-spec. (Essentially, what you've included in your description.) Adding one won't be hard; you could just use one of the other tests for reference. Also, could I please bother you to verify the behaviour of {{HCatClient.getPartitions()}}, for case insensitivity? If it's broken too, I'd rather we fixed both here. I expect that this should be alright, since it goes through the {{listPartitionsByFilter()}} API, but it would be good to have confirmation. Mithun Enhancement while getting partitions via webhcat client --- Key: HIVE-7762 URL: https://issues.apache.org/jira/browse/HIVE-7762 Project: Hive Issue Type: Improvement Components: WebHCat Reporter: Suhas Vasu Priority: Minor Attachments: HIVE-7762.2.patch, HIVE-7762.patch Hcatalog creates partitions in lower case, whereas getting partitions from hcatalog via webhcat client doesn't handle this. So the client starts throwing exceptions. Ex: CREATE EXTERNAL TABLE in_table (word STRING, cnt INT) PARTITIONED BY (Year STRING, Month STRING, Date STRING, Hour STRING, Minute STRING) STORED AS TEXTFILE LOCATION '/user/suhas/hcat-data/in/'; Then i try to get partitions by: {noformat} String inputTableName = in_table; String database = default; MapString, String partitionSpec = new HashMapString, String(); partitionSpec.put(Year, 2014); partitionSpec.put(Month, 08); partitionSpec.put(Date, 11); partitionSpec.put(Hour, 00); partitionSpec.put(Minute, 00); HCatClient client = get(catalogUrl); HCatPartition hCatPartition = client.getPartition(database, inputTableName, partitionSpec); {noformat} This throws up saying: {noformat} Exception in thread main org.apache.hcatalog.common.HCatException : 9001 : Exception occurred while processing HCat request : Invalid partition-key specified: year at org.apache.hcatalog.api.HCatClientHMSImpl.getPartition(HCatClientHMSImpl.java:366) at com.inmobi.demo.HcatPartitions.main(HcatPartitions.java:34) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) {noformat} The same code works if i do {noformat} partitionSpec.put(year, 2014); partitionSpec.put(month, 08); partitionSpec.put(date, 11); partitionSpec.put(hour, 00); partitionSpec.put(minute, 00); {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7762) Enhancement while getting partitions via webhcat client
[ https://issues.apache.org/jira/browse/HIVE-7762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102306#comment-14102306 ] Hive QA commented on HIVE-7762: --- {color:green}Overall{color}: +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12662754/HIVE-7762.2.patch {color:green}SUCCESS:{color} +1 5819 tests passed Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/402/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/402/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-402/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12662754 Enhancement while getting partitions via webhcat client --- Key: HIVE-7762 URL: https://issues.apache.org/jira/browse/HIVE-7762 Project: Hive Issue Type: Improvement Components: WebHCat Reporter: Suhas Vasu Priority: Minor Attachments: HIVE-7762.2.patch, HIVE-7762.patch Hcatalog creates partitions in lower case, whereas getting partitions from hcatalog via webhcat client doesn't handle this. So the client starts throwing exceptions. Ex: CREATE EXTERNAL TABLE in_table (word STRING, cnt INT) PARTITIONED BY (Year STRING, Month STRING, Date STRING, Hour STRING, Minute STRING) STORED AS TEXTFILE LOCATION '/user/suhas/hcat-data/in/'; Then i try to get partitions by: {noformat} String inputTableName = in_table; String database = default; MapString, String partitionSpec = new HashMapString, String(); partitionSpec.put(Year, 2014); partitionSpec.put(Month, 08); partitionSpec.put(Date, 11); partitionSpec.put(Hour, 00); partitionSpec.put(Minute, 00); HCatClient client = get(catalogUrl); HCatPartition hCatPartition = client.getPartition(database, inputTableName, partitionSpec); {noformat} This throws up saying: {noformat} Exception in thread main org.apache.hcatalog.common.HCatException : 9001 : Exception occurred while processing HCat request : Invalid partition-key specified: year at org.apache.hcatalog.api.HCatClientHMSImpl.getPartition(HCatClientHMSImpl.java:366) at com.inmobi.demo.HcatPartitions.main(HcatPartitions.java:34) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) {noformat} The same code works if i do {noformat} partitionSpec.put(year, 2014); partitionSpec.put(month, 08); partitionSpec.put(date, 11); partitionSpec.put(hour, 00); partitionSpec.put(minute, 00); {noformat} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7762) Enhancement while getting partitions via webhcat client
[ https://issues.apache.org/jira/browse/HIVE-7762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100849#comment-14100849 ] Hive QA commented on HIVE-7762: --- {color:red}Overall{color}: -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12662470/HIVE-7762.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/382/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/382/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-382/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-382/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java' Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java' Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java' Reverted 'service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java' Reverted 'service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java' Reverted 'service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java' Reverted 'service/src/java/org/apache/hive/service/cli/CLIService.java' Reverted 'service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java' Reverted 'service/src/java/org/apache/hive/service/cli/session/SessionManager.java' Reverted 'service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java' ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hwi/target common/target common/src/gen service/target service/src/java/org/apache/hive/service/server/ThreadWithGarbageCleanup.java service/src/java/org/apache/hive/service/server/ThreadFactoryWithGarbageCleanup.java contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target + svn update Uant/src/org/apache/hadoop/hive/ant/QTestGenTask.java U testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestParser.java Fetching external item into 'hcatalog/src/test/e2e/harness' Updated external to revision 1618663. Updated to revision 1618663. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0,