[jira] [Commented] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13258454#comment-13258454 ] Carl Steinbach commented on HIVE-2646: -- @Thomas: I'm going to try to fix the pdk problem later today. In the meantime I was wondering if you could look at the eclipse templates and make sure that they're up to date? Thanks. Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs Key: HIVE-2646 URL: https://issues.apache.org/jira/browse/HIVE-2646 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Andrew Bayer Assignee: Andrew Bayer Priority: Critical Fix For: 0.9.0 Attachments: HIVE-2646-fixtests.patch, HIVE-2646.D2133.1.patch, HIVE-2646.D2133.10.patch, HIVE-2646.D2133.11.patch, HIVE-2646.D2133.12.patch, HIVE-2646.D2133.13.patch, HIVE-2646.D2133.14.patch, HIVE-2646.D2133.15.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.D2133.4.patch, HIVE-2646.D2133.5.patch, HIVE-2646.D2133.6.patch, HIVE-2646.D2133.7.patch, HIVE-2646.D2133.8.patch, HIVE-2646.D2133.9.patch, HIVE-2646.D2883.1.patch, HIVE-2646.diff.txt The current Hive Ivy dependency logic for its Hadoop dependencies is problematic - depending on the tarball and extracting the jars from there, rather than depending on the jars directly. It'd be great if this was fixed to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[
https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13258452#comment-13258452
]
Carl Steinbach commented on HIVE-2646:
--
@Thomas: I tried running the tests on Jenkins and noticed that the build is
failing despite all tests passing. It looks like this is because compilation of
the pdk subproject is failing:
{noformat}
BUILD FAILED
/var/lib/hudson/production/workspace/carl-HIVE-patch-test1/build.xml:316: The
following error occurred while executing this line:
/var/lib/hudson/production/workspace/carl-HIVE-patch-test1/build.xml:321: The
following error occurred while executing this line:
/var/lib/hudson/production/workspace/carl-HIVE-patch-test1/pdk/build.xml:52:
The following error occurred while executing this line:
/var/lib/hudson/production/workspace/carl-HIVE-patch-test1/build/dist/scripts/pdk/build-plugin.xml:58:
/var/lib/hudson/production/workspace/carl-HIVE-patch-test1/build/pdk/test-plugin/${build.ivy.lib.dir}/default
does not exist.
at
org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:474)
at org.apache.tools.ant.types.FileSet.iterator(FileSet.java:69)
at
org.apache.tools.ant.types.resources.Union.getCollection(Union.java:123)
at
org.apache.tools.ant.types.resources.Union.getCollection(Union.java:107)
at
org.apache.tools.ant.types.resources.BaseResourceCollectionContainer.cacheCollection(BaseResourceCollectionContainer.java:265)
at
org.apache.tools.ant.types.resources.BaseResourceCollectionContainer.iterator(BaseResourceCollectionContainer.java:142)
at org.apache.tools.ant.types.Path.iterator(Path.java:704)
at org.apache.tools.ant.types.Path.iterator(Path.java:698)
at
org.apache.tools.ant.types.resources.Union.getCollection(Union.java:123)
at org.apache.tools.ant.types.resources.Union.list(Union.java:86)
at org.apache.tools.ant.types.Path.list(Path.java:372)
at org.apache.tools.ant.types.Path.addExisting(Path.java:331)
at org.apache.tools.ant.types.Path.addExisting(Path.java:319)
at org.apache.tools.ant.types.Path.concatSpecialPath(Path.java:566)
at org.apache.tools.ant.types.Path.concatSystemClasspath(Path.java:526)
at
org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter.getCompileClasspath(DefaultCompilerAdapter.java:155)
at
org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter.setupJavacCommandlineSwitches(DefaultCompilerAdapter.java:183)
at
org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter.setupModernJavacCommandlineSwitches(DefaultCompilerAdapter.java:321)
at
org.apache.tools.ant.taskdefs.compilers.DefaultCompilerAdapter.setupModernJavacCommand(DefaultCompilerAdapter.java:368)
at
org.apache.tools.ant.taskdefs.compilers.Javac13.execute(Javac13.java:48)
at org.apache.tools.ant.taskdefs.Javac.compile(Javac.java:1097)
at org.apache.tools.ant.taskdefs.Javac.execute(Javac.java:906)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at
[jira] [Commented] (HIVE-2961) Remove need for storage descriptors for view partitions
[ https://issues.apache.org/jira/browse/HIVE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257292#comment-13257292 ] Carl Steinbach commented on HIVE-2961: -- Based on the discussion at today's contrib meeting it sounds like we can drop this patch and instead backout HIVE-2795 and HIVE-2612. Does that sound good? Remove need for storage descriptors for view partitions --- Key: HIVE-2961 URL: https://issues.apache.org/jira/browse/HIVE-2961 Project: Hive Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2961.D2877.1.patch Storage descriptors were introduced for view partitions as part of HIVE-2795. This was to allow view partitions to have the concept of a region as well as to fix a NPE that resulted from calling describe formatted on them. Since regions are no longer necessary for view partitions and the NPE can be fixed by not displaying storage information for view partitions (or displaying the view's storage information if this is preferred, although, since a view partition is purely metadata, this does not seem necessary), these are no longer needed. This also means the Python script added which retroactively adds storage descriptors to existing view partitions can be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2961) Remove need for storage descriptors for view partitions
[ https://issues.apache.org/jira/browse/HIVE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257294#comment-13257294 ] Carl Steinbach commented on HIVE-2961: -- But if we end up taking that route, I think we should leave the upgrade-0.8.0-to-0.9.0.xxx.sql scripts for the sake of consistency. Remove need for storage descriptors for view partitions --- Key: HIVE-2961 URL: https://issues.apache.org/jira/browse/HIVE-2961 Project: Hive Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2961.D2877.1.patch Storage descriptors were introduced for view partitions as part of HIVE-2795. This was to allow view partitions to have the concept of a region as well as to fix a NPE that resulted from calling describe formatted on them. Since regions are no longer necessary for view partitions and the NPE can be fixed by not displaying storage information for view partitions (or displaying the view's storage information if this is preferred, although, since a view partition is purely metadata, this does not seem necessary), these are no longer needed. This also means the Python script added which retroactively adds storage descriptors to existing view partitions can be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2961) Remove need for storage descriptors for view partitions
[ https://issues.apache.org/jira/browse/HIVE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257772#comment-13257772 ] Carl Steinbach commented on HIVE-2961: -- bq. Do you mean delete the content of the file, thus effectively leaving empty files, since there is no upgrade needed when we revert those patches? Yup, that's basically what I meant, but now that I think more about it I agree that it doesn't make much sense. Remove need for storage descriptors for view partitions --- Key: HIVE-2961 URL: https://issues.apache.org/jira/browse/HIVE-2961 Project: Hive Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2961.D2877.1.patch Storage descriptors were introduced for view partitions as part of HIVE-2795. This was to allow view partitions to have the concept of a region as well as to fix a NPE that resulted from calling describe formatted on them. Since regions are no longer necessary for view partitions and the NPE can be fixed by not displaying storage information for view partitions (or displaying the view's storage information if this is preferred, although, since a view partition is purely metadata, this does not seem necessary), these are no longer needed. This also means the Python script added which retroactively adds storage descriptors to existing view partitions can be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2931) conf settings may be ignored
[ https://issues.apache.org/jira/browse/HIVE-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253155#comment-13253155 ] Carl Steinbach commented on HIVE-2931: -- This ticket is a duplicate of HIVE-2918. I have provided a patch for HIVE-2918 which provides the same fix along with two new testcases that verify the behavior of hive.exec.max.dynamic.partitions. Please test and commit that patch instead. conf settings may be ignored Key: HIVE-2931 URL: https://issues.apache.org/jira/browse/HIVE-2931 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Kevin Wilfong Attachments: HIVE-2931.D2781.1.patch, hive.2931.1.patch This is a pretty serious problem. If a conf variable is changed, Hive may not pick up the variable unless the metastore variables are changed. When any session variables are changed, it might be simpler to update the corresponding Hive conf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2931) conf settings may be ignored
[ https://issues.apache.org/jira/browse/HIVE-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253158#comment-13253158 ] Carl Steinbach commented on HIVE-2931: -- Also, this patch doesn't add any tests to verify that it actually fixes the problem. Please either add tests or commit HIVE-2918 instead. conf settings may be ignored Key: HIVE-2931 URL: https://issues.apache.org/jira/browse/HIVE-2931 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Kevin Wilfong Attachments: HIVE-2931.D2781.1.patch, hive.2931.1.patch This is a pretty serious problem. If a conf variable is changed, Hive may not pick up the variable unless the metastore variables are changed. When any session variables are changed, it might be simpler to update the corresponding Hive conf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2918) Hive Dynamic Partition Insert - move task not considering 'hive.exec.max.dynamic.partitions' from CLI
[ https://issues.apache.org/jira/browse/HIVE-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253785#comment-13253785 ] Carl Steinbach commented on HIVE-2918: -- @Kevin: I tried running escape1.q and escape2.q on trunk and got diffs in both tests. Are you able to run either of these tests? Hive Dynamic Partition Insert - move task not considering 'hive.exec.max.dynamic.partitions' from CLI - Key: HIVE-2918 URL: https://issues.apache.org/jira/browse/HIVE-2918 Project: Hive Issue Type: Bug Affects Versions: 0.7.1, 0.8.0, 0.8.1 Environment: Cent OS 64 bit Reporter: Bejoy KS Assignee: Carl Steinbach Attachments: HIVE-2918.D2703.1.patch Dynamic Partition insert showing an error with the number of partitions created even after the default value of 'hive.exec.max.dynamic.partitions' is bumped high to 2000. Error Message: Failed with exception Number of dynamic partitions created is 1413, which is more than 1000. To solve this try to set hive.exec.max.dynamic.partitions to at least 1413. These are the following properties set on hive CLI hive set hive.exec.dynamic.partition=true; hive set hive.exec.dynamic.partition.mode=nonstrict; hive set hive.exec.max.dynamic.partitions=2000; hive set hive.exec.max.dynamic.partitions.pernode=2000; This is the query with console error log hive INSERT OVERWRITE TABLE partn_dyn Partition (pobox) SELECT country,state,pobox FROM non_partn_dyn; Total MapReduce jobs = 2 Launching Job 1 out of 2 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201204021529_0002, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201204021529_0002 Kill Command = /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201204021529_0002 2012-04-02 16:05:28,619 Stage-1 map = 0%, reduce = 0% 2012-04-02 16:05:39,701 Stage-1 map = 100%, reduce = 0% 2012-04-02 16:05:50,800 Stage-1 map = 100%, reduce = 100% Ended Job = job_201204021529_0002 Ended Job = 248865587, job is filtered out (removed at runtime). Moving data to: hdfs://0.0.0.0/tmp/hive-cloudera/hive_2012-04-02_16-05-24_919_5976014408587784412/-ext-1 Loading data to table default.partn_dyn partition (pobox=null) Failed with exception Number of dynamic partitions created is 1413, which is more than 1000. To solve this try to set hive.exec.max.dynamic.partitions to at least 1413. FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask I checked the job.xml of the first map only job, there the value hive.exec.max.dynamic.partitions=2000 is reflected but the move task is taking the default value from hive-site.xml . If I change the value in hive-site.xml then the job completes successfully. Bottom line,the property 'hive.exec.max.dynamic.partitions'set on CLI is not being considered by move task -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252905#comment-13252905 ] Carl Steinbach commented on HIVE-2646: -- bq. org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udfnull TestNegativeCliDriver.udfnull fails consistently for me. The cause of the failure is that the hive-contrib and hive_contrib JARs are getting included on the HIVE_HADOOP_TEST_CLASSPATH. @Andrew: Can you please modify the patch so that the hive-contrib and hive_contrib JARs are not included in HIVE_HADOOP_TEST_CLASSPATH? Thanks. Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs Key: HIVE-2646 URL: https://issues.apache.org/jira/browse/HIVE-2646 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Andrew Bayer Assignee: Andrew Bayer Priority: Critical Fix For: 0.9.0 Attachments: HIVE-2646.D2133.1.patch, HIVE-2646.D2133.10.patch, HIVE-2646.D2133.11.patch, HIVE-2646.D2133.12.patch, HIVE-2646.D2133.13.patch, HIVE-2646.D2133.14.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.D2133.4.patch, HIVE-2646.D2133.5.patch, HIVE-2646.D2133.6.patch, HIVE-2646.D2133.7.patch, HIVE-2646.D2133.8.patch, HIVE-2646.D2133.9.patch, HIVE-2646.diff.txt The current Hive Ivy dependency logic for its Hadoop dependencies is problematic - depending on the tarball and extracting the jars from there, rather than depending on the jars directly. It'd be great if this was fixed to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[
https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252929#comment-13252929
]
Carl Steinbach commented on HIVE-2646:
--
Here's the exception stack from the TestLazySimpleSerDe test failure:
{noformat}
[junit] Running org.apache.hadoop.hive.serde2.lazy.TestLazySimpleSerDe
[junit] java.lang.ArrayIndexOutOfBoundsException: -65
[junit] at
org.apache.commons.codec.binary.Base64.isBase64(Base64.java:137)
[junit] at
org.apache.commons.codec.binary.Base64.isArrayByteBase64(Base64.java:163)
[junit] at
org.apache.hadoop.hive.serde2.lazy.LazyBinary.init(LazyBinary.java:52)
[junit] at
org.apache.hadoop.hive.serde2.lazy.LazyStruct.uncheckedGetField(LazyStruct.java:219)
[junit] at
org.apache.hadoop.hive.serde2.lazy.LazyStruct.getField(LazyStruct.java:192)
[junit] at
org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldData(LazySimpleStructObjectInspector.java:188)
[junit] at
org.apache.hadoop.hive.serde2.lazy.TestLazySimpleSerDe.deserializeAndSerialize(TestLazySimpleSerDe.java:92)
[junit] at
org.apache.hadoop.hive.serde2.lazy.TestLazySimpleSerDe.testLazySimpleSerDe(TestLazySimpleSerDe.java:73)
{noformat}
The ArrayIndexOutOfBoundsException thrown by Base64.isBase64() is due to a bug
in commons-codec 1.3. This was fixed in CODEC-22 in commons-codec 1.4. Right
now the test classpath contains copies of versions 1.3 and 1.4 of
commons-codec, and the version 1.3 JAR precedes version 1.4. Ideally we can
prevent the 1.3 JAR from getting pulled down entirely, but at the least we need
to make sure it appears on the classpath after the v1.4 JAR.
@Andrew: can you please look into this? Thanks.
Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
Key: HIVE-2646
URL: https://issues.apache.org/jira/browse/HIVE-2646
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.8.0
Reporter: Andrew Bayer
Assignee: Andrew Bayer
Priority: Critical
Fix For: 0.9.0
Attachments: HIVE-2646.D2133.1.patch, HIVE-2646.D2133.10.patch,
HIVE-2646.D2133.11.patch, HIVE-2646.D2133.12.patch, HIVE-2646.D2133.13.patch,
HIVE-2646.D2133.14.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch,
HIVE-2646.D2133.4.patch, HIVE-2646.D2133.5.patch, HIVE-2646.D2133.6.patch,
HIVE-2646.D2133.7.patch, HIVE-2646.D2133.8.patch, HIVE-2646.D2133.9.patch,
HIVE-2646.diff.txt
The current Hive Ivy dependency logic for its Hadoop dependencies is
problematic - depending on the tarball and extracting the jars from there,
rather than depending on the jars directly. It'd be great if this was fixed
to actually have the jar dependencies defined directly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2928) Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo)
[ https://issues.apache.org/jira/browse/HIVE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251979#comment-13251979 ] Carl Steinbach commented on HIVE-2928: -- @Mithun: That's great news! Thanks for experimenting with this. Is the patch ready for review? If so can you please click the submit patch button? Thanks. Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo) - Key: HIVE-2928 URL: https://issues.apache.org/jira/browse/HIVE-2928 Project: Hive Issue Type: New Feature Components: Metastore Affects Versions: 0.9.0 Reporter: Mithun Radhakrishnan Attachments: HIVE-2928.patch I'm trying to get the Hive-Metastore to work when backed by an Oracle backend. There's a change to hive's package.jdo that I'd like advice/comments on. One sticking point on working with Oracle has been the TBLS table (MTable) and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT). Oracle doesn't support more than one LONGVARCHAR property per table (for reason of legacy), and prefers that one use CLOBs instead. If one switches to CLOB properties, with no modification to hive's package.jdo, one sees the following exception: quote Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException: Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. at org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521) at org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2 /quote But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of LONGVARCHARs, things look promising: 1. The exception no longer occurs. Things seem to work with Oracle. (I've yet to scale-test.) 2. These modified hive-libraries work as is with pre-existing mysql metastores. Migrating data isn't a worry. 3. The unit-tests seem to run through. Would there be opposition to changing the package.jdo's LONGVARCHAR references to CLOB, if this works with mysql and with Oracle? Mithun P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm testing, for the related issue of creating the required tables in Oracle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2928) Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo)
[ https://issues.apache.org/jira/browse/HIVE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250981#comment-13250981 ] Carl Steinbach commented on HIVE-2928: -- We currently have JDO schema validation disabled by default since it affects performance. However, if modifying Hive's package.jdo to work with Oracle means that we're forfeiting this feature, I'd at least like to know that up front. Also, there was an exchange a couple years ago on the hive-user list where John made an interesting suggestion: http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201006.mbox/%3cbd1fe08f-5edf-4d90-a741-8b703cd06...@facebook.com%3E bq. Another option is to precreate your schema in Oracle and then tell JDO not to try to create/update it automatically. Would you mind trying this out to see if it works? If it does then I think that might be the optimal solution for now. Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo) - Key: HIVE-2928 URL: https://issues.apache.org/jira/browse/HIVE-2928 Project: Hive Issue Type: New Feature Components: Metastore Affects Versions: 0.9.0 Reporter: Mithun Radhakrishnan Attachments: HIVE-2928.patch I'm trying to get the Hive-Metastore to work when backed by an Oracle backend. There's a change to hive's package.jdo that I'd like advice/comments on. One sticking point on working with Oracle has been the TBLS table (MTable) and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT). Oracle doesn't support more than one LONGVARCHAR property per table (for reason of legacy), and prefers that one use CLOBs instead. If one switches to CLOB properties, with no modification to hive's package.jdo, one sees the following exception: quote Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException: Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. at org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521) at org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2 /quote But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of LONGVARCHARs, things look promising: 1. The exception no longer occurs. Things seem to work with Oracle. (I've yet to scale-test.) 2. These modified hive-libraries work as is with pre-existing mysql metastores. Migrating data isn't a worry. 3. The unit-tests seem to run through. Would there be opposition to changing the package.jdo's LONGVARCHAR references to CLOB, if this works with mysql and with Oracle? Mithun P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm testing, for the related issue of creating the required tables in Oracle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2935) Implement HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250072#comment-13250072 ] Carl Steinbach commented on HIVE-2935: -- This is an umbrella JIRA for the HiveServer2 work. A proposal for the HiveServer2 Thrift API is available here: https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Thrift+API Implement HiveServer2 - Key: HIVE-2935 URL: https://issues.apache.org/jira/browse/HIVE-2935 Project: Hive Issue Type: New Feature Components: Server Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Labels: HiveServer2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2938) ALTER TABLE ADD IF NOT EXISTS PARTITION is not atomic
[ https://issues.apache.org/jira/browse/HIVE-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250162#comment-13250162 ] Carl Steinbach commented on HIVE-2938: -- Support for ALTER TABLE ADD IF NOT EXISTS PARTITION was added in HIVE-1106. However, this patch introduced a time-of-check-to-time-of-use bug since the check to see if the partition already exists is done in a transaction separate from the one in which the partition is actually created. We need to provide support for IF NOT EXISTS at the level of the Metastore Thrift API in order to support this correctly. ALTER TABLE ADD IF NOT EXISTS PARTITION is not atomic - Key: HIVE-2938 URL: https://issues.apache.org/jira/browse/HIVE-2938 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.8.1 Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1977) DESCRIBE TABLE syntax doesn't support specifying a database qualified table name
[
https://issues.apache.org/jira/browse/HIVE-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250193#comment-13250193
]
Carl Steinbach commented on HIVE-1977:
--
bq. 3. $describe column;
This doesn't make sense since tables in the same schema are allowed to have
columns with the same name. The correct syntax for describing a column is:
DESCRIBE table column
or
DESCRIBE db.table column
DESCRIBE TABLE syntax doesn't support specifying a database qualified table
name
Key: HIVE-1977
URL: https://issues.apache.org/jira/browse/HIVE-1977
Project: Hive
Issue Type: Bug
Components: Database/Schema, Query Processor, SQL
Reporter: Carl Steinbach
Assignee: Zhenxiao Luo
The syntax for DESCRIBE is broken. It should be:
{code}
DESCRIBE [EXTENDED] [database DOT]table [column]
{code}
but is actually
{code}
DESCRIBE [EXTENDED] table[DOT col_name]
{code}
Ref: http://dev.mysql.com/doc/refman/5.0/en/describe.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2928) Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo)
[
https://issues.apache.org/jira/browse/HIVE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250361#comment-13250361
]
Carl Steinbach commented on HIVE-2928:
--
@Mithun: What happens on MySQL if you turn column validation on (e.g.
datanucleus.validateColumns=true at startup)? There's a note on the Datanucleus
site that indicates this won't work:
http://www.datanucleus.org/products/datanucleus/rdbms/support.html
{quote}
You can specify BLOB, CLOB JDBC types when using MySQL with DataNucleus but
you must turn validation of columns OFF. This is because these types are not
supported by the MySQL JDBC driver and it returns them as
LONGVARBINARY/LONGVARCHAR when querying the column type.
{quote}
Support for Oracle-backed Hive-Metastore (longvarchar to clob in
package.jdo)
-
Key: HIVE-2928
URL: https://issues.apache.org/jira/browse/HIVE-2928
Project: Hive
Issue Type: New Feature
Components: Metastore
Affects Versions: 0.9.0
Reporter: Mithun Radhakrishnan
Attachments: HIVE-2928.patch
I'm trying to get the Hive-Metastore to work when backed by an Oracle
backend. There's a change to hive's package.jdo that I'd like advice/comments
on.
One sticking point on working with Oracle has been the TBLS table (MTable)
and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT).
Oracle doesn't support more than one LONGVARCHAR property per table (for
reason of legacy), and prefers that one use CLOBs instead. If one switches to
CLOB properties, with no modification to hive's package.jdo, one sees the
following exception:
quote
Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB
(datastore), but type expected was LONGVARCHAR (metadata). Please check that
the type in the datastore and the type specified in the MetaData are
consistent.
org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException:
Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB
(datastore), but type expected was LONGVARCHAR (metadata). Please check that
the type in the datastore and the type specified in the MetaData are
consistent.
at
org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521)
at
org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2
/quote
But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of
LONGVARCHARs, things look promising:
1. The exception no longer occurs. Things seem to work with Oracle. (I've yet
to scale-test.)
2. These modified hive-libraries work as is with pre-existing mysql
metastores. Migrating data isn't a worry.
3. The unit-tests seem to run through.
Would there be opposition to changing the package.jdo's LONGVARCHAR
references to CLOB, if this works with mysql and with Oracle?
Mithun
P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm
testing, for the related issue of creating the required tables in Oracle.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-80) Add testcases for concurrent query execution
[ https://issues.apache.org/jira/browse/HIVE-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249465#comment-13249465 ] Carl Steinbach commented on HIVE-80: HiveServer can't support concurrent connections due to a limitation of the current HiveServer Thrift API. There's a proposal for a new HiveServer2 Thrift API which fixes these problems located here: https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Thrift+API Add testcases for concurrent query execution Key: HIVE-80 URL: https://issues.apache.org/jira/browse/HIVE-80 Project: Hive Issue Type: Test Components: Query Processor, Server Infrastructure Reporter: Raghotham Murthy Assignee: Carl Steinbach Priority: Critical Labels: concurrency Attachments: hive_input_format_race-2.patch Can use one driver object per query. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247759#comment-13247759 ] Carl Steinbach commented on HIVE-2926: -- bin/hive uses Bash. I think we should do the same thing here. Also note that on most platforms /bin/sh actually maps to bash, or to something entirely different (e.g. https://wiki.ubuntu.com/DashAsBinSh). Expose some information about the metastore through JMX --- Key: HIVE-2926 URL: https://issues.apache.org/jira/browse/HIVE-2926 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.9.0 Attachments: metastore-jmx.patch Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247784#comment-13247784 ] Carl Steinbach commented on HIVE-2926: -- Sounds good. Please submit a phabricator request (https://cwiki.apache.org/Hive/phabricatorcodereview.html) when the patch is ready for review. Expose some information about the metastore through JMX --- Key: HIVE-2926 URL: https://issues.apache.org/jira/browse/HIVE-2926 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.9.0 Attachments: hive-2926.1.patch, metastore-jmx.patch Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2909) SHOW COLUMNS table_name; to provide a comma-delimited list of columns.
[ https://issues.apache.org/jira/browse/HIVE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247891#comment-13247891 ] Carl Steinbach commented on HIVE-2909: -- MySQL has a SHOW COLUMNS command: http://dev.mysql.com/doc/refman/5.0/en/show-columns.html Seems like we should follow convention and try to duplicate that instead of inventing our own behavior. Also, I think returning each column name in a separate row is much more convenient (and consistent) for clients than forcing them to tokenize the names by splitting on ',' SHOW COLUMNS table_name; to provide a comma-delimited list of columns. -- Key: HIVE-2909 URL: https://issues.apache.org/jira/browse/HIVE-2909 Project: Hive Issue Type: New Feature Reporter: Adam Kramer Assignee: Dikang Gu Priority: Minor Attachments: HIVE-2909.D2595.1.patch, HIVE-2909.D2595.2.patch, HIVE-2909.D2595.3.patch, HIVE-2909.D2595.4.patch Due to the way that SELECT * and partitioning works, it is frequently obnoxious to insert data into tables of the same schema. This could be fixed in a number of ways, all murky; this feature request reduces the obnoxicity of the current situation. SHOW COLUMNS foo; OK bar, baz, tball, ds ...then I could just copy the first three and not the last. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2920) TestStatsPublisherEnhanced throws NPE on JDBC connection failure
[
https://issues.apache.org/jira/browse/HIVE-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244501#comment-13244501
]
Carl Steinbach commented on HIVE-2920:
--
I have seen intermittent failures in TestStatsPublisherEnhanced with the
following stack:
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.Utilities.prepareWithRetry(Utilities.java:2176)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsAggregator.cleanUp(JDBCStatsAggregator.java:215)
at
org.apache.hadoop.hive.ql.exec.TestStatsPublisherEnhanced.setUp(TestStatsPublisherEnhanced.java:60)
at junit.framework.TestCase.runBare(TestCase.java:132)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
{noformat}
TestStatsPublisherEnhanced throws NPE on JDBC connection failure
Key: HIVE-2920
URL: https://issues.apache.org/jira/browse/HIVE-2920
Project: Hive
Issue Type: Bug
Components: Statistics, Testing Infrastructure
Reporter: Carl Steinbach
Assignee: Carl Steinbach
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2921) StatsPublisher and StatsAggregator interfaces should throw exceptions
[ https://issues.apache.org/jira/browse/HIVE-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244650#comment-13244650 ] Carl Steinbach commented on HIVE-2921: -- Currently all of the methods in the StatsAggregator and StatsPublisher interfaces use a boolean return value to indicate success or failure, which ends up hiding a lot of information when things go wrong since underlying methods are required to squelch exceptions. These methods should instead wrap and then re-throw the underlying exception. StatsPublisher and StatsAggregator interfaces should throw exceptions - Key: HIVE-2921 URL: https://issues.apache.org/jira/browse/HIVE-2921 Project: Hive Issue Type: Bug Components: Statistics Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2860) TestNegativeCliDriver autolocal1.q fails on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244912#comment-13244912 ] Carl Steinbach commented on HIVE-2860: -- @Ashutosh: Which tests failed? TestNegativeCliDriver autolocal1.q fails on 0.23 Key: HIVE-2860 URL: https://issues.apache.org/jira/browse/HIVE-2860 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.9.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.9.0 Attachments: HIVE-2860.D2253.1.patch, HIVE-2860.D2253.1.patch, HIVE-2860.D2565.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2819) Closed range scans on hbase keys
[ https://issues.apache.org/jira/browse/HIVE-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235848#comment-13235848 ] Carl Steinbach commented on HIVE-2819: -- +1. @Ashutosh: Please commit if tests pass. Thanks! Closed range scans on hbase keys - Key: HIVE-2819 URL: https://issues.apache.org/jira/browse/HIVE-2819 Project: Hive Issue Type: Improvement Components: HBase Handler Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-2819.D1923.1.patch, HIVE-2819.D1923.2.patch, HIVE-2819.D1923.3.patch This patch pushes range scans on keys of closed form into hbase -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1721) use bloom filters to improve the performance of joins
[ https://issues.apache.org/jira/browse/HIVE-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236248#comment-13236248 ] Carl Steinbach commented on HIVE-1721: -- @Srinivasan: Doesn't look like it. Feel free to have a go at it. use bloom filters to improve the performance of joins - Key: HIVE-1721 URL: https://issues.apache.org/jira/browse/HIVE-1721 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Labels: gsoc, gsoc2012, optimization In case of map-joins, it is likely that the big table will not find many matching rows from the small table. Currently, we perform a hash-map lookup for every row in the big table, which can be pretty expensive. It might be useful to try out a bloom-filter containing all the elements in the small table. Each element from the big table is first searched in the bloom filter, and only in case of a positive match, the small table hash table is explored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2877) TABLESAMPLE(x PERCENT) tests fail on 0.22/0.23
[ https://issues.apache.org/jira/browse/HIVE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233888#comment-13233888 ] Carl Steinbach commented on HIVE-2877: -- There are two distinct problems: 1) Many of the queries in split_sample.q and sample_islocalmode_hook.q are nondeterministic. This can be fixed by adding ORDER BY clauses. 2) The second problem is more serious. Both of the tests set mapred.max.split.size=300 and hive.merge.smallfiles.avgsize=1 in an effort to force the generation of multiple splits and multiple output files. However, Hadoop 0.20 is incapable of generating splits smaller than the block size when using CombineFileInputFormat, so only one split is generated. This has a significant impact on the results of the TABLESAMPLE(x PERCENT). This issue was fixed in MAPREDUCE-2046 which is included in 0.23. Suggested Fixes: # Make the queries deterministic # Restrict these tests to Hadoop versions = 0.22 TABLESAMPLE(x PERCENT) tests fail on 0.22/0.23 -- Key: HIVE-2877 URL: https://issues.apache.org/jira/browse/HIVE-2877 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Carl Steinbach Assignee: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2878) TestHBaseMinimrCliDriver hbase_bulk.m fails on 0.23
[
https://issues.apache.org/jira/browse/HIVE-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232989#comment-13232989
]
Carl Steinbach commented on HIVE-2878:
--
On Hadoop 0.20 the 'dfs -count' command outputs a fully qualified file location:
{noformat}
dfs -count /build/ql/test/data/warehouse/hbpartition;
13 23380
hdfs://localhost.localdomain:37780/tmp/hbsort/cf
{noformat}
But on 0.23 the file location is not qualified:
{noformat}
dfs -count /build/ql/test/data/warehouse/hbpartition;
13 24792 /tmp/hbsort/cf
{noformat}
This is a causing a test failure in TestHBaseMinimrCliDriver.hbase_bulk.m
TestHBaseMinimrCliDriver hbase_bulk.m fails on 0.23
---
Key: HIVE-2878
URL: https://issues.apache.org/jira/browse/HIVE-2878
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Carl Steinbach
Assignee: Carl Steinbach
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2878) TestHBaseMinimrCliDriver hbase_bulk.m fails on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232996#comment-13232996 ] Carl Steinbach commented on HIVE-2878: -- Note: This problem only affects the 0.8.x branch. Trunk already uses different code for masking test outputs that prevents this problem from occurring. TestHBaseMinimrCliDriver hbase_bulk.m fails on 0.23 --- Key: HIVE-2878 URL: https://issues.apache.org/jira/browse/HIVE-2878 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.8.1 Reporter: Carl Steinbach Assignee: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2778) Fail on table sampling
[
https://issues.apache.org/jira/browse/HIVE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13229698#comment-13229698
]
Carl Steinbach commented on HIVE-2778:
--
@Navis: Can you please grant license rights to the ASF? Patch looks good
otherwise. Thanks.
Fail on table sampling
---
Key: HIVE-2778
URL: https://issues.apache.org/jira/browse/HIVE-2778
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.9.0
Environment: Reproduced only on hadoop-0.20.2-CDH3u1, work fine on
hadoop-0.20.2
Reporter: Navis
Assignee: Navis
Fix For: 0.9.0
Attachments: HIVE-2778.D1593.1.patch, HIVE-2778.D1593.2.patch
Trying table sampling on any non-empty table throws NPE. This does not occur
by test on mini-MR.
{noformat}
select count(*) from emp tablesample (0.1 percent);
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapred.reduce.tasks=number
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.sampleSplits(CombineHiveInputFormat.java:450)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:403)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:971)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:963)
at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:432)
at
org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:134)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Job Submission failed with exception 'java.lang.NullPointerException(null)'
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MapRedTask
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2856) Fix TestCliDriver escape1.q failure on MR2
[ https://issues.apache.org/jira/browse/HIVE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227341#comment-13227341 ] Carl Steinbach commented on HIVE-2856: -- +1. Will commit if tests pass. Fix TestCliDriver escape1.q failure on MR2 -- Key: HIVE-2856 URL: https://issues.apache.org/jira/browse/HIVE-2856 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo Attachments: HIVE-2856.1.patch.txt, escape1.q.out Additional '^' in escape test: [junit] Begin query: escape1.q [junit] Copying file: file:/home/cloudera/Code/hive/data/files/escapetest.txt [junit] 12/01/23 15:22:15 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 15:22:15 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* more -I job_[0-9]*_[0-9]* -I USING 'java -cp /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/escape1.q.out /home/cloudera/Code/hive/ql/src/test/results/clientpositive/escape1.q.out [junit] 893d892 [junit] 1 1 ^ [junit] junit.framework.AssertionFailedError: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. [junit] at junit.framework.Assert.fail(Assert.java:50) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1(TestCliDriver.java:131) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.lang.reflect.Method.invoke(Method.java:616) [junit] at junit.framework.TestCase.runTest(TestCase.java:168) [junit] at junit.framework.TestCase.runBare(TestCase.java:134) [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) [junit] at junit.framework.TestResult.run(TestResult.java:113) [junit] at junit.framework.TestCase.run(TestCase.java:124) [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) [junit] at junit.framework.TestSuite.run(TestSuite.java:238) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) [junit] Exception: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2856) When integrating into MapReduce2, additional '^' in escape test
[ https://issues.apache.org/jira/browse/HIVE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13226716#comment-13226716 ] Carl Steinbach commented on HIVE-2856: -- @Zhenxiao: Please attach an updated copy of escape1.q.out. You can either attach this separately, or generate a new diff that includes binary changes using the command git diff --binary HEAD^ patch.txt. Thanks. When integrating into MapReduce2, additional '^' in escape test --- Key: HIVE-2856 URL: https://issues.apache.org/jira/browse/HIVE-2856 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo Attachments: HIVE-2856.1.patch.txt Additional '^' in escape test: [junit] Begin query: escape1.q [junit] Copying file: file:/home/cloudera/Code/hive/data/files/escapetest.txt [junit] 12/01/23 15:22:15 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 15:22:15 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* more -I job_[0-9]*_[0-9]* -I USING 'java -cp /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/escape1.q.out /home/cloudera/Code/hive/ql/src/test/results/clientpositive/escape1.q.out [junit] 893d892 [junit] 1 1 ^ [junit] junit.framework.AssertionFailedError: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. [junit] at junit.framework.Assert.fail(Assert.java:50) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1(TestCliDriver.java:131) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.lang.reflect.Method.invoke(Method.java:616) [junit] at junit.framework.TestCase.runTest(TestCase.java:168) [junit] at junit.framework.TestCase.runBare(TestCase.java:134) [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) [junit] at junit.framework.TestResult.run(TestResult.java:113) [junit] at junit.framework.TestCase.run(TestCase.java:124) [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) [junit] at junit.framework.TestSuite.run(TestSuite.java:238) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) [junit] Exception: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2860) TestNegativeCliDriver autolocal1.q fails on 0.23
[
https://issues.apache.org/jira/browse/HIVE-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13226749#comment-13226749
]
Carl Steinbach commented on HIVE-2860:
--
The expected result is:
{noformat}
PREHOOK: Output:
file:/data/users/jssarma/hive_trunk/build/ql/scratchdir/hive_2010-07-25_01-23-08_631_3140637565055726778/-mr-1
Job Submission failed with exception 'java.lang.RuntimeException(Not a
host:port pair: abracadabra)'
{noformat}
But on 0.23, the actual result is:
{noformat}
PREHOOK: Output:
file:/Users/carl/Work/repos/hive1/build/ql/scratchdir/hive_2012-03-09_17-27-12_806_7440178579833934359/-mr-1
Job Submission failed with exception 'java.lang.IllegalArgumentException(Does
not contain a valid host:port authority: abracadabra)'
{noformat}
QTestUtil already masks output lines containing the string
java.lang.RuntimeException. It looks like it should also mask occurrences of
java.lang.IllegalArgumentException.
TestNegativeCliDriver autolocal1.q fails on 0.23
Key: HIVE-2860
URL: https://issues.apache.org/jira/browse/HIVE-2860
Project: Hive
Issue Type: Bug
Components: Testing Infrastructure
Affects Versions: 0.9.0
Reporter: Carl Steinbach
Assignee: Carl Steinbach
Fix For: 0.9.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2857) QTestUtil.cleanUp() fails with FileNotException on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225808#comment-13225808 ] Carl Steinbach commented on HIVE-2857: -- On 0.23 FileSystem.listStatus() throws FileNotFoundException instead of returning null. This causes the MinimrCliDriver tests to fail since QTestUtil.cleanUp is called before the warehouse directory is created. QTestUtil.cleanUp() fails with FileNotException on 0.23 --- Key: HIVE-2857 URL: https://issues.apache.org/jira/browse/HIVE-2857 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.9.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.9.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1634) Allow access to Primitive types stored in binary format in HBase
[ https://issues.apache.org/jira/browse/HIVE-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225071#comment-13225071 ] Carl Steinbach commented on HIVE-1634: -- +1. @Ashutosh: can you please commit this? Thanks. Allow access to Primitive types stored in binary format in HBase Key: HIVE-1634 URL: https://issues.apache.org/jira/browse/HIVE-1634 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.7.0, 0.8.0, 0.9.0 Reporter: Basab Maulik Assignee: Ashutosh Chauhan Fix For: 0.9.0 Attachments: HIVE-1634.0.patch, HIVE-1634.1.patch, HIVE-1634.D1581.1.patch, HIVE-1634.D1581.2.patch, HIVE-1634.D1581.3.patch, TestHiveHBaseExternalTable.java, hive-1634_3.patch This addresses HIVE-1245 in part, for atomic or primitive types. The serde property hbase.columns.storage.types = -,b,b,b,b,b,b,b,b is a specification of the storage option for the corresponding column in the serde property hbase.columns.mapping. Allowed values are '-' for table default, 's' for standard string storage, and 'b' for binary storage as would be obtained from o.a.h.hbase.utils.Bytes. Map types for HBase column families use a colon separated pair such as 's:b' for the key and value part specifiers respectively. See the test cases and queries for HBase handler for additional examples. There is also a table property hbase.table.default.storage.type = string to specify a table level default storage type. The other valid specification is binary. The table level default is overridden by a column level specification. This control is available for the boolean, tinyint, smallint, int, bigint, float, and double primitive types. The attached patch also relaxes the mapping of map types to HBase column families to allow any primitive type to be the map key. Attached is a program for creating a table and populating it in HBase. The external table in Hive can access the data as shown in the example below. hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties (hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double) tblproperties (hbase.table.name = TestHiveHBaseExternalTable); OK Time taken: 0.691 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 NULLNULLNULLNULLNULLTest-String NULLNULL Time taken: 0.346 seconds hive drop table TestHiveHBaseExternalTable; OK Time taken: 0.139 seconds hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ( hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double, hbase.columns.storage.types = -,b,b,b,b,b,b,b,b ) tblproperties ( hbase.table.name = TestHiveHBaseExternalTable, hbase.table.default.storage.type = string); OK Time taken: 0.139 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 true-128-32768 -2147483648 -9223372036854775808 Test-String -2.1793132E-11 2.01345E291 Time taken: 0.151 seconds hive drop table TestHiveHBaseExternalTable; OK Time taken: 0.154 seconds hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ( hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double, hbase.columns.storage.types = -,b,b,b,b,b,-,b,b ) tblproperties (hbase.table.name = TestHiveHBaseExternalTable); OK Time taken: 0.347 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 true-128-32768 -2147483648 -9223372036854775808 Test-String -2.1793132E-11 2.01345E291 Time taken: 0.245 seconds hive -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2840) INPUT__FILE__NAME virtual column returns unqualified paths on Hadoop 0.23
[
https://issues.apache.org/jira/browse/HIVE-2840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13222983#comment-13222983
]
Carl Steinbach commented on HIVE-2840:
--
On 0.20.1 the values generated by the INPUT__FILE__NAME virtual column are
fully qualified:
{noformat}
hive SELECT INPUT__FILE__NAME FROM src LIMIT 1;
file:/user/hive/warehouse/src/kv1.txt
{noformat}
However, the same query on 0.23.1 produces unqualified results:
{noformat}
hive SELECT INPUT__FILE__NAME FROM src LIMIT 1;
/user/hive/warehouse/src/kv1.txt
{noformat}
INPUT__FILE__NAME virtual column returns unqualified paths on Hadoop 0.23
-
Key: HIVE-2840
URL: https://issues.apache.org/jira/browse/HIVE-2840
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Carl Steinbach
Assignee: Carl Steinbach
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2831) TestContribCliDriver.dboutput and TestCliDriver.input45 fail on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220452#comment-13220452 ] Carl Steinbach commented on HIVE-2831: -- @Ed: I think in the past we have handled problems like this by updating the list of strings that diff should ignore. The patch I just attached does the same. TestContribCliDriver.dboutput and TestCliDriver.input45 fail on 0.23 Key: HIVE-2831 URL: https://issues.apache.org/jira/browse/HIVE-2831 Project: Hive Issue Type: Bug Components: Tests Reporter: Carl Steinbach Assignee: Carl Steinbach Attachments: HIVE-2831.1.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2833) Fix test failures caused by HIVE-2716
[
https://issues.apache.org/jira/browse/HIVE-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220553#comment-13220553
]
Carl Steinbach commented on HIVE-2833:
--
{quote}
@Ashutosh, it broke facebook's internal branch.
Did you test it with mysql db - I haven't debugged it, but reverting this fixes
the problem.
{quote}
Fix test failures caused by HIVE-2716
-
Key: HIVE-2833
URL: https://issues.apache.org/jira/browse/HIVE-2833
Project: Hive
Issue Type: Bug
Components: Metastore
Reporter: Carl Steinbach
Assignee: Enis Soztutar
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2831) TestContribCliDriver.dboutput and TestCliDriver.input45 fail on 0.23
[
https://issues.apache.org/jira/browse/HIVE-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218846#comment-13218846
]
Carl Steinbach commented on HIVE-2831:
--
Running the 'dfs' command on 0.20 produces the following output:
{noformat}
hive dfs -rmr ../build/test_dboutput_db;
Deleted file:/data/users/sdong/www/open-source-hive3/build/test_dboutput_db
{noformat}
But on 0.23 the output of FsShell commands has changed slightly:
{noformat}
hive dfs -rmr ../build/test_dboutput_db;
Deleted ../build/test_dboutput_db
{noformat}
This difference is causing test failures in TestConbribCliDriver.dboutput and
TestCliDriver.input45
TestContribCliDriver.dboutput and TestCliDriver.input45 fail on 0.23
Key: HIVE-2831
URL: https://issues.apache.org/jira/browse/HIVE-2831
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Carl Steinbach
Assignee: Carl Steinbach
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2810) Implement NULL-safe equality operator =
[ https://issues.apache.org/jira/browse/HIVE-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13217081#comment-13217081 ] Carl Steinbach commented on HIVE-2810: -- I think it's fine to separate if there's a way to disable = from appearing in JOIN conditions. Otherwise, think it needs to be fixed here. Implement NULL-safe equality operator = - Key: HIVE-2810 URL: https://issues.apache.org/jira/browse/HIVE-2810 Project: Hive Issue Type: New Feature Components: Query Processor, UDF Affects Versions: 0.9.0 Reporter: Carl Steinbach Assignee: Navis Fix For: 0.9.0 Attachments: HIVE-2810.D1791.1.patch Ref: http://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html#operator_equal-to -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215020#comment-13215020 ] Carl Steinbach commented on HIVE-2811: -- @Ed: Can you please attach a copy of the patch and grant license rights to the ASF? Thanks! Export LANG=en_US.UTF-8 to environment while running tests -- Key: HIVE-2811 URL: https://issues.apache.org/jira/browse/HIVE-2811 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.9.0 Reporter: Ashutosh Chauhan Assignee: Edward Capriolo Fix For: 0.9.0 Attachments: HIVE-2811.D1905.1.patch Most OS has this value by default, but in some cases (on apache build machines) it has a different value making inputddl5.q test to fail. This variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215053#comment-13215053 ] Carl Steinbach commented on HIVE-2811: -- Unfortunately, yes. I think John is looking into making arc do this automatically. In the meantime, sorry for the inconvenience. Export LANG=en_US.UTF-8 to environment while running tests -- Key: HIVE-2811 URL: https://issues.apache.org/jira/browse/HIVE-2811 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.9.0 Reporter: Ashutosh Chauhan Assignee: Edward Capriolo Fix For: 0.9.0 Attachments: HIVE-2811.D1905.1.patch, HIVE-2811.D1905.2.patch Most OS has this value by default, but in some cases (on apache build machines) it has a different value making inputddl5.q test to fail. This variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2803) utc_from_timestamp and utc_to_timestamp returns incorrect results.
[
https://issues.apache.org/jira/browse/HIVE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214221#comment-13214221
]
Carl Steinbach commented on HIVE-2803:
--
@tamtam180: If this patch is ready for review can you please change the status
to 'Patch Available' and submit a review request using Phabricator?
Instructions are located here:
https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-ReviewProcess
Thanks!
utc_from_timestamp and utc_to_timestamp returns incorrect results.
--
Key: HIVE-2803
URL: https://issues.apache.org/jira/browse/HIVE-2803
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.8.0
Reporter: tamtam180
Attachments: HIVE-2803.1.patch.txt
How to reproduce:
{noformat}
$ echo 2011-12-25 09:00:00.123456 /tmp/data5.txt
hive create table ts1(t1 timestamp);
hive load data local inpath '/tmp/data5.txt' overwrite into table ts1;
hive select t1, from_utc_timestamp(t1, 'JST'), from_utc_timestamp(t1, 'JST')
from ts1 limit 1;
{noformat}
The following result is expected:
{noformat}
2011-12-25 09:00:00.123456 2011-12-25 18:00:00.123456 2011-12-25
18:00:00.123456
{noformat}
However, the above query return incorrect result like this:
{noformat}
2011-12-26 03:00:00.492456 2011-12-26 03:00:00.492456 2011-12-26
03:00:00.492456
{noformat}
This is because GenericUDFFromUtcTimestamp.applyOffset() does setTime()
improperly.
On evaluating query, timestamp argument always returns the same instance.
GenericUDFFromUtcTimestamp.applyOffset() does setTime() on the instance.
That means it adds all offsets in the query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2504) Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory
[ https://issues.apache.org/jira/browse/HIVE-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214376#comment-13214376 ] Carl Steinbach commented on HIVE-2504: -- @Sho: Can you please file a JIRA for this? Thanks! Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory - Key: HIVE-2504 URL: https://issues.apache.org/jira/browse/HIVE-2504 Project: Hive Issue Type: Bug Components: Metastore Reporter: Carl Steinbach Assignee: Chinna Rao Lalam Fix For: 0.9.0 Attachments: HIVE-2504.patch When the Hive Metastore creates a subdirectory in the Hive warehouse for a new table it does so with the default HDFS permissions. Since the default dfs.umask value is 022, this means that the new subdirectory will not inherit the group write permissions of the hive warehouse directory. We should make the umask used by Warehouse.mkdirs() configurable, and set it to use a default value of 002. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2810) Implement NULL-safe equality operator =
[ https://issues.apache.org/jira/browse/HIVE-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214381#comment-13214381 ] Carl Steinbach commented on HIVE-2810: -- Have you looked at the code changes made in HIVE-741? My understanding is that before that patch was committed Hive actually treated equi-joins as null-safe-equijoins, so it seems like the patch should give you a pretty good idea of what needs to be modified. Implement NULL-safe equality operator = - Key: HIVE-2810 URL: https://issues.apache.org/jira/browse/HIVE-2810 Project: Hive Issue Type: New Feature Components: Query Processor, UDF Affects Versions: 0.9.0 Reporter: Carl Steinbach Assignee: Navis Fix For: 0.9.0 Attachments: HIVE-2810.D1791.1.patch Ref: http://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html#operator_equal-to -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-895) Add SerDe for Avro serialized data
[ https://issues.apache.org/jira/browse/HIVE-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13212335#comment-13212335 ] Carl Steinbach commented on HIVE-895: - @Jakob: Do you think we can get this committed in time for the 0.9.0 release? Are you willing to attach a patch? Thanks. Add SerDe for Avro serialized data -- Key: HIVE-895 URL: https://issues.apache.org/jira/browse/HIVE-895 Project: Hive Issue Type: New Feature Components: Serializers/Deserializers Reporter: Jeff Hammerbacher Assignee: Jakob Homan As Avro continues to mature, having a SerDe to allow HiveQL queries over Avro data seems like a solid win. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2799) change the following thrift apis to add a region
[ https://issues.apache.org/jira/browse/HIVE-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207193#comment-13207193 ] Carl Steinbach commented on HIVE-2799: -- I recommend adding methods that take a single request object as input, e.g: void create_table(1: CreateTableReq req) throws (1:AlreadyExistsException o1, 2:InvalidObjectException o2, 3:MetaException o3, 4:NoSuchObjectException o4) My understanding of Thrift is that new properties can be added to the CreateTableReq object without breaking backwards compatibility. change the following thrift apis to add a region Key: HIVE-2799 URL: https://issues.apache.org/jira/browse/HIVE-2799 Project: Hive Issue Type: New Feature Reporter: Namit Jain Assignee: Kevin Wilfong liststring get_tables(1: string db_name, 2: string pattern) throws (1: MetaException o1) liststring get_all_tables(1: string db_name) throws (1: MetaException o1) Table get_table(1:string dbname, 2:string tbl_name) throws (1:MetaException o1, 2:NoSuchObjectException o2) listTable get_table_objects_by_name(1:string dbname, 2:liststring tbl_names) throws (1:MetaException o1, 2:InvalidOperationException o2, 3:UnknownDBException o3) liststring get_table_names_by_filter(1:string dbname, 2:string filter, 3:i16 max_tables=-1) throws (1:MetaException o1, 2:InvalidOperationException o2, 3:UnknownDBException o3) Partition add_partition(1:Partition new_part) throws(1:InvalidObjectException o1, 2:AlreadyExistsException o2, 3:MetaException o3) i32 add_partitions(1:listPartition new_parts) throws(1:InvalidObjectException o1, 2:AlreadyExistsException o2, 3:MetaException o3) Partition append_partition(1:string db_name, 2:string tbl_name, 3:liststring part_vals) throws (1:InvalidObjectException o1, 2:AlreadyExistsException o2, 3:MetaException o3) Partition append_partition_by_name(1:string db_name, 2:string tbl_name, 3:string part_name) throws (1:InvalidObjectException o1, 2:AlreadyExistsException o2, 3:MetaException o3) bool drop_partition(1:string db_name, 2:string tbl_name, 3:liststring part_vals, 4:bool deleteData) throws(1:NoSuchObjectException o1, 2:MetaException o2) bool drop_partition_by_name(1:string db_name, 2:string tbl_name, 3:string part_name, 4:bool deleteData) throws(1:NoSuchObjectException o1, 2:MetaException o2) Partition get_partition(1:string db_name, 2:string tbl_name, 3:liststring part_vals) throws(1:MetaException o1, 2:NoSuchObjectException o2) Partition get_partition_with_auth(1:string db_name, 2:string tbl_name, 3:liststring part_vals, 4: string user_name, 5: liststring group_names) throws(1:MetaException o1, 2:NoSuchObjectException o2) Partition get_partition_by_name(1:string db_name 2:string tbl_name, 3:string part_name) throws(1:MetaException o1, 2:NoSuchObjectException o2) listPartition get_partitions(1:string db_name, 2:string tbl_name, 3:i16 max_parts=-1) throws(1:NoSuchObjectException o1, 2:MetaException o2) listPartition get_partitions_with_auth(1:string db_name, 2:string tbl_name, 3:i16 max_parts=-1, 4: string user_name, 5: liststring group_names) throws(1:NoSuchObjectException o1, 2:MetaException o2) liststring get_partition_names(1:string db_name, 2:string tbl_name, 3:i16 max_parts=-1) throws(1:MetaException o2) listPartition get_partitions_ps(1:string db_name 2:string tbl_name 3:liststring part_vals, 4:i16 max_parts=-1) throws(1:MetaException o1, 2:NoSuchObjectException o2) listPartition get_partitions_ps_with_auth(1:string db_name, 2:string tbl_name, 3:liststring part_vals, 4:i16 max_parts=-1, 5: string user_name, 6: liststring group_names) throws(1:NoSuchObjectException o1, 2:MetaException o2) liststring get_partition_names_ps(1:string db_name, 2:string tbl_name, 3:liststring part_vals, 4:i16 max_parts=-1) throws(1:MetaException o1, 2:NoSuchObjectException o2) listPartition get_partitions_by_filter(1:string db_name 2:string tbl_name 3:string filter, 4:i16 max_parts=-1) throws(1:MetaException o1, 2:NoSuchObjectException o2) listPartition get_partitions_by_names(1:string db_name 2:string tbl_name 3:liststring names) throws(1:MetaException o1, 2:NoSuchObjectException o2) bool drop_index_by_name(1:string db_name, 2:string tbl_name, 3:string index_name, 4:bool deleteData)
[jira] [Commented] (HIVE-2804) Task log retrieval fails on secure cluster
[
https://issues.apache.org/jira/browse/HIVE-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207354#comment-13207354
]
Carl Steinbach commented on HIVE-2804:
--
On a secure Hadoop cluster TaskLogProcessor.getErrors() fails:
{noformat}
Ended Job = job_201202112025_0356 with errors
Error during job, obtaining debugging information...
Examining task ID: task_201202112025_0356_m_03 (and more) from job
job_201202112025_0356
Exception in thread Thread-22 java.lang.RuntimeException: Error while reading
from task log url
at
org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
at
org.apache.hadoop.hive.ql.exec.JobDebugger.showJobFailDebugInfo(JobDebugger.java:211)
at org.apache.hadoop.hive.ql.exec.JobDebugger.run(JobDebugger.java:81)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Server returned HTTP response code: 400 for
URL:
http://c0509.hal.cloudera.com:50060/tasklog?taskid=attempt_201202112025_0356_r_00_1start=-8193
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
at java.net.URL.openStream(URL.java:1010)
at
org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
... 3 more
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
{noformat}
Task log retrieval fails on secure cluster
--
Key: HIVE-2804
URL: https://issues.apache.org/jira/browse/HIVE-2804
Project: Hive
Issue Type: Bug
Components: Diagnosability, Query Processor, Security
Reporter: Carl Steinbach
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2782) New BINARY type produces unexpected results with supported UDFS when using MapReduce2
[ https://issues.apache.org/jira/browse/HIVE-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13204092#comment-13204092 ] Carl Steinbach commented on HIVE-2782: -- @Zhenxiao: Can you please explain how this is an improvement on the version I provided in my patch? New BINARY type produces unexpected results with supported UDFS when using MapReduce2 - Key: HIVE-2782 URL: https://issues.apache.org/jira/browse/HIVE-2782 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Carl Steinbach Attachments: HIVE-2782.D1653.1.patch, HIVE-2782.D1653.1.patch When using MapReduce2 for Hive ba_table_udfs is failing with unexpected output: [junit] Begin query: ba_table_udfs.q [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* more -I job_[0-9]*_[0-9]* -I USING 'java -cp /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/ba_table_udfs.q.out /home/cloudera/Code/hive/ql/src/test/results/clientpositive/ba_table_udfs.q.out [junit] 20,26c20,26 [junit] 2 10val_101 [junit] 3 164val_164 1 [junit] 3 150val_150 1 [junit] 2 18val_181 [junit] 3 177val_177 1 [junit] 2 12val_121 [junit] 2 11val_111 [junit] — [junit] 3 120val_120 1 [junit] 3 192val_192 1 [junit] 3 119val_119 1 [junit] 3 187val_187 1 [junit] 3 176val_176 1 [junit] 3 199val_199 1 [junit] 3 118val_118 1 [junit] Exception: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. [junit] junit.framework.AssertionFailedError: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. [junit] at junit.framework.Assert.fail(Assert.java:50) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs(TestCliDriver.java:129) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.lang.reflect.Method.invoke(Method.java:616) [junit] at junit.framework.TestCase.runTest(TestCase.java:168) [junit] at junit.framework.TestCase.runBare(TestCase.java:134) [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) [junit] at junit.framework.TestResult.run(TestResult.java:113) [junit] at junit.framework.TestCase.run(TestCase.java:124) [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) [junit] at junit.framework.TestSuite.run(TestSuite.java:238) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs.) [junit] Cleaning up TestCliDriver [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 10.751 sec [junit] Test org.apache.hadoop.hive.cli.TestCliDriver FAILED [for] /home/cloudera/Code/hive/ql/build.xml: The following error occurred while executing this line: [for] /home/cloudera/Code/hive/build.xml:328: The following error occurred while executing this line: [for] /home/cloudera/Code/hive/build-common.xml:453: Tests failed! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2782) New BINARY type produces unexpected results with supported UDFS when using MapReduce2
[ https://issues.apache.org/jira/browse/HIVE-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13204202#comment-13204202 ] Carl Steinbach commented on HIVE-2782: -- @Ashutosh: I also found what I think is a bug in the way SUBSTR handles BINARY inputs. Can you please take a look? https://issues.apache.org/jira/browse/HIVE-2792 New BINARY type produces unexpected results with supported UDFS when using MapReduce2 - Key: HIVE-2782 URL: https://issues.apache.org/jira/browse/HIVE-2782 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Carl Steinbach Attachments: HIVE-2782.D1653.1.patch, HIVE-2782.D1653.1.patch When using MapReduce2 for Hive ba_table_udfs is failing with unexpected output: [junit] Begin query: ba_table_udfs.q [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* more -I job_[0-9]*_[0-9]* -I USING 'java -cp /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/ba_table_udfs.q.out /home/cloudera/Code/hive/ql/src/test/results/clientpositive/ba_table_udfs.q.out [junit] 20,26c20,26 [junit] 2 10val_101 [junit] 3 164val_164 1 [junit] 3 150val_150 1 [junit] 2 18val_181 [junit] 3 177val_177 1 [junit] 2 12val_121 [junit] 2 11val_111 [junit] — [junit] 3 120val_120 1 [junit] 3 192val_192 1 [junit] 3 119val_119 1 [junit] 3 187val_187 1 [junit] 3 176val_176 1 [junit] 3 199val_199 1 [junit] 3 118val_118 1 [junit] Exception: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. [junit] junit.framework.AssertionFailedError: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. [junit] at junit.framework.Assert.fail(Assert.java:50) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs(TestCliDriver.java:129) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.lang.reflect.Method.invoke(Method.java:616) [junit] at junit.framework.TestCase.runTest(TestCase.java:168) [junit] at junit.framework.TestCase.runBare(TestCase.java:134) [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) [junit] at junit.framework.TestResult.run(TestResult.java:113) [junit] at junit.framework.TestCase.run(TestCase.java:124) [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) [junit] at junit.framework.TestSuite.run(TestSuite.java:238) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs.) [junit] Cleaning up TestCliDriver [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 10.751 sec [junit] Test org.apache.hadoop.hive.cli.TestCliDriver FAILED [for] /home/cloudera/Code/hive/ql/build.xml: The following error occurred while executing this line: [for] /home/cloudera/Code/hive/build.xml:328: The following error occurred while executing this line: [for] /home/cloudera/Code/hive/build-common.xml:453: Tests failed! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2793) Disable loadpart_err.q on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13204224#comment-13204224 ] Carl Steinbach commented on HIVE-2793: -- This testcase was previously disabled for all Hadoop versions in HIVE-1491. On 0.23 this test fails with the same error as the one produced on 0.20 (if enabled). Disable loadpart_err.q on 0.23 -- Key: HIVE-2793 URL: https://issues.apache.org/jira/browse/HIVE-2793 Project: Hive Issue Type: Bug Components: Testing Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2776) Setting test.silent=false has not effect
[
https://issues.apache.org/jira/browse/HIVE-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199175#comment-13199175
]
Carl Steinbach commented on HIVE-2776:
--
QTestUtil and the various CliDriver templates all print out the following help
message:
{noformat}
try ant test ... -Dtest.silent=false to get more logs.
{noformat}
However, it turns out that test.silent=false by default, and you change the
behavior of the tests only by setting test.silent=true.
From MapredLocalTask:
{noformat}
String isSilent =
true.equalsIgnoreCase(System.getProperty(test.silent)) ? -nolog : ;
{noformat}
From MapRedTask:
{noformat}
String isSilent = true.equalsIgnoreCase(System
.getProperty(test.silent)) ? -nolog : ;
{noformat}
Setting test.silent=false has not effect
Key: HIVE-2776
URL: https://issues.apache.org/jira/browse/HIVE-2776
Project: Hive
Issue Type: Bug
Components: Testing Infrastructure
Reporter: Carl Steinbach
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2743) Enable SASL mode for HiveServer
[ https://issues.apache.org/jira/browse/HIVE-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196627#comment-13196627 ] Carl Steinbach commented on HIVE-2743: -- I'm planning to work on it, but probably won't be able to start for a couple of weeks. Enable SASL mode for HiveServer --- Key: HIVE-2743 URL: https://issues.apache.org/jira/browse/HIVE-2743 Project: Hive Issue Type: New Feature Components: Security, Server Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2749) CONV returns incorrect results sometimes
[ https://issues.apache.org/jira/browse/HIVE-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196426#comment-13196426 ] Carl Steinbach commented on HIVE-2749: -- Is this ready for review? CONV returns incorrect results sometimes Key: HIVE-2749 URL: https://issues.apache.org/jira/browse/HIVE-2749 Project: Hive Issue Type: Bug Reporter: Jonathan Chang Attachments: HIVE-2749.D1437.1.patch, HIVE-2749.D1437.2.patch ...because it fails to reset state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2753) Remove empty java files
[
https://issues.apache.org/jira/browse/HIVE-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196428#comment-13196428
]
Carl Steinbach commented on HIVE-2753:
--
@Owen: Can you try submitting this patch with arc? It would be good to see if
arc handles deleted files properly. Thanks.
Remove empty java files
---
Key: HIVE-2753
URL: https://issues.apache.org/jira/browse/HIVE-2753
Project: Hive
Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Attachments: h-2753.patch
When looking at the 0.8.1 rc1, I discovered there were a set of empty Java
files that were likely left over from using 'patch' without the -E.
{quote}
jdbc/src/java/org/apache/hadoop/hive/jdbc/JdbcSessionState.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeIndexEvaluator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinObject.java
ql/src/java/org/apache/hadoop/hive/ql/exec/PathUtil.java
ql/src/java/org/apache/hadoop/hive/ql/exec/TypedBytesRecordReader.java
ql/src/java/org/apache/hadoop/hive/ql/plan/AlterPartitionProtectModeDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/TouchDesc.java
ql/src/test/org/apache/hadoop/hive/ql/plan/TestAddPartition.java
serde/src/gen-java/org/apache/hadoop/hive/serde/test/Constants.java
shims/src/0.20/java/org/apache/hadoop/fs/ProxyFileSystem.java
shims/src/0.20/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2749) CONV returns incorrect results sometimes
[ https://issues.apache.org/jira/browse/HIVE-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196528#comment-13196528 ] Carl Steinbach commented on HIVE-2749: -- Great! Can you change the status to Patch Available? CONV returns incorrect results sometimes Key: HIVE-2749 URL: https://issues.apache.org/jira/browse/HIVE-2749 Project: Hive Issue Type: Bug Reporter: Jonathan Chang Attachments: HIVE-2749.D1437.1.patch, HIVE-2749.D1437.2.patch ...because it fails to reset state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2761) Remove lib/javaewah-0.3.jar
[ https://issues.apache.org/jira/browse/HIVE-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13195881#comment-13195881 ] Carl Steinbach commented on HIVE-2761: -- We're not managing javaewah with Ivy because it's not available from any of the public Maven repos (as far as I know). Remove lib/javaewah-0.3.jar --- Key: HIVE-2761 URL: https://issues.apache.org/jira/browse/HIVE-2761 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0, 0.8.1, 0.9.0 Reporter: Ashutosh Chauhan Fix For: 0.9.0 After HIVE-2391 it is retrieved from maven repo via ivy, we can get rid of it from our lib/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2760) TestCliDriver should log elapsed time
[
https://issues.apache.org/jira/browse/HIVE-2760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13195168#comment-13195168
]
Carl Steinbach commented on HIVE-2760:
--
It would be nice if TestCliDriver logged the elapsed time of each testcase to
stdout, e.g:
{noformat}
[junit] Begin query: alter2.q
[junit] diff -a
/Users/carl/Work/repos/hive2/build/ql/test/logs/clientpositive/alter2.q.out
/Users/carl/Work/repos/hive2/ql/src/test/results/clientpositive/alter2.q.out
[junit] Done query: alter2.q elapsedTime=5s
{noformat}
TestCliDriver should log elapsed time
-
Key: HIVE-2760
URL: https://issues.apache.org/jira/browse/HIVE-2760
Project: Hive
Issue Type: Improvement
Components: Testing Infrastructure
Reporter: Carl Steinbach
Assignee: Carl Steinbach
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-123) refactor DDL code (both DDLWork and DDLTask)
[ https://issues.apache.org/jira/browse/HIVE-123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13195251#comment-13195251 ] Carl Steinbach commented on HIVE-123: - -1. Supplied patch has nothing to do with DDL code. Please resubmit against the right JIRA issue. refactor DDL code (both DDLWork and DDLTask) Key: HIVE-123 URL: https://issues.apache.org/jira/browse/HIVE-123 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: Ashutosh Chauhan Labels: cleanup, refactoring Attachments: HIVE-123.D1485.1.patch, hive-123.patch It might be good to break DDLTask into separate tasks. The abstract class DDLWork can have various subclasses: showTablesWork, DescribeTableWork etc. and a separate task for each of them. This will make them completely independent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2743) Enable SASL mode for HiveServer
[ https://issues.apache.org/jira/browse/HIVE-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13192413#comment-13192413 ] Carl Steinbach commented on HIVE-2743: -- HIVE-842 made it possible to run the MetaStoreServer in SASL mode. We need to enable the same mode for the HiveServer Thrift interface. Enable SASL mode for HiveServer --- Key: HIVE-2743 URL: https://issues.apache.org/jira/browse/HIVE-2743 Project: Hive Issue Type: New Feature Components: Security, Server Infrastructure Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[ https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13192563#comment-13192563 ] Carl Steinbach commented on HIVE-2698: -- Patches should apply cleanly with -p0. Please see this page for more info: https://cwiki.apache.org/Hive/howtocontribute.html Enable Hadoop-1.0.0 in Hive --- Key: HIVE-2698 URL: https://issues.apache.org/jira/browse/HIVE-2698 Project: Hive Issue Type: New Feature Components: Security, Shims Affects Versions: 0.9.0 Reporter: Enis Soztutar Assignee: Enis Soztutar Labels: hadoop, hadoop-1.0, jars Attachments: HIVE-2698.D1389.1.patch, HIVE-2698_v1.patch, HIVE-2698_v2.patch, HIVE-2698_v3.patch Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the 0.20S release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2744) Remove build dependency on Hadoop tarballs
[ https://issues.apache.org/jira/browse/HIVE-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13192582#comment-13192582 ] Carl Steinbach commented on HIVE-2744: -- The Hive build currently satisfies its Hadoop dependencies by pulling down a complete Hadoop tarball. Instead, it should use Ivy to depend on individual Hadoop JAR artifacts. Remove build dependency on Hadoop tarballs --- Key: HIVE-2744 URL: https://issues.apache.org/jira/browse/HIVE-2744 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2720) Merge MetaStoreListener and HiveMetaHook interfaces
[ https://issues.apache.org/jira/browse/HIVE-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13191809#comment-13191809 ] Carl Steinbach commented on HIVE-2720: -- bq. The patch as it is, does not change the event listener calling semantics or hivemetahook calling semantics. Actually, I think this is a pretty significant change. HiveMetaStoreListener is purely a listener interface. All of the methods defined in the interface are triggered after the underlying metastore transaction has been committed or rolled back, which means that all of the methods are effectively post methods. Supporting more than one listener is uncomplicated because the listeners don't have the ability to affect the outcome of the operation. This patch replaces the listener interface with one that allows plugins to fail metastore operations by raising exceptions in pre-methods. I don't think this interface can support more than one plugin in a maintainable fashion for the reasons I mentioned earlier. bq. introduce HiveExtendedMetaHook interface to limited private to replace HiveMetaStoreEventListener, and allow only hive and hcat implementations (by documenting it). We guarantee correct behavior by carefully ordering the hooks. I don't see any reason to remove HiveMetaStoreListener. It's useful in its current form for writing audit and logging plugins, the implementation is good, and people are probably already using it since it appeared in the 0.8.0 release. As for the proposed HiveExtendedMetaHook interface, in practice I don't think you will be able to guarantee correct behavior by ordering the hooks, and making this a requirement places a significant burden on hook implementors. Merge MetaStoreListener and HiveMetaHook interfaces --- Key: HIVE-2720 URL: https://issues.apache.org/jira/browse/HIVE-2720 Project: Hive Issue Type: Sub-task Components: JDBC, Metastore, ODBC, Security Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HIVE-2720.D1299.1.patch, HIVE-2720.D1299.2.patch, HIVE-2720.D1299.3.patch MetaStoreListener and HiveMetaHook both serve as a notification mechanism for metastore-related events. The former is used by hcat and the latter is by the hbase-storage handler, and invoked by the client. I propose to merge these interfaces, and extend the MetaStoreListener, to add most of the on- and pre- methods at the Thrift interface. This way, extending metastore will be easier, and validation, storage-driver notification, and enforcement can be delegated to individual listeners. Besides, more functionality can be plugged-in by Hcat at this level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2542) DROP DATABASE CASCADE does not drop non-native tables.
[ https://issues.apache.org/jira/browse/HIVE-2542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13190234#comment-13190234 ] Carl Steinbach commented on HIVE-2542: -- @Vandana: Do you have a link for the review request? Phabricator is supposed to automatically update the JIRA ticket. Did you forget to specify the JIRA ID when you ran arc? DROP DATABASE CASCADE does not drop non-native tables. --- Key: HIVE-2542 URL: https://issues.apache.org/jira/browse/HIVE-2542 Project: Hive Issue Type: Bug Components: Metastore Reporter: Vandana Ayyalasomayajula Assignee: Vandana Ayyalasomayajula Labels: handler, hive, metastore, storage Attachments: Hive-2542.patch The hive meta store client does not delete the non-native tables during the drop database dbname cascade operation. As a result even though the database is deleted the tables still exist. This is related to HCATALOG-144. A deeper look at the HiveMetaStoreClient's dropDatabase function, tells us that the function does not utilize the hive meta hooks of the tables in the database for dropping the non-native tables. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2612) support hive table/partitions coexistes in more than one clusters
[ https://issues.apache.org/jira/browse/HIVE-2612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189526#comment-13189526 ] Carl Steinbach commented on HIVE-2612: -- bq. A table T1's primary cluster is C1 meaning :1) C1 contains all data that is available in all other clusters. Does this mean that if T1's primary cluster is C1, then all of the partitions in T1 must also have have their primary partition set to C1? If that's the case then primary cluster should probably be a table level property, and the list of replica clusters can be a table/partition level property. bq. 2) write is only allowed in this cluster for table C1. but need to allow exceptions here What are the exceptions? bq. 4) all data changes to T1 happened in the primary cluster should be replicated to other clusters if there are any secondary clusters. but there should be a conf to disable it as there are some exception situations. What are the exceptions? How will dynamic partitions work? Where will new partitions get the list of replica clusters from? Will they inherit it from the table definition? Hive now supports insert-append into a partition (HIVE-306). Suppose that the metadata for a particular partition indicates that it is replicated to clusters C2 and C3. If I insert new data into the partition in the primary cluster C1, then the metadata is now invalid. How is this going to be handled? bq. 2) add new interfaces which do exactly the same set of functionalities as old ones but using a different name (use _on_cluster suffix maybe?) and have a cluster parameter This is going to introduce new codepaths that need to be tested separately, and also double the amount of work people need to do every time a new metastore API call is created. I don't think this is a good approach. bq. 3) overwrite database name for the purpose of cluster name. And allow a table co-exist in multiple databases. But that require to promote table to top level citizen, and degrade database. For example, show tables used to scan all tables in current db, but now need to scan all tables in all databases. I don't think this is an option since it breaks backwards compatibility and effectively changes the whole notion of what a db/schema is. A lot of people in the community already depend on this feature. bq. 1) add a cluster parameter to existing thrift interfaces This sounds like the best option to me. I think Thrift supports API evolution via default values for missing parameters, but setting a default value in this case may be a little tricky. Also, instead of modifying the Thrift interface, is it possible that you could instead leverage the work that's being done in HIVE-2720? support hive table/partitions coexistes in more than one clusters - Key: HIVE-2612 URL: https://issues.apache.org/jira/browse/HIVE-2612 Project: Hive Issue Type: New Feature Components: Metastore Reporter: He Yongqiang Assignee: He Yongqiang 1) add cluster object into hive metastore 2) each partition/table has a creation cluster and a list of living clusters, and also data location in each cluster -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2725) Fix flaky testing infrastructure
[ https://issues.apache.org/jira/browse/HIVE-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188803#comment-13188803 ] Carl Steinbach commented on HIVE-2725: -- I also verified that with HIVE-2589 in place you can run tests individually and get them to pass. However, if you run all of TestCliDriver then 149 tests reliably fail. Someone needs to start with the entire testsuite and gradually cut away tests until they get to smallest set possible that reliably triggers the failure in repair.q, and then drill down from there. Fix flaky testing infrastructure - Key: HIVE-2725 URL: https://issues.apache.org/jira/browse/HIVE-2725 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.9.0 Reporter: Ashutosh Chauhan To begin with org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert2_overwrite_partitions and org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl5 are failing on trunk for a while now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2163) SquirrelSQL 3.2.1 doesn't work with Hive 0.7.0
[
https://issues.apache.org/jira/browse/HIVE-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188879#comment-13188879
]
Carl Steinbach commented on HIVE-2163:
--
@Todd: can you provide more details about this other situation you encountered?
How can I reproduce it? Thanks.
SquirrelSQL 3.2.1 doesn't work with Hive 0.7.0
--
Key: HIVE-2163
URL: https://issues.apache.org/jira/browse/HIVE-2163
Project: Hive
Issue Type: Bug
Components: JDBC
Affects Versions: 0.7.0
Reporter: Carl Steinbach
Assignee: Carl Steinbach
Fix For: 0.7.1
I tried using SquirrelSQL 3.2.1 with Hive 0.7.0 following the directions
located here: http://wiki.apache.org/hadoop/Hive/HiveJDBCInterface
I registered the Hive JDBC driver with SquirrelSQL, and then get the
following exception when I try to connect to the Hive Server process:
{noformat}
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: java.io.IOException: failure to login
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:226)
at org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:72)
at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:110)
at
net.sourceforge.squirrel_sql.fw.sql.SQLDriverManager.getConnection(SQLDriverManager.java:133)
at
net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.execute(OpenConnectionCommand.java:97)
at
net.sourceforge.squirrel_sql.client.mainframe.action.ConnectToAliasCommand$SheetHandler.run(ConnectToAliasCommand.java:281)
at
net.sourceforge.squirrel_sql.client.mainframe.action.ConnectToAliasCommand$SheetHandler.performOK(ConnectToAliasCommand.java:238)
at
net.sourceforge.squirrel_sql.client.gui.db.ConnectionInternalFrame.connect(ConnectionInternalFrame.java:311)
at
net.sourceforge.squirrel_sql.client.gui.db.ConnectionInternalFrame.access$300(ConnectionInternalFrame.java:56)
at
net.sourceforge.squirrel_sql.client.gui.db.ConnectionInternalFrame$MyOkClosePanelListener.okPressed(ConnectionInternalFrame.java:461)
at
net.sourceforge.squirrel_sql.client.gui.OkClosePanel.fireButtonPressed(OkClosePanel.java:148)
at
net.sourceforge.squirrel_sql.client.gui.OkClosePanel.access$100(OkClosePanel.java:33)
at
net.sourceforge.squirrel_sql.client.gui.OkClosePanel$1.actionPerformed(OkClosePanel.java:174)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2028)
at
javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2351)
at
javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
at
javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
at java.awt.Component.processMouseEvent(Component.java:6374)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3267)
at java.awt.Component.processEvent(Component.java:6139)
at java.awt.Container.processEvent(Container.java:2085)
at java.awt.Component.dispatchEventImpl(Component.java:4736)
at java.awt.Container.dispatchEventImpl(Container.java:2143)
at java.awt.Component.dispatchEvent(Component.java:4566)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4621)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4282)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4212)
at java.awt.Container.dispatchEventImpl(Container.java:2129)
at java.awt.Window.dispatchEventImpl(Window.java:2478)
at java.awt.Component.dispatchEvent(Component.java:4566)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:680)
at java.awt.EventQueue.access$000(EventQueue.java:86)
at java.awt.EventQueue$1.run(EventQueue.java:639)
at java.awt.EventQueue$1.run(EventQueue.java:637)
at java.security.AccessController.doPrivileged(Native Method)
at
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:98)
at java.awt.EventQueue$2.run(EventQueue.java:653)
at java.awt.EventQueue$2.run(EventQueue.java:651)
at java.security.AccessController.doPrivileged(Native Method)
at
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:650)
at net.sourceforge.squirrel_sql.client.Main$1.dispatchEvent(Main.java:93)
at
java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:296)
at
java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:211)
at
[jira] [Commented] (HIVE-2163) SquirrelSQL 3.2.1 doesn't work with Hive 0.7.0
[
https://issues.apache.org/jira/browse/HIVE-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188892#comment-13188892
]
Carl Steinbach commented on HIVE-2163:
--
Todd, thanks for finding this. I filed HIVE-2729 to track the
ContextClassLoader issue.
SquirrelSQL 3.2.1 doesn't work with Hive 0.7.0
--
Key: HIVE-2163
URL: https://issues.apache.org/jira/browse/HIVE-2163
Project: Hive
Issue Type: Bug
Components: JDBC
Affects Versions: 0.7.0
Reporter: Carl Steinbach
Assignee: Carl Steinbach
Fix For: 0.7.1
I tried using SquirrelSQL 3.2.1 with Hive 0.7.0 following the directions
located here: http://wiki.apache.org/hadoop/Hive/HiveJDBCInterface
I registered the Hive JDBC driver with SquirrelSQL, and then get the
following exception when I try to connect to the Hive Server process:
{noformat}
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: java.io.IOException: failure to login
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:226)
at org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:72)
at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:110)
at
net.sourceforge.squirrel_sql.fw.sql.SQLDriverManager.getConnection(SQLDriverManager.java:133)
at
net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.execute(OpenConnectionCommand.java:97)
at
net.sourceforge.squirrel_sql.client.mainframe.action.ConnectToAliasCommand$SheetHandler.run(ConnectToAliasCommand.java:281)
at
net.sourceforge.squirrel_sql.client.mainframe.action.ConnectToAliasCommand$SheetHandler.performOK(ConnectToAliasCommand.java:238)
at
net.sourceforge.squirrel_sql.client.gui.db.ConnectionInternalFrame.connect(ConnectionInternalFrame.java:311)
at
net.sourceforge.squirrel_sql.client.gui.db.ConnectionInternalFrame.access$300(ConnectionInternalFrame.java:56)
at
net.sourceforge.squirrel_sql.client.gui.db.ConnectionInternalFrame$MyOkClosePanelListener.okPressed(ConnectionInternalFrame.java:461)
at
net.sourceforge.squirrel_sql.client.gui.OkClosePanel.fireButtonPressed(OkClosePanel.java:148)
at
net.sourceforge.squirrel_sql.client.gui.OkClosePanel.access$100(OkClosePanel.java:33)
at
net.sourceforge.squirrel_sql.client.gui.OkClosePanel$1.actionPerformed(OkClosePanel.java:174)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2028)
at
javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2351)
at
javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
at
javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
at java.awt.Component.processMouseEvent(Component.java:6374)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3267)
at java.awt.Component.processEvent(Component.java:6139)
at java.awt.Container.processEvent(Container.java:2085)
at java.awt.Component.dispatchEventImpl(Component.java:4736)
at java.awt.Container.dispatchEventImpl(Container.java:2143)
at java.awt.Component.dispatchEvent(Component.java:4566)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4621)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4282)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4212)
at java.awt.Container.dispatchEventImpl(Container.java:2129)
at java.awt.Window.dispatchEventImpl(Window.java:2478)
at java.awt.Component.dispatchEvent(Component.java:4566)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:680)
at java.awt.EventQueue.access$000(EventQueue.java:86)
at java.awt.EventQueue$1.run(EventQueue.java:639)
at java.awt.EventQueue$1.run(EventQueue.java:637)
at java.security.AccessController.doPrivileged(Native Method)
at
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:98)
at java.awt.EventQueue$2.run(EventQueue.java:653)
at java.awt.EventQueue$2.run(EventQueue.java:651)
at java.security.AccessController.doPrivileged(Native Method)
at
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:650)
at net.sourceforge.squirrel_sql.client.Main$1.dispatchEvent(Main.java:93)
at
java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:296)
at
java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:211)
at
[jira] [Commented] (HIVE-2719) Fix test failures caused by HIVE-2589
[ https://issues.apache.org/jira/browse/HIVE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187880#comment-13187880 ] Carl Steinbach commented on HIVE-2719: -- @Yongqiang: I think this proves that the diffs were introduced by HIVE-2589: https://builds.apache.org/job/Hive-trunk-h0.21-dbg/4/ I'm +1 on backing stuff out to fix the tests. Fix test failures caused by HIVE-2589 - Key: HIVE-2719 URL: https://issues.apache.org/jira/browse/HIVE-2719 Project: Hive Issue Type: Task Reporter: Carl Steinbach Assignee: Ashutosh Chauhan Priority: Blocker 149 new test failures appeared on Hudson after HIVE-2589 and HIVE-2682 were committed: https://builds.apache.org/job/Hive-trunk-h0.21/1195/ @Ashutosh: Can you please take a look? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2719) Fix test failures caused by HIVE-2589
[ https://issues.apache.org/jira/browse/HIVE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187903#comment-13187903 ] Carl Steinbach commented on HIVE-2719: -- @Yongqiang: I'd like to get this fixed too. I can test/commit your patch. I recommend we back out HIVE-2589 first since that's the cause of 149 of the failures, and then work back from there. Sound good? Fix test failures caused by HIVE-2589 - Key: HIVE-2719 URL: https://issues.apache.org/jira/browse/HIVE-2719 Project: Hive Issue Type: Task Reporter: Carl Steinbach Assignee: Ashutosh Chauhan Priority: Blocker 149 new test failures appeared on Hudson after HIVE-2589 and HIVE-2682 were committed: https://builds.apache.org/job/Hive-trunk-h0.21/1195/ @Ashutosh: Can you please take a look? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2719) Fix test failures caused by HIVE-2589
[ https://issues.apache.org/jira/browse/HIVE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187932#comment-13187932 ] Carl Steinbach commented on HIVE-2719: -- @Yongqiang: I'm running the tests now. Fix test failures caused by HIVE-2589 - Key: HIVE-2719 URL: https://issues.apache.org/jira/browse/HIVE-2719 Project: Hive Issue Type: Task Reporter: Carl Steinbach Assignee: Ashutosh Chauhan Priority: Blocker Attachments: HIVE-2719-revert.hive-2589.1.patch, HIVE-2719.D1281.1.patch 149 new test failures appeared on Hudson after HIVE-2589 and HIVE-2682 were committed: https://builds.apache.org/job/Hive-trunk-h0.21/1195/ @Ashutosh: Can you please take a look? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2715) Upgrade Thrift dependency to 0.9.0
[ https://issues.apache.org/jira/browse/HIVE-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188099#comment-13188099 ] Carl Steinbach commented on HIVE-2715: -- Also, Hive 0.8.0 depends on Thrift 0.7.0, not 0.5.0. Upgrade Thrift dependency to 0.9.0 -- Key: HIVE-2715 URL: https://issues.apache.org/jira/browse/HIVE-2715 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.8.0 Reporter: Mithun Radhakrishnan Fix For: 0.9.0 I work on HCatalog (0.2). Recently, we ran into HCat_server running out of memory every few days, and it boiled down to a bug in thrift, (THRIFT-1468, recently fixed). HCat-0.2-branch depends on Hive-0.8, which in turn depends on thrift-0.5.0. (The bug also exists on 0.7.0.) May I please enquire if Hive can't depend on a more current version of thrift? (Does it break the metastore?) I'm afraid I'm not privy to the reasoning behind Hive's dependency on a slightly dated thrift-lib. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2676) The row count that loaded to a table may not right
[ https://issues.apache.org/jira/browse/HIVE-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188104#comment-13188104 ] Carl Steinbach commented on HIVE-2676: -- @binlijin: Is this patch ready for review? The row count that loaded to a table may not right - Key: HIVE-2676 URL: https://issues.apache.org/jira/browse/HIVE-2676 Project: Hive Issue Type: Improvement Reporter: binlijin Priority: Minor Labels: patch Attachments: HIVE-2676.patch create table tablename as SELECT *** At the end hive will print a number that show how many Rows loaded to the tablename, but sometimes the number is not right. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2719) Fix test failures caused by HIVE-2589
[ https://issues.apache.org/jira/browse/HIVE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188134#comment-13188134 ] Carl Steinbach commented on HIVE-2719: -- @Ashutosh: Build #3 was against revision 1230379, which actually corresponds to HIVE-2682. Build #4 corresponds to revision 1230390 (HIVE-2589), and this build fails with +149 errors compared to build #3. Build #5 corresponds to revision 1230091 (HIVE-2705) and has two failures. I believe Jenkins is confused about the changes because I wasn't running through the revisions in order. Here's the logical order of the builds: HIVE-2705 r1230091 https://builds.apache.org/job/Hive-trunk-h0.21-dbg/5/ 2 Failures HIVE-2682 r1230379 https://builds.apache.org/job/Hive-trunk-h0.21-dbg/3/ 2 Failures HIVE-2589 r1230390 https://builds.apache.org/job/Hive-trunk-h0.21-dbg/4/ 151 Failures Fix test failures caused by HIVE-2589 - Key: HIVE-2719 URL: https://issues.apache.org/jira/browse/HIVE-2719 Project: Hive Issue Type: Task Reporter: Carl Steinbach Assignee: He Yongqiang Priority: Blocker Attachments: HIVE-2719-revert.hive-2589.1.patch, HIVE-2719.D1281.1.patch 149 new test failures appeared on Hudson after HIVE-2589 and HIVE-2682 were committed: https://builds.apache.org/job/Hive-trunk-h0.21/1195/ @Ashutosh: Can you please take a look? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2612) support hive table/partitions coexistes in more than one clusters
[ https://issues.apache.org/jira/browse/HIVE-2612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188156#comment-13188156 ] Carl Steinbach commented on HIVE-2612: -- This is a big feature to implement. I really think it needs a design document before we start committing patches. Yongqiang, do you think you can write something up? support hive table/partitions coexistes in more than one clusters - Key: HIVE-2612 URL: https://issues.apache.org/jira/browse/HIVE-2612 Project: Hive Issue Type: New Feature Reporter: He Yongqiang Assignee: He Yongqiang 1) add cluster object into hive metastore 2) each partition/table has a creation cluster and a list of living clusters, and also data location in each cluster -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2724) Remove unused lib/log4j-1.2.15.jar
[ https://issues.apache.org/jira/browse/HIVE-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188166#comment-13188166 ] Carl Steinbach commented on HIVE-2724: -- +1. Will commit if tests pass. Remove unused lib/log4j-1.2.15.jar -- Key: HIVE-2724 URL: https://issues.apache.org/jira/browse/HIVE-2724 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Andrew Bayer Attachments: HIVE-2724.diff.txt There's still a file, lib/log4j-1.2.15.jar, even though log4j is now pulled in via Ivy. As a result, this older log4j gets pulled into the Hive release tarball, and may end up in classpaths as well. It should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2719) Fix test failures caused by HIVE-2589
[ https://issues.apache.org/jira/browse/HIVE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188186#comment-13188186 ] Carl Steinbach commented on HIVE-2719: -- Tests are still running, but I see that TestCliDriver.ppr_pushdown2 failed with a timeout. I'm going to restart the job on a faster Hudson slave and hope that it finishes in the next 3 hours. Fix test failures caused by HIVE-2589 - Key: HIVE-2719 URL: https://issues.apache.org/jira/browse/HIVE-2719 Project: Hive Issue Type: Task Reporter: Carl Steinbach Assignee: He Yongqiang Priority: Blocker Attachments: HIVE-2719-revert.hive-2589.1.patch, HIVE-2719.D1281.1.patch 149 new test failures appeared on Hudson after HIVE-2589 and HIVE-2682 were committed: https://builds.apache.org/job/Hive-trunk-h0.21/1195/ @Ashutosh: Can you please take a look? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2616) Passing user identity from metastore client to server in non-secure mode
[ https://issues.apache.org/jira/browse/HIVE-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188313#comment-13188313 ] Carl Steinbach commented on HIVE-2616: -- +1 Passing user identity from metastore client to server in non-secure mode Key: HIVE-2616 URL: https://issues.apache.org/jira/browse/HIVE-2616 Project: Hive Issue Type: Bug Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.1, 0.9.0 Attachments: hive-2616.patch, hive-2616_1.patch, hive-2616_3.patch, hive-2616_4.patch, hive-2616_5.patch Currently in unsecure mode client don't pass on user identity. As a result hdfs and other operations done by server gets executed by user running metastore process instead of being done in context of client. This results in problem as reported here: http://mail-archives.apache.org/mod_mbox/hive-user/20.mbox/%3CCAK0mCrRC3aPqtRHDe2J25Rm0JX6TS1KXxd7KPjqJjoqBjg=a...@mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2719) Fix test failures caused by HIVE-2589
[ https://issues.apache.org/jira/browse/HIVE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187082#comment-13187082 ] Carl Steinbach commented on HIVE-2719: -- I was also able to reproduce the test failures on our internal Jenkins server, so this is not something unique to Apache Hudson. Fix test failures caused by HIVE-2589 - Key: HIVE-2719 URL: https://issues.apache.org/jira/browse/HIVE-2719 Project: Hive Issue Type: Task Reporter: Carl Steinbach Assignee: Ashutosh Chauhan Priority: Blocker 149 new test failures appeared on Hudson after HIVE-2589 and HIVE-2682 were committed: https://builds.apache.org/job/Hive-trunk-h0.21/1195/ @Ashutosh: Can you please take a look? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2691) Specify location of log4j configuration files via configuration properties
[
https://issues.apache.org/jira/browse/HIVE-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184714#comment-13184714
]
Carl Steinbach commented on HIVE-2691:
--
@Zhenxiao: Looks like Arc is creating new review requests instead of updating
your original request. Can you please verify that you are following the
directions for updating review requests:
https://cwiki.apache.org/confluence/display/Hive/PhabricatorCodeReview#PhabricatorCodeReview-UpdatingPatches%28git%29
Specify location of log4j configuration files via configuration properties
--
Key: HIVE-2691
URL: https://issues.apache.org/jira/browse/HIVE-2691
Project: Hive
Issue Type: New Feature
Components: Configuration, Logging
Reporter: Carl Steinbach
Assignee: Zhenxiao Luo
Attachments: HIVE-2691.D1131.1.patch, HIVE-2691.D1203.1.patch
Oozie needs to be able to override the default location of the log4j
configuration
files from the Hive command line, e.g:
{noformat}
hive -hiveconf hive.log4j.file=/home/carl/hive-log4j.properties -hiveconf
hive.log4j.exec.file=/home/carl/hive-exec-log4j.properties
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2707) Ant's untar task can't handle symlinks
[ https://issues.apache.org/jira/browse/HIVE-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184736#comment-13184736 ] Carl Steinbach commented on HIVE-2707: -- @Ashutosh: Chances are that if tar is not available on your system, then you probably shouldn't try to build Hive there anyway (e.g. Windows). Ant's untar task can't handle symlinks -- Key: HIVE-2707 URL: https://issues.apache.org/jira/browse/HIVE-2707 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Attachments: HIVE-2707.1.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2694) Add FORMAT UDF
[ https://issues.apache.org/jira/browse/HIVE-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13180747#comment-13180747 ] Carl Steinbach commented on HIVE-2694: -- Ref: http://dev.mysql.com/doc/refman/5.1/en/string-functions.html#function_format Formats the number X to a format like '#,###,###.##', rounded to D decimal places, and returns the result as a string. If D is 0, the result has no decimal point or fractional part. mysql SELECT FORMAT(12332.123456, 4); - '12,332.1235' mysql SELECT FORMAT(12332.1,4); - '12,332.1000' mysql SELECT FORMAT(12332.2,0); - '12,332' Add FORMAT UDF -- Key: HIVE-2694 URL: https://issues.apache.org/jira/browse/HIVE-2694 Project: Hive Issue Type: New Feature Components: UDF Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2695) Add PRINTF() Udf
[ https://issues.apache.org/jira/browse/HIVE-2695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13180759#comment-13180759 ] Carl Steinbach commented on HIVE-2695: -- Add a PRINTF(String format, Obj... args) Udf that can format strings according to printf-style format strings. Ref: http://docs.oracle.com/javase/1.5.0/docs/api/java/util/Formatter.html Add PRINTF() Udf Key: HIVE-2695 URL: https://issues.apache.org/jira/browse/HIVE-2695 Project: Hive Issue Type: New Feature Components: UDF Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2573) Create per-session function registry
[ https://issues.apache.org/jira/browse/HIVE-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13179212#comment-13179212 ] Carl Steinbach commented on HIVE-2573: -- @Navis: Is this ready for a final review? Create per-session function registry - Key: HIVE-2573 URL: https://issues.apache.org/jira/browse/HIVE-2573 Project: Hive Issue Type: Improvement Components: Server Infrastructure Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-2573.1.patch.txt, HIVE-2573.2.patch.txt, HIVE-2573.3.patch.txt Currently the function registry is shared resource and could be overrided by other users when using HiveServer. If per-session function registry is provided, this situation could be prevented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2503) HiveServer should provide per session configuration
[ https://issues.apache.org/jira/browse/HIVE-2503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13179213#comment-13179213 ] Carl Steinbach commented on HIVE-2503: -- @Navis: Is this ticket ready for a final review? HiveServer should provide per session configuration --- Key: HIVE-2503 URL: https://issues.apache.org/jira/browse/HIVE-2503 Project: Hive Issue Type: Bug Components: CLI, Server Infrastructure Affects Versions: 0.9.0 Reporter: Navis Assignee: Navis Fix For: 0.9.0 Attachments: HIVE-2503.1.patch.txt Currently ThriftHiveProcessorFactory returns same HiveConf instance to HiveServerHandler, making impossible to use per sesssion configuration. Just wrapping 'conf' - 'new HiveConf(conf)' seemed to solve this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2687) Test classpath should resemble runtime classpath
[ https://issues.apache.org/jira/browse/HIVE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13177418#comment-13177418 ] Carl Steinbach commented on HIVE-2687: -- The scripts in the bin/ directory execute Hive using the 'hadoop jar' command. Consequently Hive inherits and gives precedence to the contents of Hadoop's lib/ directory. We need to make sure that Hive tests run with a similar classpath. Test classpath should resemble runtime classpath Key: HIVE-2687 URL: https://issues.apache.org/jira/browse/HIVE-2687 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.8.0 Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1877) Add java_method() as a synonym for the reflect() UDF
[ https://issues.apache.org/jira/browse/HIVE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175329#comment-13175329 ] Carl Steinbach commented on HIVE-1877: -- +1. Will commit if tests pass. Add java_method() as a synonym for the reflect() UDF Key: HIVE-1877 URL: https://issues.apache.org/jira/browse/HIVE-1877 Project: Hive Issue Type: Improvement Components: UDF Affects Versions: 0.7.0 Reporter: Carl Steinbach Assignee: Zhenxiao Luo Attachments: HIVE-1877.1.patch.txt, HIVE-1877.2.patch.txt, HIVE-1877.3.patch.txt HIVE-471 added the reflect() UDF which allows people to invoke static Java methods from within HQL queries. In my opinion the name is confusing since it describes how the UDF works instead of what it does. I propose changing the name of (or providing a synonym for) the UDF to something like 'invoke_method' or 'java_method', or something similar. I'm open to suggestions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2616) Passing user identity from metastore client to server in non-secure mode
[ https://issues.apache.org/jira/browse/HIVE-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175345#comment-13175345 ] Carl Steinbach commented on HIVE-2616: -- +1. Will commit if tests pass. Passing user identity from metastore client to server in non-secure mode Key: HIVE-2616 URL: https://issues.apache.org/jira/browse/HIVE-2616 Project: Hive Issue Type: Bug Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: hive-2616.patch, hive-2616_1.patch, hive-2616_3.patch, hive-2616_4.patch, hive-2616_5.patch Currently in unsecure mode client don't pass on user identity. As a result hdfs and other operations done by server gets executed by user running metastore process instead of being done in context of client. This results in problem as reported here: http://mail-archives.apache.org/mod_mbox/hive-user/20.mbox/%3CCAK0mCrRC3aPqtRHDe2J25Rm0JX6TS1KXxd7KPjqJjoqBjg=a...@mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2672) CLI fails to start when run on Hadoop 0.23.0
[
https://issues.apache.org/jira/browse/HIVE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13174514#comment-13174514
]
Carl Steinbach commented on HIVE-2672:
--
The CLI won't start when run against Hadoop 0.23.0:
{noformat}
% ant clean package -Dhadoop.version=0.23.0 -Dhadoop.security.version=0.23.0
-Dhadoop.security.version.prefix=0.23
% export HIVE_HOME=`pwd`/build/dist
% export HADOOP_HOME=`pwd`/build/hadoopcore/hadoop-0.23.0
% export PATH=$HIVE_HOME/bin:$HADOOP_HOME/bin:$PATH
% hive -hiveconf hive.root.logger=INFO,console
log4j:ERROR Could not find value for key log4j.appender.NullAppender
log4j:ERROR Could not instantiate appender named NullAppender.
log4j:ERROR Could not find value for key log4j.appender.NullAppender
log4j:ERROR Could not instantiate appender named NullAppender.
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
11/12/20 22:09:41 WARN conf.HiveConf: hive-site.xml not found on CLASSPATH
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/hadoop/mapred/JobConf
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:805)
at org.apache.hadoop.hive.conf.HiveConf.init(HiveConf.java:772)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:576)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:200)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
... 9 more
{noformat}
CLI fails to start when run on Hadoop 0.23.0
Key: HIVE-2672
URL: https://issues.apache.org/jira/browse/HIVE-2672
Project: Hive
Issue Type: Bug
Components: CLI, Shims
Reporter: Carl Steinbach
Assignee: Carl Steinbach
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2673) Eclipse launch configurations fail due to unsatisfied builtins JAR dependency
[
https://issues.apache.org/jira/browse/HIVE-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13174537#comment-13174537
]
Carl Steinbach commented on HIVE-2673:
--
* Generate eclipse templates and load the project into Eclipse.
* Run the TestJdbc launch configuration, get the following exception:
{noformat}
java.lang.RuntimeException: Failed to load Hive builtin functions
at
org.apache.hadoop.hive.ql.session.SessionState.init(SessionState.java:190)
at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.init(HiveServer.java:135)
at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.init(HiveServer.java:121)
at
org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:76)
at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:104)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at
org.apache.hadoop.hive.jdbc.TestJdbcDriver.setUp(TestJdbcDriver.java:87)
at junit.framework.TestCase.runBare(TestCase.java:132)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.init(ZipFile.java:114)
at java.util.jar.JarFile.init(JarFile.java:135)
at java.util.jar.JarFile.init(JarFile.java:72)
at sun.net.www.protocol.jar.URLJarFile.init(URLJarFile.java:72)
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:48)
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:55)
at
sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:104)
at
sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:132)
at java.net.URL.openStream(URL.java:1010)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerFunctionsFromPluginJar(FunctionRegistry.java:1196)
at
org.apache.hadoop.hive.ql.session.SessionState.init(SessionState.java:187)
... 20 more
{noformat}
Eclipse launch configurations fail due to unsatisfied builtins JAR dependency
-
Key: HIVE-2673
URL: https://issues.apache.org/jira/browse/HIVE-2673
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Reporter: Carl Steinbach
Assignee: John Sichi
Fix For: 0.8.1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2672) CLI fails to start when run on Hadoop 0.23.0
[
https://issues.apache.org/jira/browse/HIVE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13174554#comment-13174554
]
Carl Steinbach commented on HIVE-2672:
--
Update: I can get the CLI to start if I first set HADOOP_CLASSPATH as follows:
{noformat}
% export
HADOOP_CLASSPATH=`pwd`/build/hadoopcore/hadoop-0.23.0/modules/hadoop-mapreduce-client-core-0.23.0.jar
{noformat}
CLI fails to start when run on Hadoop 0.23.0
Key: HIVE-2672
URL: https://issues.apache.org/jira/browse/HIVE-2672
Project: Hive
Issue Type: Bug
Components: CLI, Shims
Reporter: Carl Steinbach
Assignee: Carl Steinbach
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2662) Add Ant configuration property for dumping classpath of tests
[ https://issues.apache.org/jira/browse/HIVE-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171420#comment-13171420 ] Carl Steinbach commented on HIVE-2662: -- This patch adds a 'test.print.classpath' property to build-common.xml When set to 'true', it will dump the classpath of each test before running it. Add Ant configuration property for dumping classpath of tests - Key: HIVE-2662 URL: https://issues.apache.org/jira/browse/HIVE-2662 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Attachments: HIVE-2662.D903.1.patch, HIVE-2662.D903.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2631) Make Hive work with Hadoop 1.0.0
[ https://issues.apache.org/jira/browse/HIVE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170572#comment-13170572 ] Carl Steinbach commented on HIVE-2631: -- +1. Will commit if tests pass. bq. Since, its a straight forward patch will commit unless some one has an objection. Hive committers are not allowed to commit their own patches, right? Make Hive work with Hadoop 1.0.0 Key: HIVE-2631 URL: https://issues.apache.org/jira/browse/HIVE-2631 Project: Hive Issue Type: Bug Components: Shims Affects Versions: 0.8.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: hive-2631.patch With Hadoop 1.0.0 around the corner ( http://mail-archives.apache.org/mod_mbox/hadoop-general/20.mbox/%3C9D6B6144-F4E0-4A31-883F-2AC504727A1F%40hortonworks.com%3E ), it will be useful to make Hive work with it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2616) Passing user identity from metastore client to server in non-secure mode
[ https://issues.apache.org/jira/browse/HIVE-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170685#comment-13170685 ] Carl Steinbach commented on HIVE-2616: -- @John: I'm looking at the patch now. Will respond soon with some comments. Passing user identity from metastore client to server in non-secure mode Key: HIVE-2616 URL: https://issues.apache.org/jira/browse/HIVE-2616 Project: Hive Issue Type: Bug Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: hive-2616.patch, hive-2616_1.patch, hive-2616_3.patch Currently in unsecure mode client don't pass on user identity. As a result hdfs and other operations done by server gets executed by user running metastore process instead of being done in context of client. This results in problem as reported here: http://mail-archives.apache.org/mod_mbox/hive-user/20.mbox/%3CCAK0mCrRC3aPqtRHDe2J25Rm0JX6TS1KXxd7KPjqJjoqBjg=a...@mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2659) SHOW FUNCTIONS still returns internal operators
[ https://issues.apache.org/jira/browse/HIVE-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170744#comment-13170744 ] Carl Steinbach commented on HIVE-2659: -- HIVE-1892 implemented a mechanism that filters the output of the SHOW FUNCTIONS command in order to remove type operators, e.g. DOUBLE, BOOLEAN, etc. However, we're still not removing other non-function functions such as '+', '', 'if', 'in', etc. Instead of filtering the output of SHOW FUNCTIONS, I think a better approach is to add registerInternalUDF and registerInternalGenericUDF methods to FunctionRegistry, and then use the information captured by these methods to influence the generation of the list of functions returned to SHOW FUNCTIONS. SHOW FUNCTIONS still returns internal operators --- Key: HIVE-2659 URL: https://issues.apache.org/jira/browse/HIVE-2659 Project: Hive Issue Type: Bug Components: UDF Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-727) Hive Server getSchema() returns wrong schema for Explain queries
[ https://issues.apache.org/jira/browse/HIVE-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13169914#comment-13169914 ] Carl Steinbach commented on HIVE-727: - +1. Will commit if tests pass. Hive Server getSchema() returns wrong schema for Explain queries -- Key: HIVE-727 URL: https://issues.apache.org/jira/browse/HIVE-727 Project: Hive Issue Type: Bug Reporter: Eric Hwang Assignee: Prasad Mujumdar Attachments: HIVE-727.patch, HIVE-727.patch.1, Hive-727.patch The Hive Server's getSchema() function will return a schema with zero fields when executing an Explain... query. A quick fix might be to set the default schema to be exactly one column of type string. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2643) Recent patch prevents Hadoop confs from loading in 0.20.204
[
https://issues.apache.org/jira/browse/HIVE-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167820#comment-13167820
]
Carl Steinbach commented on HIVE-2643:
--
@Jakob: Can you please try testing HIVE-2643.D759.2.patch? This version has
some changes that were not included in HIVE-2643.1.patch.txt. Thanks!
Recent patch prevents Hadoop confs from loading in 0.20.204
---
Key: HIVE-2643
URL: https://issues.apache.org/jira/browse/HIVE-2643
Project: Hive
Issue Type: Bug
Components: Configuration
Reporter: Jakob Homan
Assignee: Carl Steinbach
Priority: Blocker
Attachments: HIVE-2643.1.patch.txt, HIVE-2643.D747.1.patch,
HIVE-2643.D753.1.patch, HIVE-2643.D759.1.patch, HIVE-2643.D759.2.patch,
HIVE-2643.D759.2.patch, HIVE-2643.D765.1.patch, HIVE-2643.D765.2.patch,
HIVE-2643.D789.1.patch
After the application of HIVE-2362, we're unable to run queries against the
Hive cluster (RC2 and 3):
{noformat}java.io.IOException: Failed to set permissions of path:
/user/jhoman-667872320/.staging to 0700
at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:680)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:483)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318)
at
org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183)
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116){noformat}
this is due to all the hadoop confs not being loaded and the LocalJobRunner
(which builds staging directories in this manner) being used instead of the
regular one. It's also not possible to access the hdfs since no default fs is
specified.
Reverting 2362 fixes this, but I've not yet looked at that patch to see the
exact cause.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2638) Tests fail when Hive is run against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HIVE-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165550#comment-13165550 ] Carl Steinbach commented on HIVE-2638: -- The 0.23 Jenkins job: https://builds.apache.org/job/Hive-trunk-h0.23.0/ Failing tests: TestCliDriver.auto_join0 TestNegativeCliDriver TestHiveServer TestContribCliDriver TestHBaseCliDriver TestHWISessionManager TestJDBCDriver TestExecDriver Tests fail when Hive is run against Hadoop 0.23 --- Key: HIVE-2638 URL: https://issues.apache.org/jira/browse/HIVE-2638 Project: Hive Issue Type: Bug Components: Shims Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13165046#comment-13165046 ] Carl Steinbach commented on HIVE-2629: -- bq. Do we want all 3 shims to be compiled as part of the build? Yes, I support doing it this way. Make a single Hive binary work with both 0.20.x and 0.23.0 -- Key: HIVE-2629 URL: https://issues.apache.org/jira/browse/HIVE-2629 Project: Hive Issue Type: Bug Components: Shims Reporter: Carl Steinbach Assignee: Thomas Weise -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163900#comment-13163900 ] Carl Steinbach commented on HIVE-2629: -- Relevant comment from HIVE-2468: https://issues.apache.org/jira/browse/HIVE-2468?focusedCommentId=13163313page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13163313 Make a single Hive binary work with both 0.20.x and 0.23.0 -- Key: HIVE-2629 URL: https://issues.apache.org/jira/browse/HIVE-2629 Project: Hive Issue Type: Bug Components: Shims Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2362) HiveConf properties not appearing in the output of 'set' or 'set -v'
[ https://issues.apache.org/jira/browse/HIVE-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163342#comment-13163342 ] Carl Steinbach commented on HIVE-2362: -- All tests pass for me on trunk. HiveConf properties not appearing in the output of 'set' or 'set -v' Key: HIVE-2362 URL: https://issues.apache.org/jira/browse/HIVE-2362 Project: Hive Issue Type: Bug Components: CLI, Configuration Reporter: Carl Steinbach Assignee: Carl Steinbach Priority: Blocker Fix For: 0.8.0 Attachments: HIVE-2362.1.patch.txt, HIVE-2362.2.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
