[jira] [Commented] (HIVE-6170) Upgrade to the latest version of bonecp
[ https://issues.apache.org/jira/browse/HIVE-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13871430#comment-13871430 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-6170: - There is a need to upgrade the bonecp to the latest release version since some database connectivity issues have been resolved with the latest release. Patch uploaded Upgrade to the latest version of bonecp --- Key: HIVE-6170 URL: https://issues.apache.org/jira/browse/HIVE-6170 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-6170.1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6170) Upgrade to the latest version of bonecp
[ https://issues.apache.org/jira/browse/HIVE-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-6170: Attachment: HIVE-6170.1.patch Upgrade to the latest version of bonecp --- Key: HIVE-6170 URL: https://issues.apache.org/jira/browse/HIVE-6170 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-6170.1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6054) HiveServer2 does not log the output of LogUtils.initHiveLog4j();
[ https://issues.apache.org/jira/browse/HIVE-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-6054: Attachment: HIVE-6054.1.patch HiveServer2 does not log the output of LogUtils.initHiveLog4j(); Key: HIVE-6054 URL: https://issues.apache.org/jira/browse/HIVE-6054 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-6054.1.patch Inside the main(), we just call LogUtils.initHiveLog4j() and do not log this information. This needs to be logged to see if the user has configured log4j.properties correctly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6054) HiveServer2 does not log the output of LogUtils.initHiveLog4j();
[ https://issues.apache.org/jira/browse/HIVE-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-6054: Status: Patch Available (was: Open) Logging the information. Patch attached HiveServer2 does not log the output of LogUtils.initHiveLog4j(); Key: HIVE-6054 URL: https://issues.apache.org/jira/browse/HIVE-6054 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-6054.1.patch Inside the main(), we just call LogUtils.initHiveLog4j() and do not log this information. This needs to be logged to see if the user has configured log4j.properties correctly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6170) Upgrade to the latest version of bonecp
Hari Sankar Sivarama Subramaniyan created HIVE-6170: --- Summary: Upgrade to the latest version of bonecp Key: HIVE-6170 URL: https://issues.apache.org/jira/browse/HIVE-6170 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6137) Hive should report that the file/path doesn’t exist when it doesn’t (it now reports SocketTimeoutException)
Hari Sankar Sivarama Subramaniyan created HIVE-6137: --- Summary: Hive should report that the file/path doesn’t exist when it doesn’t (it now reports SocketTimeoutException) Key: HIVE-6137 URL: https://issues.apache.org/jira/browse/HIVE-6137 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Hive should report that the file/path doesn’t exist when it doesn’t (it now reports SocketTimeoutException): Execute a Hive DDL query with a reference to a non-existent blob (such as CREATE EXTERNAL TABLE...) and check Hive logs (stderr): FAILED: Error in metadata: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask This error message is not intuitive. If a file doesn't exist, Hive should report FileNotFoundException. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (HIVE-6088) Change StreamingDelegator to pass distributed cache files to jar delegator
[ https://issues.apache.org/jira/browse/HIVE-6088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan reassigned HIVE-6088: --- Assignee: Hari Sankar Sivarama Subramaniyan Change StreamingDelegator to pass distributed cache files to jar delegator -- Key: HIVE-6088 URL: https://issues.apache.org/jira/browse/HIVE-6088 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (HIVE-6075) Remove hive.metastore.warehouse.dir from webhcat config
[ https://issues.apache.org/jira/browse/HIVE-6075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan reassigned HIVE-6075: --- Assignee: Hari Sankar Sivarama Subramaniyan Remove hive.metastore.warehouse.dir from webhcat config - Key: HIVE-6075 URL: https://issues.apache.org/jira/browse/HIVE-6075 Project: Hive Issue Type: Task Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6054) HiveServer2 does not log the output of LogUtils.initHiveLog4j();
Hari Sankar Sivarama Subramaniyan created HIVE-6054: --- Summary: HiveServer2 does not log the output of LogUtils.initHiveLog4j(); Key: HIVE-6054 URL: https://issues.apache.org/jira/browse/HIVE-6054 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Inside the main(), we just call LogUtils.initHiveLog4j() and do not log this information. This needs to be logged to see if the user has configured log4j.properties correctly. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HIVE-5796) percentjobcomplete returned by webhcat is null
[
https://issues.apache.org/jira/browse/HIVE-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-5796:
Fix Version/s: 0.13.0
percentjobcomplete returned by webhcat is null
--
Key: HIVE-5796
URL: https://issues.apache.org/jira/browse/HIVE-5796
Project: Hive
Issue Type: Bug
Components: WebHCat
Affects Versions: 0.12.0
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Fix For: 0.13.0
Attachments: HIVE-5796.1.patch.txt
Currently the WebHCat E2E test TestMapReduce_1 fails when comparing the job
status field percentComplete which is returned as null, the expected value is
map 100% reduce 100%.
In the templeton.log I see the following message:
{noformat}
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Using Hadoop
Version: 0.23
INFO | 07 Nov 2013 17:14:08,106 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
WEBHCAT_CONF_DIR=null
WARN | 07 Nov 2013 17:14:08,106 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Could not
find
D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\bin\null\override-container-log4j.properties.
Monitoring of Hadoop jobs submitted through WebHCat may be affected.
{noformat}
To get past this problem I explicitly set the WEBHCAT_CONF_DIR environment
variable to HCATALOG_HOME/etc/webhcat where
override-container-log4j.properties is present.
After fixing the above and rerunning, I still see the following error:
{noformat}
INFO | 07 Nov 2013 18:29:39,517 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
WEBHCAT_CONF_DIR=D:\hdp\\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat
INFO | 07 Nov 2013 18:29:39,517 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Found
D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat\override-container-log4j.properties
to use for job submission.
ERROR | 07 Nov 2013 18:29:39,519 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob is not
properly initialized. Monitoring of Hadoop jobs submitted through WebHCat may
be affected.
java.lang.IllegalArgumentException: Pathname /d:/hadoop/hdfs/tmp from
d:/hadoop/hdfs/tmp is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:93)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:82)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.copyLog4JtoFileSystem(TempletonControllerJob.java:82)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.clinit(TempletonControllerJob.java:126)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:104)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:101)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator.queueAsUser(LauncherDelegator.java:101)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator.enqueueController(LauncherDelegator.java:82)
at
org.apache.hive.hcatalog.templeton.JarDelegator.run(JarDelegator.java:55)
at
org.apache.hive.hcatalog.templeton.Server.mapReduceJar(Server.java:690)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown
[jira] [Updated] (HIVE-5796) percentjobcomplete returned by webhcat is null
[
https://issues.apache.org/jira/browse/HIVE-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-5796:
Affects Version/s: 0.12.0
percentjobcomplete returned by webhcat is null
--
Key: HIVE-5796
URL: https://issues.apache.org/jira/browse/HIVE-5796
Project: Hive
Issue Type: Bug
Components: WebHCat
Affects Versions: 0.12.0
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Fix For: 0.13.0
Attachments: HIVE-5796.1.patch.txt
Currently the WebHCat E2E test TestMapReduce_1 fails when comparing the job
status field percentComplete which is returned as null, the expected value is
map 100% reduce 100%.
In the templeton.log I see the following message:
{noformat}
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Using Hadoop
Version: 0.23
INFO | 07 Nov 2013 17:14:08,106 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
WEBHCAT_CONF_DIR=null
WARN | 07 Nov 2013 17:14:08,106 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Could not
find
D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\bin\null\override-container-log4j.properties.
Monitoring of Hadoop jobs submitted through WebHCat may be affected.
{noformat}
To get past this problem I explicitly set the WEBHCAT_CONF_DIR environment
variable to HCATALOG_HOME/etc/webhcat where
override-container-log4j.properties is present.
After fixing the above and rerunning, I still see the following error:
{noformat}
INFO | 07 Nov 2013 18:29:39,517 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
WEBHCAT_CONF_DIR=D:\hdp\\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat
INFO | 07 Nov 2013 18:29:39,517 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Found
D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat\override-container-log4j.properties
to use for job submission.
ERROR | 07 Nov 2013 18:29:39,519 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob is not
properly initialized. Monitoring of Hadoop jobs submitted through WebHCat may
be affected.
java.lang.IllegalArgumentException: Pathname /d:/hadoop/hdfs/tmp from
d:/hadoop/hdfs/tmp is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:93)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:82)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.copyLog4JtoFileSystem(TempletonControllerJob.java:82)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.clinit(TempletonControllerJob.java:126)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:104)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:101)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator.queueAsUser(LauncherDelegator.java:101)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator.enqueueController(LauncherDelegator.java:82)
at
org.apache.hive.hcatalog.templeton.JarDelegator.run(JarDelegator.java:55)
at
org.apache.hive.hcatalog.templeton.Server.mapReduceJar(Server.java:690)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown
[jira] [Created] (HIVE-5835) Null pointer exception in DeleteDelegator in templeton code
Hari Sankar Sivarama Subramaniyan created HIVE-5835: --- Summary: Null pointer exception in DeleteDelegator in templeton code Key: HIVE-5835 URL: https://issues.apache.org/jira/browse/HIVE-5835 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Fix For: 0.13.0 The following NPE is possible with the current implementation: ERROR | 13 Nov 2013 08:01:04,292 | org.apache.hcatalog.templeton.CatchallExceptionMapper | java.lang.NullPointerException at org.apache.hcatalog.templeton.tool.JobState.getChildren(JobState.java:180) at org.apache.hcatalog.templeton.DeleteDelegator.run(DeleteDelegator.java:51) at org.apache.hcatalog.templeton.Server.deleteJobId(Server.java:849) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1480) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1411) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1360) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1350) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:382) at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:85) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:349) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:724) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5836) Improve logging in ExecReducer.reduce()
Hari Sankar Sivarama Subramaniyan created HIVE-5836: --- Summary: Improve logging in ExecReducer.reduce() Key: HIVE-5836 URL: https://issues.apache.org/jira/browse/HIVE-5836 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Fix For: 0.13.0 Although ExecReducer.reduce() logs the used JVM heap memory, we need to log the committed, max and used memory (http://docs.oracle.com/javase/6/docs/api/java/lang/management/MemoryUsage.html) to debug any future outofmemory issues. l4j.info(ExecReducer: processing + cntr + rows: used memory = + used_memory); i.e. the above log info should include max_memory as well as committed_memory for a propery analysis of memory usage at runtime. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5796) percentjobcomplete returned by webhcat is null
Hari Sankar Sivarama Subramaniyan created HIVE-5796:
---
Summary: percentjobcomplete returned by webhcat is null
Key: HIVE-5796
URL: https://issues.apache.org/jira/browse/HIVE-5796
Project: Hive
Issue Type: Bug
Components: WebHCat
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Currently the WebHCat E2E test TestMapReduce_1 fails when comparing the job
status field percentComplete which is returned as null, the expected value is
map 100% reduce 100%.
In the templeton.log I see the following message:
{noformat}
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Using Hadoop
Version: 0.23
INFO | 07 Nov 2013 17:14:08,106 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
WEBHCAT_CONF_DIR=null
WARN | 07 Nov 2013 17:14:08,106 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Could not find
D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\bin\null\override-container-log4j.properties.
Monitoring of Hadoop jobs submitted through WebHCat may be affected.
{noformat}
To get past this problem I explicitly set the WEBHCAT_CONF_DIR environment
variable to HCATALOG_HOME/etc/webhcat where override-container-log4j.properties
is present.
After fixing the above and rerunning, I still see the following error:
{noformat}
INFO | 07 Nov 2013 18:29:39,517 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
WEBHCAT_CONF_DIR=D:\hdp\\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat
INFO | 07 Nov 2013 18:29:39,517 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Found
D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat\override-container-log4j.properties
to use for job submission.
ERROR | 07 Nov 2013 18:29:39,519 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob is not properly
initialized. Monitoring of Hadoop jobs submitted through WebHCat may be
affected.
java.lang.IllegalArgumentException: Pathname /d:/hadoop/hdfs/tmp from
d:/hadoop/hdfs/tmp is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:93)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:82)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.copyLog4JtoFileSystem(TempletonControllerJob.java:82)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.clinit(TempletonControllerJob.java:126)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:104)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:101)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator.queueAsUser(LauncherDelegator.java:101)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator.enqueueController(LauncherDelegator.java:82)
at
org.apache.hive.hcatalog.templeton.JarDelegator.run(JarDelegator.java:55)
at
org.apache.hive.hcatalog.templeton.Server.mapReduceJar(Server.java:690)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
[jira] [Updated] (HIVE-5796) percentjobcomplete returned by webhcat is null
[
https://issues.apache.org/jira/browse/HIVE-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-5796:
Attachment: HIVE-5796.1.patch.txt
percentjobcomplete returned by webhcat is null
--
Key: HIVE-5796
URL: https://issues.apache.org/jira/browse/HIVE-5796
Project: Hive
Issue Type: Bug
Components: WebHCat
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-5796.1.patch.txt
Currently the WebHCat E2E test TestMapReduce_1 fails when comparing the job
status field percentComplete which is returned as null, the expected value is
map 100% reduce 100%.
In the templeton.log I see the following message:
{noformat}
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Using Hadoop
Version: 0.23
INFO | 07 Nov 2013 17:14:08,106 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
WEBHCAT_CONF_DIR=null
WARN | 07 Nov 2013 17:14:08,106 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Could not
find
D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\bin\null\override-container-log4j.properties.
Monitoring of Hadoop jobs submitted through WebHCat may be affected.
{noformat}
To get past this problem I explicitly set the WEBHCAT_CONF_DIR environment
variable to HCATALOG_HOME/etc/webhcat where
override-container-log4j.properties is present.
After fixing the above and rerunning, I still see the following error:
{noformat}
INFO | 07 Nov 2013 18:29:39,517 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
WEBHCAT_CONF_DIR=D:\hdp\\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat
INFO | 07 Nov 2013 18:29:39,517 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Found
D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat\override-container-log4j.properties
to use for job submission.
ERROR | 07 Nov 2013 18:29:39,519 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob is not
properly initialized. Monitoring of Hadoop jobs submitted through WebHCat may
be affected.
java.lang.IllegalArgumentException: Pathname /d:/hadoop/hdfs/tmp from
d:/hadoop/hdfs/tmp is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:93)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:82)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.copyLog4JtoFileSystem(TempletonControllerJob.java:82)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.clinit(TempletonControllerJob.java:126)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:104)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:101)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator.queueAsUser(LauncherDelegator.java:101)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator.enqueueController(LauncherDelegator.java:82)
at
org.apache.hive.hcatalog.templeton.JarDelegator.run(JarDelegator.java:55)
at
org.apache.hive.hcatalog.templeton.Server.mapReduceJar(Server.java:690)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
[jira] [Updated] (HIVE-5796) percentjobcomplete returned by webhcat is null
[
https://issues.apache.org/jira/browse/HIVE-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-5796:
Status: Patch Available (was: Open)
percentjobcomplete returned by webhcat is null
--
Key: HIVE-5796
URL: https://issues.apache.org/jira/browse/HIVE-5796
Project: Hive
Issue Type: Bug
Components: WebHCat
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Currently the WebHCat E2E test TestMapReduce_1 fails when comparing the job
status field percentComplete which is returned as null, the expected value is
map 100% reduce 100%.
In the templeton.log I see the following message:
{noformat}
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Using Hadoop
Version: 0.23
INFO | 07 Nov 2013 17:14:08,106 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
WEBHCAT_CONF_DIR=null
WARN | 07 Nov 2013 17:14:08,106 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Could not
find
D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\bin\null\override-container-log4j.properties.
Monitoring of Hadoop jobs submitted through WebHCat may be affected.
{noformat}
To get past this problem I explicitly set the WEBHCAT_CONF_DIR environment
variable to HCATALOG_HOME/etc/webhcat where
override-container-log4j.properties is present.
After fixing the above and rerunning, I still see the following error:
{noformat}
INFO | 07 Nov 2013 18:29:39,517 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
WEBHCAT_CONF_DIR=D:\hdp\\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat
INFO | 07 Nov 2013 18:29:39,517 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Found
D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat\override-container-log4j.properties
to use for job submission.
ERROR | 07 Nov 2013 18:29:39,519 |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob is not
properly initialized. Monitoring of Hadoop jobs submitted through WebHCat may
be affected.
java.lang.IllegalArgumentException: Pathname /d:/hadoop/hdfs/tmp from
d:/hadoop/hdfs/tmp is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:93)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:82)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.copyLog4JtoFileSystem(TempletonControllerJob.java:82)
at
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.clinit(TempletonControllerJob.java:126)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:104)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:101)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator.queueAsUser(LauncherDelegator.java:101)
at
org.apache.hive.hcatalog.templeton.LauncherDelegator.enqueueController(LauncherDelegator.java:82)
at
org.apache.hive.hcatalog.templeton.JarDelegator.run(JarDelegator.java:55)
at
org.apache.hive.hcatalog.templeton.Server.mapReduceJar(Server.java:690)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
[jira] [Assigned] (HIVE-5056) MapJoinProcessor ignores order of values in removing RS
[ https://issues.apache.org/jira/browse/HIVE-5056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan reassigned HIVE-5056: --- Assignee: Hari Sankar Sivarama Subramaniyan (was: Navis) MapJoinProcessor ignores order of values in removing RS --- Key: HIVE-5056 URL: https://issues.apache.org/jira/browse/HIVE-5056 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Hari Sankar Sivarama Subramaniyan Fix For: 0.12.0 Attachments: HIVE-5056.D12147.1.patch, HIVE-5056.D12147.2.patch, HIVE-5056.D12147.3.patch http://www.mail-archive.com/user@hive.apache.org/msg09073.html -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HIVE-5056) MapJoinProcessor ignores order of values in removing RS
[ https://issues.apache.org/jira/browse/HIVE-5056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan reassigned HIVE-5056: --- Assignee: Navis (was: Hari Sankar Sivarama Subramaniyan) MapJoinProcessor ignores order of values in removing RS --- Key: HIVE-5056 URL: https://issues.apache.org/jira/browse/HIVE-5056 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Fix For: 0.12.0 Attachments: HIVE-5056.D12147.1.patch, HIVE-5056.D12147.2.patch, HIVE-5056.D12147.3.patch http://www.mail-archive.com/user@hive.apache.org/msg09073.html -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5348) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters
[ https://issues.apache.org/jira/browse/HIVE-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5348: Attachment: HIVE-5348.3.patch.txt If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters -- Key: HIVE-5348 URL: https://issues.apache.org/jira/browse/HIVE-5348 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5348.1.patch.txt, HIVE-5348.2.patch.txt, HIVE-5348.3.patch.txt Follow-up JIRA for HIVE-5202 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5336) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user
[ https://issues.apache.org/jira/browse/HIVE-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5336: Attachment: (was: HIVE-5336.3.patch) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user -- Key: HIVE-5336 URL: https://issues.apache.org/jira/browse/HIVE-5336 Project: Hive Issue Type: Bug Components: HCatalog Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5336.1.patch.txt, HIVE-5336.2.patch HCatSchema.remove currently does not renumber the fieldPositionMap which can be a problem when there are interleaving append() and remove() calls. 1. We should document that fieldPositionMap should not be cached by the end-user 2. We should make sure that the fieldPositionMap gets renumbered after remove() because HcatSchema.get will otherwise return wrong FieldSchemas. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5336) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user
[ https://issues.apache.org/jira/browse/HIVE-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5336: Attachment: (was: HIVE-5336.2.patch) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user -- Key: HIVE-5336 URL: https://issues.apache.org/jira/browse/HIVE-5336 Project: Hive Issue Type: Bug Components: HCatalog Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5336.1.patch.txt HCatSchema.remove currently does not renumber the fieldPositionMap which can be a problem when there are interleaving append() and remove() calls. 1. We should document that fieldPositionMap should not be cached by the end-user 2. We should make sure that the fieldPositionMap gets renumbered after remove() because HcatSchema.get will otherwise return wrong FieldSchemas. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5336) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user
[ https://issues.apache.org/jira/browse/HIVE-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5336: Attachment: HIVE-5336.2.patch.txt HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user -- Key: HIVE-5336 URL: https://issues.apache.org/jira/browse/HIVE-5336 Project: Hive Issue Type: Bug Components: HCatalog Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5336.1.patch.txt, HIVE-5336.2.patch.txt HCatSchema.remove currently does not renumber the fieldPositionMap which can be a problem when there are interleaving append() and remove() calls. 1. We should document that fieldPositionMap should not be cached by the end-user 2. We should make sure that the fieldPositionMap gets renumbered after remove() because HcatSchema.get will otherwise return wrong FieldSchemas. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5436) Hive's casting behavior needs to be consistent
[
https://issues.apache.org/jira/browse/HIVE-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809670#comment-13809670
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-5436:
-
[~xuefuz] Please see the comment above and let me know if you have any
questions.
Hive's casting behavior needs to be consistent
--
Key: HIVE-5436
URL: https://issues.apache.org/jira/browse/HIVE-5436
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Priority: Critical
Hive's casting behavior is inconsistent and the behavior of casting from one
type to another undocumented as of now when the casted value is out of range.
For example, casting out of range values from one type to another can result
in incorrect results.
Eg:
1. select cast('1000' as tinyint) from t1;
NULL
2. select 1000Y from t1;
FAILED: SemanticException [Error 10029]: Line 1:7 Invalid numerical constant
'1000Y'
3. select cast(1000 as tinyint) from t1;
-24
4.select cast(1.1e3-1000/0 as tinyint) from t1;
0
5. select cast(10/0 as tinyint) from pw18;
-1
The hive user can accidently try to typecast an out of range value. For
example in the e.g. 4/5 even though the final result is NaN, Hive can
typecast to a random result. Either we should document that the end user
should take care of overflow, underflow, division by 0, etc. by
himself/herself or we should return NULLs when the final result is out of
range.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Fix Version/s: 0.13.0 Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Fix For: 0.13.0 Attachments: HIVE-5519.1.patch.txt, HIVE-5519.2.patch.txt, HIVE-5519.3.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5617) Add webhcat e2e tests using 1. jobs (GET) 2. jobs/:jobid (GET) 3. jobs/:jobid (DELETE) apis
[ https://issues.apache.org/jira/browse/HIVE-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5617: Fix Version/s: 0.13.0 Status: Patch Available (was: Open) Replacing deprecated webhcat APIs with the ones specified in https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference Add webhcat e2e tests using 1. jobs (GET) 2. jobs/:jobid (GET) 3. jobs/:jobid (DELETE) apis --- Key: HIVE-5617 URL: https://issues.apache.org/jira/browse/HIVE-5617 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Fix For: 0.13.0 Attachments: HIVE-5617.1.patch.txt The current e2e test driver module(TestDriverCurl.pm) uses the deprecated API for hive 0.12. Use the jobs api introduced in Hive 0.12 for killing a job, getting the status of a job, etc. The reference is https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5617) Add webhcat e2e tests using 1. jobs (GET) 2. jobs/:jobid (GET) 3. jobs/:jobid (DELETE) apis
[ https://issues.apache.org/jira/browse/HIVE-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5617: Attachment: HIVE-5617.1.patch.txt Add webhcat e2e tests using 1. jobs (GET) 2. jobs/:jobid (GET) 3. jobs/:jobid (DELETE) apis --- Key: HIVE-5617 URL: https://issues.apache.org/jira/browse/HIVE-5617 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Fix For: 0.13.0 Attachments: HIVE-5617.1.patch.txt The current e2e test driver module(TestDriverCurl.pm) uses the deprecated API for hive 0.12. Use the jobs api introduced in Hive 0.12 for killing a job, getting the status of a job, etc. The reference is https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HIVE-5436) Hive's casting behavior needs to be consistent
[
https://issues.apache.org/jira/browse/HIVE-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan reassigned HIVE-5436:
---
Assignee: Hari Sankar Sivarama Subramaniyan
Hive's casting behavior needs to be consistent
--
Key: HIVE-5436
URL: https://issues.apache.org/jira/browse/HIVE-5436
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Priority: Critical
Hive's casting behavior is inconsistent and the behavior of casting from one
type to another undocumented as of now when the casted value is out of range.
For example, casting out of range values from one type to another can result
in incorrect results.
Eg:
1. select cast('1000' as tinyint) from t1;
NULL
2. select 1000Y from t1;
FAILED: SemanticException [Error 10029]: Line 1:7 Invalid numerical constant
'1000Y'
3. select cast(1000 as tinyint) from t1;
-24
4.select cast(1.1e3-1000/0 as tinyint) from t1;
0
5. select cast(10/0 as tinyint) from pw18;
-1
The hive user can accidently try to typecast an out of range value. For
example in the e.g. 4/5 even though the final result is NaN, Hive can
typecast to a random result. Either we should document that the end user
should take care of overflow, underflow, division by 0, etc. by
himself/herself or we should return NULLs when the final result is out of
range.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5436) Hive's casting behavior needs to be consistent
[
https://issues.apache.org/jira/browse/HIVE-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808550#comment-13808550
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-5436:
-
Thanks [~xuefuz] for the update. Can you please let me know if HIVE-5660 is in
progress? Else, I will look at it. This is required before I make further
changes to HIVE-5382 ( to cover all the edge cases).
Hive's casting behavior needs to be consistent
--
Key: HIVE-5436
URL: https://issues.apache.org/jira/browse/HIVE-5436
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Priority: Critical
Hive's casting behavior is inconsistent and the behavior of casting from one
type to another undocumented as of now when the casted value is out of range.
For example, casting out of range values from one type to another can result
in incorrect results.
Eg:
1. select cast('1000' as tinyint) from t1;
NULL
2. select 1000Y from t1;
FAILED: SemanticException [Error 10029]: Line 1:7 Invalid numerical constant
'1000Y'
3. select cast(1000 as tinyint) from t1;
-24
4.select cast(1.1e3-1000/0 as tinyint) from t1;
0
5. select cast(10/0 as tinyint) from pw18;
-1
The hive user can accidently try to typecast an out of range value. For
example in the e.g. 4/5 even though the final result is NaN, Hive can
typecast to a random result. Either we should document that the end user
should take care of overflow, underflow, division by 0, etc. by
himself/herself or we should return NULLs when the final result is out of
range.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5436) Hive's casting behavior needs to be consistent
[
https://issues.apache.org/jira/browse/HIVE-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808625#comment-13808625
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-5436:
-
[~xuefu] Reasons why I thought of fixing the consistency first:
1. I wanted to see how the intermediate results are handled in case of
numericals. For example, for tiny int (1228+1228)/20 will lead to a in-range
result, where as the intermediate result 1228+1228 will be a non tiny int.
This scenario will be very common in case of exponential notation.
2. HIVE-5382 will need a baseline to compare the string cast results with
non-string cast results. My plan was to use testcases like this :
select cast('-1.5e2' as int)-cast(-1.5e2 as int) from tmp and verify that the
result is always 0. This will ensure consistency across cast from
string-numericals (and will expose any existing bugs which is fixed in future
for only one of the cast types since the non-string cast and string cast are
handled separately).
Hive's casting behavior needs to be consistent
--
Key: HIVE-5436
URL: https://issues.apache.org/jira/browse/HIVE-5436
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Priority: Critical
Hive's casting behavior is inconsistent and the behavior of casting from one
type to another undocumented as of now when the casted value is out of range.
For example, casting out of range values from one type to another can result
in incorrect results.
Eg:
1. select cast('1000' as tinyint) from t1;
NULL
2. select 1000Y from t1;
FAILED: SemanticException [Error 10029]: Line 1:7 Invalid numerical constant
'1000Y'
3. select cast(1000 as tinyint) from t1;
-24
4.select cast(1.1e3-1000/0 as tinyint) from t1;
0
5. select cast(10/0 as tinyint) from pw18;
-1
The hive user can accidently try to typecast an out of range value. For
example in the e.g. 4/5 even though the final result is NaN, Hive can
typecast to a random result. Either we should document that the end user
should take care of overflow, underflow, division by 0, etc. by
himself/herself or we should return NULLs when the final result is out of
range.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Attachment: HIVE-5519.3.patch.txt Thanks for the review, [~thejas] Updating the change as per [~thejas] ' comment. The bug with the previous upload was that even though the counter was incremented correctly based on the filter condition, the records which do not satisfy the filter condition do not get skipped. The fix is to continue and skip the current record, if the filter condition is not satisfied in Server.showJobList(). Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt, HIVE-5519.2.patch.txt, HIVE-5519.3.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Attachment: (was: HIVE-5519.3.patch.txt) Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt, HIVE-5519.2.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Attachment: HIVE-5519.3.patch.txt Addressing [~thejas] ' comments. Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt, HIVE-5519.2.patch.txt, HIVE-5519.3.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Attachment: (was: HIVE-5519.3.patch.txt) Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt, HIVE-5519.2.patch.txt, HIVE-5519.3.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Attachment: HIVE-5519.3.patch.txt Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt, HIVE-5519.2.patch.txt, HIVE-5519.3.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Attachment: (was: HIVE-5519.3.patch.txt) Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt, HIVE-5519.2.patch.txt, HIVE-5519.3.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Attachment: HIVE-5519.3.patch.txt Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt, HIVE-5519.2.patch.txt, HIVE-5519.3.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Attachment: HIVE-5519.2.patch.txt Addressed the comments made by [~eugene.koifman] Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt, HIVE-5519.2.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Attachment: HIVE-5519.2.patch.txt Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt, HIVE-5519.2.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Attachment: (was: HIVE-5519.2.patch.txt) Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt, HIVE-5519.2.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Attachment: HIVE-5519.1.patch.txt Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Status: Patch Available (was: Open) Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804849#comment-13804849 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5519: - [~ekoifman] Agree with what you say. Also TestDriverCurl.pm needs some modification to replace the deprecated APIs with the new ones. I will make these changes as part of HIVE-5617 Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804857#comment-13804857 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5519: - https://reviews.apache.org/r/14916/ Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5519.1.patch.txt Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5519) Use paging mechanism for templeton get requests.
[ https://issues.apache.org/jira/browse/HIVE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5519: Description: Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. was: Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to introduce a new api to retrieve a list of job details rather than retrieve all the information using a single command. Summary: Use paging mechanism for templeton get requests. (was: Support ranges of job ids for templeton) Use paging mechanism for templeton get requests. Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to use paging based encoding scheme so that we flush the contents regularly and the client does not time out. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5348) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters
[ https://issues.apache.org/jira/browse/HIVE-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5348: Attachment: (was: HIVE-5348.2.patch) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters -- Key: HIVE-5348 URL: https://issues.apache.org/jira/browse/HIVE-5348 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5348.1.patch.txt Follow-up JIRA for HIVE-5202 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5348) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters
[ https://issues.apache.org/jira/browse/HIVE-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5348: Attachment: HIVE-5348.2.patch.txt If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters -- Key: HIVE-5348 URL: https://issues.apache.org/jira/browse/HIVE-5348 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5348.1.patch.txt, HIVE-5348.2.patch.txt Follow-up JIRA for HIVE-5202 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5519) Support ranges of job ids for templeton
Hari Sankar Sivarama Subramaniyan created HIVE-5519: --- Summary: Support ranges of job ids for templeton Key: HIVE-5519 URL: https://issues.apache.org/jira/browse/HIVE-5519 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Issuing a command to retrieve the jobs field using https://mwinkledemo.azurehdinsight.net:563/templeton/v1/queue/job_id?user.name=adminfields=* --user u:p will result in timeout in windows machine. The issue happens because of the amount of data that needs to be fetched. The proposal is to introduce a new api to retrieve a list of job details rather than retrieve all the information using a single command. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5394: Attachment: (was: HIVE-5394.12.branch.txt) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Fix For: 0.13.0 Attachments: HIVE-5394.1.patch, HIVE-5394.2.patch.txt, HIVE-5394.3.patch The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5394: Attachment: HIVE-5394.12.branch.txt Thanks [~ashutoshc]. Reattaching patch for .12 branch ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Fix For: 0.13.0 Attachments: HIVE-5394.12.branch.txt, HIVE-5394.1.patch, HIVE-5394.2.patch.txt, HIVE-5394.3.patch The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5382) Allow strings represented as exponential notation to be typecasted to int/smallint/bigint/tinyint
[ https://issues.apache.org/jira/browse/HIVE-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785562#comment-13785562 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5382: - Hi Gunther, Thanks for the review for addressing those issues. It seems like e notation explicitly means 10^value. I need to look more into how java language handles similar scenarios . One question is why cant we directly use 'Double.valueOf(string)' for this conversion as defined in http://developer.classpath.org/doc/java/lang/Double-source.html Theoretically, it should tally with the results that we obtain if we cast one number type to another number type in Hive. The bottom line is that the issues that Gunther addressed in RB need to be clearly understood and addressed before we can merge in the change. This will be fixed by 0.13. Thanks Hari Allow strings represented as exponential notation to be typecasted to int/smallint/bigint/tinyint - Key: HIVE-5382 URL: https://issues.apache.org/jira/browse/HIVE-5382 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5382.1.patch Follow up jira for HIVE-5352 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5348) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters
[ https://issues.apache.org/jira/browse/HIVE-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5348: Attachment: HIVE-5348.2.patch Made changes to resolve the failures hopefully If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters -- Key: HIVE-5348 URL: https://issues.apache.org/jira/browse/HIVE-5348 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5348.1.patch.txt, HIVE-5348.2.patch Follow-up JIRA for HIVE-5202 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5336) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user
[ https://issues.apache.org/jira/browse/HIVE-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5336: Attachment: HIVE-5336.3.patch same patch as 1 and 2. HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user -- Key: HIVE-5336 URL: https://issues.apache.org/jira/browse/HIVE-5336 Project: Hive Issue Type: Bug Components: HCatalog Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5336.1.patch.txt, HIVE-5336.2.patch, HIVE-5336.3.patch HCatSchema.remove currently does not renumber the fieldPositionMap which can be a problem when there are interleaving append() and remove() calls. 1. We should document that fieldPositionMap should not be cached by the end-user 2. We should make sure that the fieldPositionMap gets renumbered after remove() because HcatSchema.get will otherwise return wrong FieldSchemas. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5436) Hive's casting behavior needs to be consistent
Hari Sankar Sivarama Subramaniyan created HIVE-5436:
---
Summary: Hive's casting behavior needs to be consistent
Key: HIVE-5436
URL: https://issues.apache.org/jira/browse/HIVE-5436
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Priority: Critical
Hive's casting behavior is inconsistent and the behavior of casting from one
type to another undocumented as of now when the casted value is out of range.
For example, casting out of range values from one type to another can result in
incorrect results.
Eg:
1. select cast('1000' as tinyint) from t1;
NULL
2. select 1000Y from t1;
FAILED: SemanticException [Error 10029]: Line 1:7 Invalid numerical constant
'1000Y'
3. select cast(1000 as tinyint) from t1;
-24
4.select cast(1.1e3-1000/0 as tinyint) from t1;
0
5. select cast(10/0 as tinyint) from pw18;
-1
The hive user can accidently try to typecast an out of range value. For example
in the e.g. 4/5 even though the final result is NaN, Hive can typecast to a
random result. Either we should document that the end user should take care of
overflow, underflow, division by 0, etc. by himself/herself or we should
return NULLs when the final result is out of range.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5394: Attachment: HIVE-5394.3.patch Test case fixed. ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Attachments: HIVE-5394.12.branch.txt, HIVE-5394.1.patch, HIVE-5394.2.patch.txt, HIVE-5394.3.patch The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5394: Attachment: (was: HIVE-5394.3.patch) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Attachments: HIVE-5394.12.branch.txt, HIVE-5394.1.patch, HIVE-5394.2.patch.txt The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5394: Attachment: HIVE-5394.3.patch ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Attachments: HIVE-5394.12.branch.txt, HIVE-5394.1.patch, HIVE-5394.2.patch.txt, HIVE-5394.3.patch The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5394: Attachment: HIVE-5394.2.patch.txt ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Attachments: HIVE-5394.12.branch.txt, HIVE-5394.1.patch, HIVE-5394.2.patch.txt The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5394: Status: Patch Available (was: Open) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Attachments: HIVE-5394.12.branch.txt, HIVE-5394.1.patch, HIVE-5394.2.patch.txt The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5382) Allow strings represented as exponential notation to be typecasted to int/smallint/bigint/tinyint
[ https://issues.apache.org/jira/browse/HIVE-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5382: Status: Patch Available (was: Open) Allow strings represented as exponential notation to be typecasted to int/smallint/bigint/tinyint - Key: HIVE-5382 URL: https://issues.apache.org/jira/browse/HIVE-5382 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5382.1.patch Follow up jira for HIVE-5352 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5382) Allow strings represented as exponential notation to be typecasted to int/smallint/bigint/tinyint
[ https://issues.apache.org/jira/browse/HIVE-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5382: Attachment: HIVE-5382.1.patch Allow exponential notation when the final result is within bounds. Allow strings represented as exponential notation to be typecasted to int/smallint/bigint/tinyint - Key: HIVE-5382 URL: https://issues.apache.org/jira/browse/HIVE-5382 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5382.1.patch Follow up jira for HIVE-5352 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5382) Allow strings represented as exponential notation to be typecasted to int/smallint/bigint/tinyint
[ https://issues.apache.org/jira/browse/HIVE-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783608#comment-13783608 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5382: - https://reviews.apache.org/r/14430/ Allow strings represented as exponential notation to be typecasted to int/smallint/bigint/tinyint - Key: HIVE-5382 URL: https://issues.apache.org/jira/browse/HIVE-5382 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5382.1.patch Follow up jira for HIVE-5352 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5348) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters
[ https://issues.apache.org/jira/browse/HIVE-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5348: Status: Patch Available (was: Open) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters -- Key: HIVE-5348 URL: https://issues.apache.org/jira/browse/HIVE-5348 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5348.1.patch.txt Follow-up JIRA for HIVE-5202 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5394: Attachment: HIVE-5394.1.patch 1. Inside ObjectInspectorConverters.getConvertedOI(), primitive type should return writable output object inspector. 2. Add more comments to understand the fn better 3. Added a test case ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Critical Attachments: HIVE-5394.1.patch The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5394: Status: Patch Available (was: Open) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Critical Attachments: HIVE-5394.1.patch The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5348) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters
[ https://issues.apache.org/jira/browse/HIVE-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5348: Attachment: (was: HIVE-5348.1.patch.txt) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters -- Key: HIVE-5348 URL: https://issues.apache.org/jira/browse/HIVE-5348 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5348.1.patch.txt Follow-up JIRA for HIVE-5202 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5348) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters
[ https://issues.apache.org/jira/browse/HIVE-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5348: Attachment: HIVE-5348.1.patch.txt If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters -- Key: HIVE-5348 URL: https://issues.apache.org/jira/browse/HIVE-5348 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5348.1.patch.txt Follow-up JIRA for HIVE-5202 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5348) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters
[ https://issues.apache.org/jira/browse/HIVE-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782172#comment-13782172 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5348: - RB Link : https://reviews.apache.org/r/14417/ If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters -- Key: HIVE-5348 URL: https://issues.apache.org/jira/browse/HIVE-5348 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5348.1.patch.txt Follow-up JIRA for HIVE-5202 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5394: Attachment: HIVE-5394.12.branch.txt I am attaching the 0.12 branch patch separately because the one in the trunk will not apply cleanly over the existing 0.12 branch. Thanks Hari ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Attachments: HIVE-5394.12.branch.txt, HIVE-5394.1.patch The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5298) AvroSerde performance problem caused by HIVE-3833
[
https://issues.apache.org/jira/browse/HIVE-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13780749#comment-13780749
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-5298:
-
[~xuefuz] and [~ashutoshc] : I looked at the exact piece of code and thought of
doing a similar optimization mentioned here while looking at one of my jiras,
HIVE-5348. It seems like
1. conf.getPathToAliases() gives the path to aliases mapping
2. conf.getPathToPartitionInfo() gives the path to partition info mapping
It is clear that (1) and (2) return HashMaps of the same size, say numPaths.
In the change [~xuefuz] added the below line ,
{code:title=MapOperator.java|borderStyle=solid}
...
SetPartitionDesc partDescSet =
new HashSetPartitionDesc(conf.getPathToPartitionInfo().values());
...
{code}
The size of partDescSet returns the number of distinct partitions associated
with the map operator. The size of the above partDescSet, say numParts, can be
way less than numPaths if a partition comprises of many files. Hence the
relatively less # of iterations.
Hence, I would +1 since the idea behind this fix looks correct.
NB: The contents of the for loop in the original code looks kind of hairy and I
am rewriting the contents of the for loop as part of HIVE-5348.
Thanks,
Hari
AvroSerde performance problem caused by HIVE-3833
-
Key: HIVE-5298
URL: https://issues.apache.org/jira/browse/HIVE-5298
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Fix For: 0.13.0
Attachments: HIVE-5298.1.patch, HIVE-5298.patch
HIVE-3833 fixed the targeted problem and made Hive to use partition-level
metadata to initialize object inspector. In doing that, however, it goes thru
every file under the table to access the partition metadata, which is very
inefficient, especially in case of multiple files per partition. This causes
more problem for AvroSerde because AvroSerde initialization accesses schema,
which is located on file system. As a result, before hive can process any
data, it needs to access every file for a table, which can take long enough
to cause job failure because of lack of job progress.
The improvement can be made so that partition metadata is only access once
per partition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5382) Allow strings represented as exponential notation to be typecasted to int/smallint/bigint/tinyint
Hari Sankar Sivarama Subramaniyan created HIVE-5382: --- Summary: Allow strings represented as exponential notation to be typecasted to int/smallint/bigint/tinyint Key: HIVE-5382 URL: https://issues.apache.org/jira/browse/HIVE-5382 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Follow up jira for HIVE-5352 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5352) cast('1.0' as int) returns null
[
https://issues.apache.org/jira/browse/HIVE-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13779760#comment-13779760
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-5352:
-
Would suggest that we document this behavior in
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types
cast('1.0' as int) returns null
---
Key: HIVE-5352
URL: https://issues.apache.org/jira/browse/HIVE-5352
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Fix For: 0.12.0
Attachments: HIVE-5352.1.patch, HIVE-5352.2.patch, HIVE-5352.3.patch,
HIVE-5352.4.patch, HIVE-5352.5.patch
Casting strings to int/smallint/bigint/tinyint yields null if the string
isn't a 'pure' integer. '1.0', '2.4' all return null. I think for those cases
the cast should return the truncated int (i.e.: if c is string, cast(c as
int) should be the same as cast(cast(c as float) as int).
This is in line with the standard and is the same behavior as mysql and
oracle. (postgres and sql server throw error, see first answer here:
http://social.msdn.microsoft.com/Forums/sqlserver/en-US/af3eff9c-737b-42fe-9016-05da9203a667/oracle-does-understand-cast10-as-int-why-sql-server-does-not)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5394: Affects Version/s: 0.12.0 ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Critical The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
Hari Sankar Sivarama Subramaniyan created HIVE-5394: --- Summary: ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Critical The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5336) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user
[ https://issues.apache.org/jira/browse/HIVE-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5336: Attachment: HIVE-5336.2.patch HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user -- Key: HIVE-5336 URL: https://issues.apache.org/jira/browse/HIVE-5336 Project: Hive Issue Type: Bug Components: HCatalog Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5336.1.patch.txt, HIVE-5336.2.patch HCatSchema.remove currently does not renumber the fieldPositionMap which can be a problem when there are interleaving append() and remove() calls. 1. We should document that fieldPositionMap should not be cached by the end-user 2. We should make sure that the fieldPositionMap gets renumbered after remove() because HcatSchema.get will otherwise return wrong FieldSchemas. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5231) Remove TestSerDe.jar from data/files
[ https://issues.apache.org/jira/browse/HIVE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13779293#comment-13779293 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5231: - [~ashutoshc] Removing a class from classpath is not trivial within ant. One method would be to rename TestSerDe.class to TestSerDe.class.unused just before junit tests are run so that this class is never accessed through test.classpath while the tests are running. Once the junit tests complete, I revert the class to the original file name so that it can be used elsewhere if required. I am uploading patch 3 with the above change. Let me know if there is a better way to implement this fix. Remove TestSerDe.jar from data/files Key: HIVE-5231 URL: https://issues.apache.org/jira/browse/HIVE-5231 Project: Hive Issue Type: Bug Affects Versions: 0.11.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5231.1.patch.txt, HIVE-5231.2.patch.txt TestSerDe.jar should be removed from data/files. Even though, TestSerDe.java is present in ql/src/test/org/apache/hadoop/hive/serde2/TestSerDe.java, this is never compiled during build process. The jar file should be created as part of build process for testing purpose rather than using a hard-coded jar file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5231) Remove TestSerDe.jar from data/files
[ https://issues.apache.org/jira/browse/HIVE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5231: Attachment: HIVE-5231.3.patch.txt Make sure that all the tests pass after data/files/TestSerDe.jar is removed. Remove TestSerDe.jar from data/files Key: HIVE-5231 URL: https://issues.apache.org/jira/browse/HIVE-5231 Project: Hive Issue Type: Bug Affects Versions: 0.11.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5231.1.patch.txt, HIVE-5231.2.patch.txt, HIVE-5231.3.patch.txt TestSerDe.jar should be removed from data/files. Even though, TestSerDe.java is present in ql/src/test/org/apache/hadoop/hive/serde2/TestSerDe.java, this is never compiled during build process. The jar file should be created as part of build process for testing purpose rather than using a hard-coded jar file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5231) Remove TestSerDe.jar from data/files
[ https://issues.apache.org/jira/browse/HIVE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5231: Attachment: HIVE-5231.4.patch.txt Deleting the class right after the jar file is constructed. Remove TestSerDe.jar from data/files Key: HIVE-5231 URL: https://issues.apache.org/jira/browse/HIVE-5231 Project: Hive Issue Type: Bug Affects Versions: 0.11.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5231.1.patch.txt, HIVE-5231.2.patch.txt, HIVE-5231.3.patch.txt, HIVE-5231.4.patch.txt TestSerDe.jar should be removed from data/files. Even though, TestSerDe.java is present in ql/src/test/org/apache/hadoop/hive/serde2/TestSerDe.java, this is never compiled during build process. The jar file should be created as part of build process for testing purpose rather than using a hard-coded jar file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5348) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters
[ https://issues.apache.org/jira/browse/HIVE-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5348: Summary: If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters (was: If both table and partition has same serde than we don't even need to check for OI conversions in ObjectInspectorConverters) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters -- Key: HIVE-5348 URL: https://issues.apache.org/jira/browse/HIVE-5348 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Follow-up JIRA for HIVE-5202 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5352) cast('1.0' as int) returns null
[
https://issues.apache.org/jira/browse/HIVE-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778002#comment-13778002
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-5352:
-
[~hagleitn] will this change work for cast('1.2e5' as int) ?
cast('1.0' as int) returns null
---
Key: HIVE-5352
URL: https://issues.apache.org/jira/browse/HIVE-5352
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Attachments: HIVE-5352.1.patch, HIVE-5352.2.patch, HIVE-5352.3.patch
Casting strings to int/smallint/bigint/tinyint yields null if the string
isn't a 'pure' integer. '1.0', '2.4', '1e5' all return null. I think for
those cases the cast should return the truncated int (i.e.: if c is string,
cast(c as int) should be the same as cast(cast(c as float) as int).
This is in line with the standard and is the same behavior as mysql and
oracle. (postgres and sql server throw error, see first answer here:
http://social.msdn.microsoft.com/Forums/sqlserver/en-US/af3eff9c-737b-42fe-9016-05da9203a667/oracle-does-understand-cast10-as-int-why-sql-server-does-not)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5352) cast('1.0' as int) returns null
[
https://issues.apache.org/jira/browse/HIVE-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778097#comment-13778097
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-5352:
-
[~hagleitn] Alright. Thanks for the explanation. The reason I asked that was
because between patch 2 and patch 4, cast('+1e5' as int) turned from a valid
casted value to NULL. Another edge case is when the input terminates with a
decimal as in cast('.' as int) or cast('2.' as int) the code would return 0
and 2 respectively instead of NULL. We should expect this behavior with this
change and it is not a bug :)
cast('1.0' as int) returns null
---
Key: HIVE-5352
URL: https://issues.apache.org/jira/browse/HIVE-5352
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Attachments: HIVE-5352.1.patch, HIVE-5352.2.patch, HIVE-5352.3.patch,
HIVE-5352.4.patch
Casting strings to int/smallint/bigint/tinyint yields null if the string
isn't a 'pure' integer. '1.0', '2.4' all return null. I think for those cases
the cast should return the truncated int (i.e.: if c is string, cast(c as
int) should be the same as cast(cast(c as float) as int).
This is in line with the standard and is the same behavior as mysql and
oracle. (postgres and sql server throw error, see first answer here:
http://social.msdn.microsoft.com/Forums/sqlserver/en-US/af3eff9c-737b-42fe-9016-05da9203a667/oracle-does-understand-cast10-as-int-why-sql-server-does-not)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5231) Remove TestSerDe.jar from data/files
[ https://issues.apache.org/jira/browse/HIVE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5231: Attachment: HIVE-5231.2.patch.txt 1. Add a new directory with ql/test/excludedClasses which contains classes that should not be in the test classpath Remove TestSerDe.jar from data/files Key: HIVE-5231 URL: https://issues.apache.org/jira/browse/HIVE-5231 Project: Hive Issue Type: Bug Affects Versions: 0.11.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5231.1.patch.txt, HIVE-5231.2.patch.txt TestSerDe.jar should be removed from data/files. Even though, TestSerDe.java is present in ql/src/test/org/apache/hadoop/hive/serde2/TestSerDe.java, this is never compiled during build process. The jar file should be created as part of build process for testing purpose rather than using a hard-coded jar file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5231) Remove TestSerDe.jar from data/files
[ https://issues.apache.org/jira/browse/HIVE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5231: Status: Patch Available (was: Open) Remove TestSerDe.jar from data/files Key: HIVE-5231 URL: https://issues.apache.org/jira/browse/HIVE-5231 Project: Hive Issue Type: Bug Affects Versions: 0.11.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5231.1.patch.txt, HIVE-5231.2.patch.txt TestSerDe.jar should be removed from data/files. Even though, TestSerDe.java is present in ql/src/test/org/apache/hadoop/hive/serde2/TestSerDe.java, this is never compiled during build process. The jar file should be created as part of build process for testing purpose rather than using a hard-coded jar file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3615) TestNegativeCliDriver_script_broken_pipe1 is failing on trunk.
[ https://issues.apache.org/jira/browse/HIVE-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778237#comment-13778237 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-3615: - [~ashutoshc] Is this issue still there on trunk. I got this error while running the unit tests for HIVE-5336 TestNegativeCliDriver_script_broken_pipe1 is failing on trunk. -- Key: HIVE-3615 URL: https://issues.apache.org/jira/browse/HIVE-3615 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.10.0 Reporter: Ashutosh Chauhan This test is failing for a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5336) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user
[ https://issues.apache.org/jira/browse/HIVE-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13776599#comment-13776599 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5336: - The above error doesn't reproduce locally and does not look related to the changes made by me. HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user -- Key: HIVE-5336 URL: https://issues.apache.org/jira/browse/HIVE-5336 Project: Hive Issue Type: Bug Components: HCatalog Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5336.1.patch.txt HCatSchema.remove currently does not renumber the fieldPositionMap which can be a problem when there are interleaving append() and remove() calls. 1. We should document that fieldPositionMap should not be cached by the end-user 2. We should make sure that the fieldPositionMap gets renumbered after remove() because HcatSchema.get will otherwise return wrong FieldSchemas. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5336) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user
[ https://issues.apache.org/jira/browse/HIVE-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5336: Status: Patch Available (was: Open) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user -- Key: HIVE-5336 URL: https://issues.apache.org/jira/browse/HIVE-5336 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5336.1.patch.txt HCatSchema.remove currently does not renumber the fieldPositionMap which can be a problem when there are interleaving append() and remove() calls. 1. We should document that fieldPositionMap should not be cached by the end-user 2. We should make sure that the fieldPositionMap gets renumbered after remove() because HcatSchema.get will otherwise return wrong FieldSchemas. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5336) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user
[ https://issues.apache.org/jira/browse/HIVE-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5336: Attachment: HIVE-5336.1.patch.txt Added code change to re-align the columns after HCatSchema.remove(). Also added a unit test case. HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user -- Key: HIVE-5336 URL: https://issues.apache.org/jira/browse/HIVE-5336 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5336.1.patch.txt HCatSchema.remove currently does not renumber the fieldPositionMap which can be a problem when there are interleaving append() and remove() calls. 1. We should document that fieldPositionMap should not be cached by the end-user 2. We should make sure that the fieldPositionMap gets renumbered after remove() because HcatSchema.get will otherwise return wrong FieldSchemas. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5336) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user
[ https://issues.apache.org/jira/browse/HIVE-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5336: Component/s: HCatalog HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user -- Key: HIVE-5336 URL: https://issues.apache.org/jira/browse/HIVE-5336 Project: Hive Issue Type: Bug Components: HCatalog Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5336.1.patch.txt HCatSchema.remove currently does not renumber the fieldPositionMap which can be a problem when there are interleaving append() and remove() calls. 1. We should document that fieldPositionMap should not be cached by the end-user 2. We should make sure that the fieldPositionMap gets renumbered after remove() because HcatSchema.get will otherwise return wrong FieldSchemas. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5348) If both table and partition has same serde than we don't even need to check for OI conversions in ObjectInspectorConverters
Hari Sankar Sivarama Subramaniyan created HIVE-5348: --- Summary: If both table and partition has same serde than we don't even need to check for OI conversions in ObjectInspectorConverters Key: HIVE-5348 URL: https://issues.apache.org/jira/browse/HIVE-5348 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Follow-up JIRA for HIVE-5202 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5202) Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types.
[ https://issues.apache.org/jira/browse/HIVE-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13775748#comment-13775748 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5202: - [~ashutoshc] Thanks. Created HIVE-5348 as a follow-up jira. Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types. --- Key: HIVE-5202 URL: https://issues.apache.org/jira/browse/HIVE-5202 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5202.2.patch.txt, HIVE-5202.patch These 3 tasks should be accomplished as part of the following jira: 1. The current implementation lacks settable union object inspector. We can run into exception inside ObjectInspectorConverters.getConvertedOI() if there is a union. 2. Implement the following public functions for all datatypes: isSettable()- Perform shallow check to see if an object inspector is inherited from settableOI type and hasAllFieldsSettable() - Perform deep check to see if this objectInspector and all the underlying object inspectors are inherited from settableOI type. 3. ObjectInspectorConverters.getConvertedOI() is inefficient. Once (1) and (2) are implemented, add the following check: outputOI.hasAllSettableFields() should be added to return outputOI immediately if the object is entirely settable in order to prevent redundant object instantiation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4773) Templeton intermittently fail to commit output to file system
[ https://issues.apache.org/jira/browse/HIVE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13775884#comment-13775884 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-4773: - +1 Templeton intermittently fail to commit output to file system - Key: HIVE-4773 URL: https://issues.apache.org/jira/browse/HIVE-4773 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Shuaishuai Nie Assignee: Shuaishuai Nie Attachments: HIVE-4773.1.patch, HIVE-4773.2.patch, HIVE-4773.3.patch With ASV as a default FS, we saw instances where output is not fully flushed to storage before the Templeton controller process exits. This results in stdout and stderr being empty even though the job completed successfully. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5202) Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types.
[ https://issues.apache.org/jira/browse/HIVE-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5202: Attachment: HIVE-5202.patch Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types. --- Key: HIVE-5202 URL: https://issues.apache.org/jira/browse/HIVE-5202 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5202.2.patch.txt, HIVE-5202.patch These 3 tasks should be accomplished as part of the following jira: 1. The current implementation lacks settable union object inspector. We can run into exception inside ObjectInspectorConverters.getConvertedOI() if there is a union. 2. Implement the following public functions for all datatypes: isSettable()- Perform shallow check to see if an object inspector is inherited from settableOI type and hasAllFieldsSettable() - Perform deep check to see if this objectInspector and all the underlying object inspectors are inherited from settableOI type. 3. ObjectInspectorConverters.getConvertedOI() is inefficient. Once (1) and (2) are implemented, add the following check: outputOI.hasAllSettableFields() should be added to return outputOI immediately if the object is entirely settable in order to prevent redundant object instantiation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4773) Templeton intermittently fail to commit output to file system
[ https://issues.apache.org/jira/browse/HIVE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13772927#comment-13772927 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-4773: - Hi Shuaishuai I have some qns: 1. I am not sure why close() should immediately close if flush() does not perform the same thing.(Eugene' qn) 2. Inside run() of Watcher why do you need to create a new object using PrintWriter writer = new PrintWriter(out); Cant you use 'out' directly instead which will call the corresponding fns depending on the underlying class. Will this not fix the issue ? 3. Even if you add CustomFilterOutputStream class, why do you need to add flush() inside close(). This looks like you are flushing twice. 4. Do you necessarily need to make CustomFilterOutputStream class public. It doesnt look like its used elsewhere. Thanks Hari Templeton intermittently fail to commit output to file system - Key: HIVE-4773 URL: https://issues.apache.org/jira/browse/HIVE-4773 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Shuaishuai Nie Assignee: Shuaishuai Nie Attachments: HIVE-4773.1.patch, HIVE-4773.2.patch With ASV as a default FS, we saw instances where output is not fully flushed to storage before the Templeton controller process exits. This results in stdout and stderr being empty even though the job completed successfully. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5336) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user
Hari Sankar Sivarama Subramaniyan created HIVE-5336: --- Summary: HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user Key: HIVE-5336 URL: https://issues.apache.org/jira/browse/HIVE-5336 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan HCatSchema.remove currently does not renumber the fieldPositionMap which can be a problem when there are interleaving append() and remove() calls. 1. We should document that fieldPositionMap should not be cached by the end-user 2. We should make sure that the fieldPositionMap gets renumbered after remove() because HcatSchema.get will otherwise return wrong FieldSchemas. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5202) Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types.
[ https://issues.apache.org/jira/browse/HIVE-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5202: Status: In Progress (was: Patch Available) Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types. --- Key: HIVE-5202 URL: https://issues.apache.org/jira/browse/HIVE-5202 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5202.2.patch.txt These 3 tasks should be accomplished as part of the following jira: 1. The current implementation lacks settable union object inspector. We can run into exception inside ObjectInspectorConverters.getConvertedOI() if there is a union. 2. Implement the following public functions for all datatypes: isSettable()- Perform shallow check to see if an object inspector is inherited from settableOI type and hasAllFieldsSettable() - Perform deep check to see if this objectInspector and all the underlying object inspectors are inherited from settableOI type. 3. ObjectInspectorConverters.getConvertedOI() is inefficient. Once (1) and (2) are implemented, add the following check: outputOI.hasAllSettableFields() should be added to return outputOI immediately if the object is entirely settable in order to prevent redundant object instantiation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5202) Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types.
[ https://issues.apache.org/jira/browse/HIVE-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5202: Status: Patch Available (was: In Progress) Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types. --- Key: HIVE-5202 URL: https://issues.apache.org/jira/browse/HIVE-5202 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5202.2.patch.txt These 3 tasks should be accomplished as part of the following jira: 1. The current implementation lacks settable union object inspector. We can run into exception inside ObjectInspectorConverters.getConvertedOI() if there is a union. 2. Implement the following public functions for all datatypes: isSettable()- Perform shallow check to see if an object inspector is inherited from settableOI type and hasAllFieldsSettable() - Perform deep check to see if this objectInspector and all the underlying object inspectors are inherited from settableOI type. 3. ObjectInspectorConverters.getConvertedOI() is inefficient. Once (1) and (2) are implemented, add the following check: outputOI.hasAllSettableFields() should be added to return outputOI immediately if the object is entirely settable in order to prevent redundant object instantiation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HIVE-5202) Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types.
[ https://issues.apache.org/jira/browse/HIVE-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-5202 started by Hari Sankar Sivarama Subramaniyan. Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types. --- Key: HIVE-5202 URL: https://issues.apache.org/jira/browse/HIVE-5202 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan These 3 tasks should be accomplished as part of the following jira: 1. The current implementation lacks settable union object inspector. We can run into exception inside ObjectInspectorConverters.getConvertedOI() if there is a union. 2. Implement the following public functions for all datatypes: isSettable()- Perform shallow check to see if an object inspector is inherited from settableOI type and hasAllFieldsSettable() - Perform deep check to see if this objectInspector and all the underlying object inspectors are inherited from settableOI type. 3. ObjectInspectorConverters.getConvertedOI() is inefficient. Once (1) and (2) are implemented, add the following check: outputOI.hasAllSettableFields() should be added to return outputOI immediately if the object is entirely settable in order to prevent redundant object instantiation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5202) Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types.
[ https://issues.apache.org/jira/browse/HIVE-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5202: Attachment: HIVE-5202.1.patch.txt Waiting for the unit tests to complete. Done: 1. Add support for isSettable() in ObjectInspector 2. Rewrote ObjectorInspectorConverters.getConvertedOI() to include caching and hence the performance 3. Added support for settableUnionObjectInspector Pending: 1. Unit tests 2. Test cases for Union embedded within non-primitive data types for partitioned/non-partitioned serdes. 3. Will upload the RB Link once unit tests pass. Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types. --- Key: HIVE-5202 URL: https://issues.apache.org/jira/browse/HIVE-5202 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5202.1.patch.txt These 3 tasks should be accomplished as part of the following jira: 1. The current implementation lacks settable union object inspector. We can run into exception inside ObjectInspectorConverters.getConvertedOI() if there is a union. 2. Implement the following public functions for all datatypes: isSettable()- Perform shallow check to see if an object inspector is inherited from settableOI type and hasAllFieldsSettable() - Perform deep check to see if this objectInspector and all the underlying object inspectors are inherited from settableOI type. 3. ObjectInspectorConverters.getConvertedOI() is inefficient. Once (1) and (2) are implemented, add the following check: outputOI.hasAllSettableFields() should be added to return outputOI immediately if the object is entirely settable in order to prevent redundant object instantiation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5202) Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types.
[ https://issues.apache.org/jira/browse/HIVE-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5202: Status: Patch Available (was: In Progress) Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types. --- Key: HIVE-5202 URL: https://issues.apache.org/jira/browse/HIVE-5202 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5202.1.patch.txt These 3 tasks should be accomplished as part of the following jira: 1. The current implementation lacks settable union object inspector. We can run into exception inside ObjectInspectorConverters.getConvertedOI() if there is a union. 2. Implement the following public functions for all datatypes: isSettable()- Perform shallow check to see if an object inspector is inherited from settableOI type and hasAllFieldsSettable() - Perform deep check to see if this objectInspector and all the underlying object inspectors are inherited from settableOI type. 3. ObjectInspectorConverters.getConvertedOI() is inefficient. Once (1) and (2) are implemented, add the following check: outputOI.hasAllSettableFields() should be added to return outputOI immediately if the object is entirely settable in order to prevent redundant object instantiation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5202) Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types.
[ https://issues.apache.org/jira/browse/HIVE-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5202: Attachment: (was: HIVE-5202.1.patch.txt) Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types. --- Key: HIVE-5202 URL: https://issues.apache.org/jira/browse/HIVE-5202 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan These 3 tasks should be accomplished as part of the following jira: 1. The current implementation lacks settable union object inspector. We can run into exception inside ObjectInspectorConverters.getConvertedOI() if there is a union. 2. Implement the following public functions for all datatypes: isSettable()- Perform shallow check to see if an object inspector is inherited from settableOI type and hasAllFieldsSettable() - Perform deep check to see if this objectInspector and all the underlying object inspectors are inherited from settableOI type. 3. ObjectInspectorConverters.getConvertedOI() is inefficient. Once (1) and (2) are implemented, add the following check: outputOI.hasAllSettableFields() should be added to return outputOI immediately if the object is entirely settable in order to prevent redundant object instantiation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5202) Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types.
[ https://issues.apache.org/jira/browse/HIVE-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5202: Attachment: HIVE-5202.2.patch.txt Support for SettableUnionObjectInspector and implement isSettable/hasAllFieldsSettable APIs for all data types. --- Key: HIVE-5202 URL: https://issues.apache.org/jira/browse/HIVE-5202 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5202.2.patch.txt These 3 tasks should be accomplished as part of the following jira: 1. The current implementation lacks settable union object inspector. We can run into exception inside ObjectInspectorConverters.getConvertedOI() if there is a union. 2. Implement the following public functions for all datatypes: isSettable()- Perform shallow check to see if an object inspector is inherited from settableOI type and hasAllFieldsSettable() - Perform deep check to see if this objectInspector and all the underlying object inspectors are inherited from settableOI type. 3. ObjectInspectorConverters.getConvertedOI() is inefficient. Once (1) and (2) are implemented, add the following check: outputOI.hasAllSettableFields() should be added to return outputOI immediately if the object is entirely settable in order to prevent redundant object instantiation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
