[jira] [Created] (HIVE-18559) Allow hive.metastore.limit.partition.request to be user configurable
Mohit Sabharwal created HIVE-18559: -- Summary: Allow hive.metastore.limit.partition.request to be user configurable Key: HIVE-18559 URL: https://issues.apache.org/jira/browse/HIVE-18559 Project: Hive Issue Type: Bug Components: Metastore Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal HIVE-13884 introduced hive.metastore.limit.partition.request to limit the number of partitions that can be requested from the Metastore for a given table. This config is set by cluster admins to prevent ad-hoc queries from putting memory pressure on HMS. But the limit can be too restrictive for some (savvy) users who require the ability to set a higher limit for a subset of workloads (during low HMS memory pressure, for example). Such users generally instruct admins not to set this confg. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-17022) Add mode in lock debug statements
Mohit Sabharwal created HIVE-17022: -- Summary: Add mode in lock debug statements Key: HIVE-17022 URL: https://issues.apache.org/jira/browse/HIVE-17022 Project: Hive Issue Type: Improvement Components: Locking Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Priority: Trivial Currently, lock debug statements print IMPLICIT/EXPLICIT as lock mode, whereas SHARED/EXCLUSIVE/SEMI_SHARED are more useful when debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-16016) Use same PersistenceManager for metadata and notification
Mohit Sabharwal created HIVE-16016: -- Summary: Use same PersistenceManager for metadata and notification Key: HIVE-16016 URL: https://issues.apache.org/jira/browse/HIVE-16016 Project: Hive Issue Type: Bug Components: repl Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal HIVE-13966 added to support for persisting notification in the same JDO transaction as the metadata event. However, the notification is currently added using a different ObjectStore object from the one used to persist the metadata event. The notification is added using the ObjectStore constructed in DbNotificationListener, whereas the metadata event is added via the thread local ObjectStore (i.e. threadLocalMS in HiveMetaStore.HMSHandler). As a result, different PersistenceManagers (different transactions) are used to persist notification and metadata events. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
Mohit Sabharwal created HIVE-15305: -- Summary: Add tests for METASTORE_EVENT_LISTENERS Key: HIVE-15305 URL: https://issues.apache.org/jira/browse/HIVE-15305 Project: Hive Issue Type: Bug Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal HIVE-15232 reused TestDbNotificationListener to test METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-15232) Add notification events for functions and indexes
Mohit Sabharwal created HIVE-15232: -- Summary: Add notification events for functions and indexes Key: HIVE-15232 URL: https://issues.apache.org/jira/browse/HIVE-15232 Project: Hive Issue Type: Improvement Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Create/Drop Function and Create/Drop/Alter Index should also generate metastore notification events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-15145) Alter partition and fs renameDir should happen in same transaction
Mohit Sabharwal created HIVE-15145: -- Summary: Alter partition and fs renameDir should happen in same transaction Key: HIVE-15145 URL: https://issues.apache.org/jira/browse/HIVE-15145 Project: Hive Issue Type: Improvement Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal In HiveAlterTable#alterPartition, the rename partition logic is as follows: - open txn - db.alterPartition(old, new) - commit txn - fs.mkdirs - if mkdirs fails -- open txn -- db.alterPartition(new, old) -- commit txn Instead, both db.alterPartition(old, new) and fs.mkdirs can be moved under a single transaction. Currently, if the second db.alterPartition(new, old) fails, we have a partition metadata committed without corresponding partition directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14187) JDOPersistenceManager objects remain cached if MetaStoreClient#close is not called
Mohit Sabharwal created HIVE-14187: -- Summary: JDOPersistenceManager objects remain cached if MetaStoreClient#close is not called Key: HIVE-14187 URL: https://issues.apache.org/jira/browse/HIVE-14187 Project: Hive Issue Type: Bug Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal JDOPersistenceManager objects are cached in JDOPersistenceManagerFactory by DataNuclues. A new JDOPersistenceManager object gets created for every HMS thread local since ObjectStore is a thread local. In non-embedded metastore mode, JDOPersistenceManager associated with a thread only gets cleaned up if IMetaStoreClient#close is called by the client (which calls ObjectStore#shutdown which calls JDOPersistenceManager#close which in turn removes the object from cache in JDOPersistenceManagerFactory#releasePersistenceManager https://github.com/datanucleus/datanucleus-api-jdo/blob/master/src/main/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L1271), i.e. the object will remain cached if client does not call close. For example: If one interrupts out of hive CLI shell (instead of using 'exit;' command), SessionState#close does not get called, and hence IMetaStoreClient#close does not get called. Instead of relying the client to call close, it's cleaner to automatically perform RawStore related cleanup at the server end via deleteContext() which gets called when the server detects a lost/closed connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13894) Fix more json related JDK8 test failures Part 2
Mohit Sabharwal created HIVE-13894: -- Summary: Fix more json related JDK8 test failures Part 2 Key: HIVE-13894 URL: https://issues.apache.org/jira/browse/HIVE-13894 Project: Hive Issue Type: Bug Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal After merge of java8 branch to master, some more json ordering related failures -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner
Mohit Sabharwal created HIVE-13884:
--
Summary: Disallow queries fetching more than a configured number
of partitions in PartitionPruner
Key: HIVE-13884
URL: https://issues.apache.org/jira/browse/HIVE-13884
Project: Hive
Issue Type: Improvement
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Currently the PartitionPruner requests either all partitions or partitions
based on filter expression. In either scenarios, if the number of partitions
accessed is large there can be significant memory pressure at the HMS server
end.
We already have a config {{hive.limit.query.max.table.partition}} that enforces
limits on number of partitions that may be scanned per operator. But this check
happens after the PartitionPruner has already fetched all partitions.
We should add an option at PartitionPruner level to disallow queries that
attempt to access number of partitions beyond a configurable limit.
Note that {{hive.mapred.mode=strict}} disallow queries without a partition
filter in PartitionPruner, but this check accepts any query with a pruning
condition, even if partitions fetched are large. In multi-tenant environments,
admins could use more control w.r.t. number of partitions allowed based on HMS
memory capacity.
One option is to have PartitionPruner first fetch the partition names (instead
of partition specs) and throw an exception if number of partitions exceeds the
configured value. Otherwise, fetch the partition specs.
Looks like the existing {{listPartitionNames}} call could be used if extended
to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13860) Fix more json related JDK8 test failures
Mohit Sabharwal created HIVE-13860: -- Summary: Fix more json related JDK8 test failures Key: HIVE-13860 URL: https://issues.apache.org/jira/browse/HIVE-13860 Project: Hive Issue Type: Sub-task Components: Test Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13657) Spark driver stderr logs should appear in hive client logs
Mohit Sabharwal created HIVE-13657:
--
Summary: Spark driver stderr logs should appear in hive client logs
Key: HIVE-13657
URL: https://issues.apache.org/jira/browse/HIVE-13657
Project: Hive
Issue Type: Bug
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Currently, spark driver exceptions are not getting logged in beeline. Instead,
the users sees the not-so-useful:
{code}
ERROR : Failed to execute spark task, with exception
'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark
client.)'
{code}
The user has to look at HS2 logs to discover the root cause:
{code}
2015-04-01 11:33:16,048 INFO org.apache.hive.spark.client.SparkClientImpl:
15/04/01 11:33:16 WARN UserGroupInformation: PriviledgedActionException as:foo
(auth:PROXY) via hive (auth:SIMPLE)
cause:org.apache.hadoop.security.AccessControlException: Permission denied:
user=foo, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
...
{code}
We should surface these critical errors in hive client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13587) Set Hive pom to use Hadoop 2.6.1
Mohit Sabharwal created HIVE-13587: -- Summary: Set Hive pom to use Hadoop 2.6.1 Key: HIVE-13587 URL: https://issues.apache.org/jira/browse/HIVE-13587 Project: Hive Issue Type: Sub-task Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal A number of the JDK8 unit test failures are due to HADOOP-10786, fixed in Hadoop 2.6.1. * TestMiniHiveKdc.testLogin * TestHiveAuthFactory.testStartTokenManagerForDBTokenStore * TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore i.e. Hive under Kerberos is broken in Java8 unless we move dependency to hadoop 2.6.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13585) Add counter metric for direct sql failures
Mohit Sabharwal created HIVE-13585: -- Summary: Add counter metric for direct sql failures Key: HIVE-13585 URL: https://issues.apache.org/jira/browse/HIVE-13585 Project: Hive Issue Type: Bug Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal In case of direct sql failure, metastore query falls back to DataNucleus. It'd be good to record how often this happens as a metrics counter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13549) Remove jdk version specific out files from Hive2
Mohit Sabharwal created HIVE-13549: -- Summary: Remove jdk version specific out files from Hive2 Key: HIVE-13549 URL: https://issues.apache.org/jira/browse/HIVE-13549 Project: Hive Issue Type: Sub-task Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13547) Switch Hive2 tests to JDK8
Mohit Sabharwal created HIVE-13547: -- Summary: Switch Hive2 tests to JDK8 Key: HIVE-13547 URL: https://issues.apache.org/jira/browse/HIVE-13547 Project: Hive Issue Type: Bug Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Tracks switching Hive2 tests to JDK8 as discussed on dev list: http://mail-archives.apache.org/mod_mbox/hive-dev/201604.mbox/%3ccabjbx5mckfczpxpkr9kvkbaenhwvymbu_mwyfmgfs9snzni...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13409) Fix JDK8 test failures related to COLUMN_STATS_ACCURATE
Mohit Sabharwal created HIVE-13409: -- Summary: Fix JDK8 test failures related to COLUMN_STATS_ACCURATE Key: HIVE-13409 URL: https://issues.apache.org/jira/browse/HIVE-13409 Project: Hive Issue Type: Bug Components: Tests Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal 126 failures have crept into JDK8 tests since we resolved HIVE-8607 http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/ Majority relate to the ordering of a "COLUMN_STATS_ACCURATE" partition property. Looks like a simple fix, use ordered map in HiveStringUtils.getPropertiesExplain() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13099) Non-SQLOperations lead to Web UI NPE
Mohit Sabharwal created HIVE-13099:
--
Summary: Non-SQLOperations lead to Web UI NPE
Key: HIVE-13099
URL: https://issues.apache.org/jira/browse/HIVE-13099
Project: Hive
Issue Type: Bug
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
To support display of live operations in the WebUI, we record SQLOperations (in
{{liveSqlOperations}}).
However, to support historic operations, we save all operations in
{{historicSqlOperations}}, including non-SQLOperations which do not have
display entries in liveSqlOperations.
This leads to a race condition depending on whether sessions use non-sql
operations. Reproduce-able by issuing a 'set' operation.
{code}
java.lang.NullPointerException
at
org.apache.hive.generated.hiveserver2.hiveserver2_jsp._jspService(hiveserver2_jsp.java:131)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
{code}
We should save only SQLOperations in historicSqlOperations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13057) Remove duplicate copies of TableDesc property values in PartitionDesc
Mohit Sabharwal created HIVE-13057: -- Summary: Remove duplicate copies of TableDesc property values in PartitionDesc Key: HIVE-13057 URL: https://issues.apache.org/jira/browse/HIVE-13057 Project: Hive Issue Type: Bug Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal For a partitioned table, each PartitionDesc has a copy of corresponding TableDesc. While TableDesc is mutable and hence cannot be interned, it's property values can be. For a simple select on a table with 100K partitions, this cut total number of String instances by ~65%. Most replicated strings were location, serde, input/output format, column, types, table name, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13045) move guava dependency back to 14 after HIVE-12952
Mohit Sabharwal created HIVE-13045: -- Summary: move guava dependency back to 14 after HIVE-12952 Key: HIVE-13045 URL: https://issues.apache.org/jira/browse/HIVE-13045 Project: Hive Issue Type: Bug Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal HIVE-12952 removed usage of EvictingQueue, so we don't need to up dependency to guava 15 at this point - avoid version related conflicts with clients if we can avoid it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13047) Disabling Web UI leads to NullPointerException
Mohit Sabharwal created HIVE-13047:
--
Summary: Disabling Web UI leads to NullPointerException
Key: HIVE-13047
URL: https://issues.apache.org/jira/browse/HIVE-13047
Project: Hive
Issue Type: Bug
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Disabling the Web UI or it's historical query display feature can lead to
NullPointerException since {{historicSqlOperations}} is unintialized.
For ex: If hive.server2.webui.port is set to 0
{code}
Caused by: java.lang.NullPointerException: null
at
org.apache.hive.service.cli.operation.OperationManager.removeOperation(OperationManager.java:212)
at
org.apache.hive.service.cli.operation.OperationManager.closeOperation(OperationManager.java:240)
at
org.apache.hive.service.cli.session.HiveSessionImpl.closeOperation(HiveSessionImpl.java:727)
at
org.apache.hive.service.cli.CLIService.closeOperation(CLIService.java:408)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.CloseOperation(ThriftCLIService.java:664)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1513)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1498)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12966) Change some ZooKeeperHiveLockManager logs to debug
Mohit Sabharwal created HIVE-12966:
--
Summary: Change some ZooKeeperHiveLockManager logs to debug
Key: HIVE-12966
URL: https://issues.apache.org/jira/browse/HIVE-12966
Project: Hive
Issue Type: Bug
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
{{ZooKeeperHiveLockManager}} prints a info level log every time it is acquiring
or releasing a lock. For a table with 10K partitions, that's 20K+ log lines.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12949) Fix description of hive.server2.logging.operation.level
Mohit Sabharwal created HIVE-12949:
--
Summary: Fix description of hive.server2.logging.operation.level
Key: HIVE-12949
URL: https://issues.apache.org/jira/browse/HIVE-12949
Project: Hive
Issue Type: Bug
Components: Configuration
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Description of {{hive.server2.logging.operation.level}} states that this
property is available to set at session level. This is not correct. The
property is only settable at HS2 level currently.
A log4j appender is created just once currently - based on logging level
specified by this config, at OperationManager init time in
initOperationLogCapture(). And there is only one instance of OperationManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12705) Java8 Jenkins jobs failing with "Error executing HIVE-TRUNK-JAVA8-*"
Mohit Sabharwal created HIVE-12705:
--
Summary: Java8 Jenkins jobs failing with "Error executing
HIVE-TRUNK-JAVA8-*"
Key: HIVE-12705
URL: https://issues.apache.org/jira/browse/HIVE-12705
Project: Hive
Issue Type: Bug
Components: Hive
Reporter: Mohit Sabharwal
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
ex:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/144/console
{code}
+ java -cp 'target/hive-ptest-1.0-classes.jar:target/lib/*'
org.apache.hive.ptest.api.client.PTestClient --endpoint
http://ec2-174-129-184-35.compute-1.amazonaws.com:8181/hive-ptest-1.0/
--logsEndpoint http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/
--command testStart --profile trunkjava8-mr2 --password c0mmit --outputDir
target/ --testHandle HIVE-TRUNK-JAVA8-144
2015-12-16 09:56:47,201 ERROR TestExecutor.run:138 Error executing
HIVE-TRUNK-JAVA8-144 java.lang.NullPointerException: branch
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:209)
at
org.apache.hive.ptest.execution.conf.TestConfiguration.(TestConfiguration.java:101)
at
org.apache.hive.ptest.execution.conf.TestConfiguration.fromInputStream(TestConfiguration.java:276)
at
org.apache.hive.ptest.execution.conf.TestConfiguration.fromFile(TestConfiguration.java:284)
at
org.apache.hive.ptest.api.server.TestExecutor.run(TestExecutor.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12670) Fix tests failing due to invalid ConnectionDriverName
Mohit Sabharwal created HIVE-12670:
--
Summary: Fix tests failing due to invalid ConnectionDriverName
Key: HIVE-12670
URL: https://issues.apache.org/jira/browse/HIVE-12670
Project: Hive
Issue Type: Bug
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Some unit tests fail when run outside the ptest environment (i.e. when run
individually on the local box like mvn test -Dtest=TestSessionHooks) with the
following error:
{code}
Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the
"BONECP" plugin to create a ConnectionPool gave an error : The specified
datastore driver ("hive-site.xml") was not found in the CLASSPATH. Please check
your CLASSPATH specification, and the name of the driver.
{code}
This is because to support TestHiveConf, we override
{{javax.jdo.option.ConnectionDriverName}} in test hive-site file
(common/src/test/resources/hive-site.xml). However, this override gets applied
for all tests. The overriden value is invalid, which causes other tests that
attempt to initialize CliService to fail.
Instead, we should use a property exclusively used for testing like
{{hive.test.dummystats.aggregator}} so that overriding it does not affect other
tests.
Not sure why these tests pass in ptest, presumably because some other test that
comes before overrides {{javax.jdo.option.ConnectionDriverName}} to a sensible
value.
Tests failing:
TestSessionHooks
TestPlainSaslHelper
TestSessionGlobalInitFile
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12561) Add sentry jars to HS2 & HMS classpath
Mohit Sabharwal created HIVE-12561: -- Summary: Add sentry jars to HS2 & HMS classpath Key: HIVE-12561 URL: https://issues.apache.org/jira/browse/HIVE-12561 Project: Hive Issue Type: Improvement Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Priority: Minor If SENTRY_HOME is either set or can be detected at conventional location, add relevant jars to HS2 and HMS classpath. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12512) Include driver logs in execution-level Operation logs
Mohit Sabharwal created HIVE-12512:
--
Summary: Include driver logs in execution-level Operation logs
Key: HIVE-12512
URL: https://issues.apache.org/jira/browse/HIVE-12512
Project: Hive
Issue Type: Bug
Components: Logging
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Priority: Minor
When {{hive.server2.logging.operation.level}} is set to {{EXECUTION}}
(default), operation logs do not include Driver logs, which contain useful
info like total number of jobs launched, stage getting executed, etc. that help
track high-level progress. It only adds a few more lines to the output.
{code}
15/11/24 14:09:12 INFO ql.Driver: Semantic Analysis Completed
15/11/24 14:09:12 INFO ql.Driver: Starting
command(queryId=hive_20151124140909_e8cbb9bd-bac0-40b2-83d0-382de25b80d1):
select count(*) from sample_08
15/11/24 14:09:12 INFO ql.Driver: Query ID =
hive_20151124140909_e8cbb9bd-bac0-40b2-83d0-382de25b80d1
15/11/24 14:09:12 INFO ql.Driver: Total jobs = 1
...
15/11/24 14:09:40 INFO ql.Driver: MapReduce Jobs Launched:
15/11/24 14:09:40 INFO ql.Driver: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative
CPU: 3.58 sec HDFS Read: 52956 HDFS Write: 4 SUCCESS
15/11/24 14:09:40 INFO ql.Driver: Total MapReduce CPU Time Spent: 3 seconds 580
msec
15/11/24 14:09:40 INFO ql.Driver: OK
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11677) Access to opHandleSet in HiveSession should be synchronized
Mohit Sabharwal created HIVE-11677: -- Summary: Access to opHandleSet in HiveSession should be synchronized Key: HIVE-11677 URL: https://issues.apache.org/jira/browse/HIVE-11677 Project: Hive Issue Type: Bug Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal In the scenario where multiple threads share the same session, reading/writing to HiveSessionImpl.opHandleSet can lead to a race condition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11534) Improve validateTableCols error message
Mohit Sabharwal created HIVE-11534:
--
Summary: Improve validateTableCols error message
Key: HIVE-11534
URL: https://issues.apache.org/jira/browse/HIVE-11534
Project: Hive
Issue Type: Improvement
Components: Hive
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Priority: Minor
For tables created without column definition in the DDL (but referencing the
schema in the underlying file format like Avro), ObjectStore.validateTableCols
throws an exception that doesn't include the table and db name. This makes it
tedious to lookup table name in schema files.
Example:
{code}
ERROR org.apache.hadoop.hive.metastore.ObjectStore: Error retrieving statistics
via jdo
MetaException(message:Column wpp_mbrshp_hix_ik doesn't exist.)
at
org.apache.hadoop.hive.metastore.ObjectStore.validateTableCols(ObjectStore.java:6061)
at
org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:6012)
at
org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:160)
at
org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:6084)
at
org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:6076)
{code}
We should add database and the table name to the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11133) Support hive.explain.user for Spark
Mohit Sabharwal created HIVE-11133:
--
Summary: Support hive.explain.user for Spark
Key: HIVE-11133
URL: https://issues.apache.org/jira/browse/HIVE-11133
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Mohit Sabharwal
User friendly explain output ({{set hive.explain.user=true}}) should support
Spark as well.
Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11099) Add support for running negative q-tests [Spark Branch]
Mohit Sabharwal created HIVE-11099: -- Summary: Add support for running negative q-tests [Spark Branch] Key: HIVE-11099 URL: https://issues.apache.org/jira/browse/HIVE-11099 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Add support for TestSparkNegativeCliDriver TestMiniSparkOnYarnNegativeCliDriver to negative q-tests -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8119) Implement Date in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-8119: -- Assignee: Dong Chen (was: Mohit Sabharwal) Implement Date in ParquetSerde -- Key: HIVE-8119 URL: https://issues.apache.org/jira/browse/HIVE-8119 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Dong Chen Attachments: HIVE-8119.1.patch, HIVE-8119.patch Date type in Parquet is discussed here: http://mail-archives.apache.org/mod_mbox/incubator-parquet-dev/201406.mbox/%3CCAKa9qDkp7xn+H8fNZC7ms3ckd=xr8gdpe7gqgj5o+pybdem...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8119) Implement Date in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311838#comment-14311838 ] Mohit Sabharwal commented on HIVE-8119: --- Thanks, [~dongc]! LGTM +1 (non-binding) [~rdblue], could you take a quick look as well ? Implement Date in ParquetSerde -- Key: HIVE-8119 URL: https://issues.apache.org/jira/browse/HIVE-8119 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Mohit Sabharwal Attachments: HIVE-8119.1.patch, HIVE-8119.patch Date type in Parquet is discussed here: http://mail-archives.apache.org/mod_mbox/incubator-parquet-dev/201406.mbox/%3CCAKa9qDkp7xn+H8fNZC7ms3ckd=xr8gdpe7gqgj5o+pybdem...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9309) schematool fails on Postgres 8.1
[
https://issues.apache.org/jira/browse/HIVE-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1429#comment-1429
]
Mohit Sabharwal commented on HIVE-9309:
---
Yes, we need to document the {{-dbOpts}} argument. And also add
{{postgres.filter.81}} as the (only) valid value for this argument.
To the The schematool Command section, we need to add:
{code}
-dbOpts databaseOpts Backend DB specific options
{code}
As well as need another section, like:
{code}
The dbOpts is optional and can be set to:
postgres.filter.81 : Postgres upgrade scripts contain set
standard_conforming_strings command, which is not applicable for Postgress
8.1. Using this option will skip this command and, hence, prevent upgrade from
failing.
{code}
Thank you, [~leftylev] !
schematool fails on Postgres 8.1
Key: HIVE-9309
URL: https://issues.apache.org/jira/browse/HIVE-9309
Project: Hive
Issue Type: Bug
Components: Database/Schema
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Fix For: 0.15.0
Attachments: HIVE-9309.patch
Postgres upgrade scripts set {{standard_conforming_strings}} which is not
allowed in 8.1:
{code}
ERROR: parameter standard_conforming_strings cannot be changed
(state=55P02,code=0)
{code}
Postgres [8.1 Release
notes|http://www.postgresql.org/docs/8.2/static/release-8-1.html] say that
standard_conforming_strings value is read-only
Postgres [8.2
notes|http://www.postgresql.org/docs/8.2/static/release-8-2.html] say that it
can be set at runtime.
It'd be nice to address this for those still using Postgres 8.1
This patch provides a schemaTool db option postgres.filter.81 which, if
set, filters out the standard_conforming_strings statement from upgrade
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9309) schematool fails on Postgres 8.1
Mohit Sabharwal created HIVE-9309:
-
Summary: schematool fails on Postgres 8.1
Key: HIVE-9309
URL: https://issues.apache.org/jira/browse/HIVE-9309
Project: Hive
Issue Type: Bug
Components: Database/Schema
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Postgres upgrade scripts set {{standard_conforming_strings}} which is not
allowed in 8.1:
{code}
ERROR: parameter standard_conforming_strings cannot be changed
(state=55P02,code=0)
{code}
Postgres [8.1 Release
notes|http://www.postgresql.org/docs/8.2/static/release-8-1.html] say that
standard_conforming_strings value is read-only
Postgres [8.2 notes|http://www.postgresql.org/docs/8.2/static/release-8-2.html]
say that it can be set at runtime.
It'd be nice to address this for those still using Postgres 8.1
This patch provides a schemaTool db option postgres.filter.81 which, if set,
filters out the standard_conforming_strings statement from upgrade scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9309) schematool fails on Postgres 8.1
[
https://issues.apache.org/jira/browse/HIVE-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9309:
--
Status: Patch Available (was: Open)
schematool fails on Postgres 8.1
Key: HIVE-9309
URL: https://issues.apache.org/jira/browse/HIVE-9309
Project: Hive
Issue Type: Bug
Components: Database/Schema
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9309.patch
Postgres upgrade scripts set {{standard_conforming_strings}} which is not
allowed in 8.1:
{code}
ERROR: parameter standard_conforming_strings cannot be changed
(state=55P02,code=0)
{code}
Postgres [8.1 Release
notes|http://www.postgresql.org/docs/8.2/static/release-8-1.html] say that
standard_conforming_strings value is read-only
Postgres [8.2
notes|http://www.postgresql.org/docs/8.2/static/release-8-2.html] say that it
can be set at runtime.
It'd be nice to address this for those still using Postgres 8.1
This patch provides a schemaTool db option postgres.filter.81 which, if
set, filters out the standard_conforming_strings statement from upgrade
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9309) schematool fails on Postgres 8.1
[
https://issues.apache.org/jira/browse/HIVE-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9309:
--
Attachment: HIVE-9309.patch
schematool fails on Postgres 8.1
Key: HIVE-9309
URL: https://issues.apache.org/jira/browse/HIVE-9309
Project: Hive
Issue Type: Bug
Components: Database/Schema
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9309.patch
Postgres upgrade scripts set {{standard_conforming_strings}} which is not
allowed in 8.1:
{code}
ERROR: parameter standard_conforming_strings cannot be changed
(state=55P02,code=0)
{code}
Postgres [8.1 Release
notes|http://www.postgresql.org/docs/8.2/static/release-8-1.html] say that
standard_conforming_strings value is read-only
Postgres [8.2
notes|http://www.postgresql.org/docs/8.2/static/release-8-2.html] say that it
can be set at runtime.
It'd be nice to address this for those still using Postgres 8.1
This patch provides a schemaTool db option postgres.filter.81 which, if
set, filters out the standard_conforming_strings statement from upgrade
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9241) Fix TestCliDriver.testCliDriver_subquery_multiinsert
[ https://issues.apache.org/jira/browse/HIVE-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-9241: -- Status: Patch Available (was: Open) Fix TestCliDriver.testCliDriver_subquery_multiinsert Key: HIVE-9241 URL: https://issues.apache.org/jira/browse/HIVE-9241 Project: Hive Issue Type: Sub-task Components: Tests Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-9241.patch In HIVE-9239, I put incorrect fix for Java 8 failure for TestCliDriver.testCliDriver_subquery_multiinsert. Generating java version specific output for this test in this patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9241) Fix TestCliDriver.testCliDriver_subquery_multiinsert
[ https://issues.apache.org/jira/browse/HIVE-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-9241: -- Attachment: HIVE-9241.patch Fix TestCliDriver.testCliDriver_subquery_multiinsert Key: HIVE-9241 URL: https://issues.apache.org/jira/browse/HIVE-9241 Project: Hive Issue Type: Sub-task Components: Tests Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-9241.patch In HIVE-9239, I put incorrect fix for Java 8 failure for TestCliDriver.testCliDriver_subquery_multiinsert. Generating java version specific output for this test in this patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9241) Fix TestCliDriver.testCliDriver_subquery_multiinsert
Mohit Sabharwal created HIVE-9241: - Summary: Fix TestCliDriver.testCliDriver_subquery_multiinsert Key: HIVE-9241 URL: https://issues.apache.org/jira/browse/HIVE-9241 Project: Hive Issue Type: Sub-task Components: Tests Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal In HIVE-9239, I put incorrect fix for Java 8 failure for TestCliDriver.testCliDriver_subquery_multiinsert. Generating java version specific output for this test in this patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9239) Fix ordering differences due to Java 8 (Part 5)
Mohit Sabharwal created HIVE-9239:
-
Summary: Fix ordering differences due to Java 8 (Part 5)
Key: HIVE-9239
URL: https://issues.apache.org/jira/browse/HIVE-9239
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate {{ColumnAccessInfo.tableToColumnAccessMap}}
to deterministic order maps. These include {{ParseContext.topToTable}} and
relevant maps in {{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
{{FilterOperator}} has predicate ({{ExprNodeDesc}}) with children in different
order. This ordering looks to be coming from antlr. Generated version specific
output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9239) Fix ordering differences due to Java 8 (Part 5)
[
https://issues.apache.org/jira/browse/HIVE-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9239:
--
Description:
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate {{ColumnAccessInfo.tableToColumnAccessMap}}
to deterministic order maps. These include {{ParseContext.topToTable}} and
relevant maps in {{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate ({{ExprNodeDesc}}) has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
was:
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate {{ColumnAccessInfo.tableToColumnAccessMap}}
to deterministic order maps. These include {{ParseContext.topToTable}} and
relevant maps in {{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
{{FilterOperator}} has predicate ({{ExprNodeDesc}}) with children in different
order. This ordering looks to be coming from antlr. Generated version specific
output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
Fix ordering differences due to Java 8 (Part 5)
---
Key: HIVE-9239
URL: https://issues.apache.org/jira/browse/HIVE-9239
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate
{{ColumnAccessInfo.tableToColumnAccessMap}} to deterministic order maps.
These include {{ParseContext.topToTable}} and relevant maps in
{{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate ({{ExprNodeDesc}}) has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9239) Fix ordering differences due to Java 8 (Part 5)
[
https://issues.apache.org/jira/browse/HIVE-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9239:
--
Description:
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate {{ColumnAccessInfo.tableToColumnAccessMap}}
to deterministic order maps. These include {{ParseContext.topToTable}} and
relevant maps in {{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate {{ExprNodeDesc}} has children in different order. This ordering looks
to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
was:
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate {{ColumnAccessInfo.tableToColumnAccessMap}}
to deterministic order maps. These include {{ParseContext.topToTable}} and
relevant maps in {{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate ({{ExprNodeDesc}}) has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
Fix ordering differences due to Java 8 (Part 5)
---
Key: HIVE-9239
URL: https://issues.apache.org/jira/browse/HIVE-9239
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate
{{ColumnAccessInfo.tableToColumnAccessMap}} to deterministic order maps.
These include {{ParseContext.topToTable}} and relevant maps in
{{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate {{ExprNodeDesc}} has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9239) Fix ordering differences due to Java 8 (Part 5)
[
https://issues.apache.org/jira/browse/HIVE-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9239:
--
Attachment: HIVE-9239.patch
Fix ordering differences due to Java 8 (Part 5)
---
Key: HIVE-9239
URL: https://issues.apache.org/jira/browse/HIVE-9239
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9239.patch
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate
{{ColumnAccessInfo.tableToColumnAccessMap}} to deterministic order maps.
These include {{ParseContext.topToTable}} and relevant maps in
{{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate {{ExprNodeDesc}} has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9239) Fix ordering differences due to Java 8 (Part 5)
[
https://issues.apache.org/jira/browse/HIVE-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9239:
--
Status: Patch Available (was: Open)
Fix ordering differences due to Java 8 (Part 5)
---
Key: HIVE-9239
URL: https://issues.apache.org/jira/browse/HIVE-9239
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9239.patch
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate
{{ColumnAccessInfo.tableToColumnAccessMap}} to deterministic order maps.
These include {{ParseContext.topToTable}} and relevant maps in
{{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate {{ExprNodeDesc}} has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
Mohit Sabharwal created HIVE-9222:
-
Summary: Fix ordering differences due to Java 8 (Part 4)
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver_unset_view_property and
TestNegativeCliDriver.testNegativeCliDriver_unset_table_property
{{DDLSemanticAnalyzer.analyzeAlterTableProps()} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets {{((ASTNode)
tree.getChild(0)}} in different order between Java 7 and Java 8. The order is
different in {{HiveParser.statement()}} itself in {{ParseDriver.parse()}} so
this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver tests:
stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q, auto_join_without_localtask.q,
auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on these
tasks may get filtered. We see that different tasks are getting filtered out by
the condition resolver in {{ConditionTask}} in Java 8 compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
[
https://issues.apache.org/jira/browse/HIVE-9222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal reassigned HIVE-9222:
-
Assignee: Mohit Sabharwal
Fix ordering differences due to Java 8 (Part 4)
---
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver_unset_view_property and
TestNegativeCliDriver.testNegativeCliDriver_unset_table_property
{{DDLSemanticAnalyzer.analyzeAlterTableProps()} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets
{{((ASTNode) tree.getChild(0)}} in different order between Java 7 and Java 8.
The order is different in {{HiveParser.statement()}} itself in
{{ParseDriver.parse()}} so this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver
tests: stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q,
auto_join_without_localtask.q, auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on
these tasks may get filtered. We see that different tasks are getting
filtered out by the condition resolver in {{ConditionTask}} in Java 8
compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
[
https://issues.apache.org/jira/browse/HIVE-9222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9222:
--
Description:
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and
unset_table_property.q
{{DDLSemanticAnalyzer.analyzeAlterTableProps()}} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets {{((ASTNode)
tree.getChild(0)}} in different order between Java 7 and Java 8. The order is
different in {{HiveParser.statement()}} itself in {{ParseDriver.parse()}} so
this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver tests:
stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q, auto_join_without_localtask.q,
auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on these
tasks may get filtered. We see that different tasks are getting filtered out by
the condition resolver in {{ConditionTask}} in Java 8 compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
was:
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver_unset_view_property and
TestNegativeCliDriver.testNegativeCliDriver_unset_table_property
{{DDLSemanticAnalyzer.analyzeAlterTableProps()} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets {{((ASTNode)
tree.getChild(0)}} in different order between Java 7 and Java 8. The order is
different in {{HiveParser.statement()}} itself in {{ParseDriver.parse()}} so
this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver tests:
stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q, auto_join_without_localtask.q,
auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on these
tasks may get filtered. We see that different tasks are getting filtered out by
the condition resolver in {{ConditionTask}} in Java 8 compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
Fix ordering differences due to Java 8 (Part 4)
---
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and
unset_table_property.q
{{DDLSemanticAnalyzer.analyzeAlterTableProps()}} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
[jira] [Updated] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
[
https://issues.apache.org/jira/browse/HIVE-9222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9222:
--
Attachment: HIVE-9222.patch
Fix ordering differences due to Java 8 (Part 4)
---
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9222.patch
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and
unset_table_property.q
{{DDLSemanticAnalyzer.analyzeAlterTableProps()}} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets
{{((ASTNode) tree.getChild(0)}} in different order between Java 7 and Java 8.
The order is different in {{HiveParser.statement()}} itself in
{{ParseDriver.parse()}} so this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver
tests: stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q,
auto_join_without_localtask.q, auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on
these tasks may get filtered. We see that different tasks are getting
filtered out by the condition resolver in {{ConditionTask}} in Java 8
compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
[
https://issues.apache.org/jira/browse/HIVE-9222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9222:
--
Status: Patch Available (was: Open)
Fix ordering differences due to Java 8 (Part 4)
---
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9222.patch
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and
unset_table_property.q
{{DDLSemanticAnalyzer.analyzeAlterTableProps()}} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets
{{((ASTNode) tree.getChild(0)}} in different order between Java 7 and Java 8.
The order is different in {{HiveParser.statement()}} itself in
{{ParseDriver.parse()}} so this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver
tests: stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q,
auto_join_without_localtask.q, auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on
these tasks may get filtered. We see that different tasks are getting
filtered out by the condition resolver in {{ConditionTask}} in Java 8
compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
[
https://issues.apache.org/jira/browse/HIVE-9222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260447#comment-14260447
]
Mohit Sabharwal commented on HIVE-9222:
---
HIVE-9220 is not deleting list_bucket_dml_10.q.out, so changes in this patch
are more accurate.
Fix ordering differences due to Java 8 (Part 4)
---
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9222.patch
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and
unset_table_property.q
{{DDLSemanticAnalyzer.analyzeAlterTableProps()}} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets
{{((ASTNode) tree.getChild(0)}} in different order between Java 7 and Java 8.
The order is different in {{HiveParser.statement()}} itself in
{{ParseDriver.parse()}} so this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver
tests: stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q,
auto_join_without_localtask.q, auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on
these tasks may get filtered. We see that different tasks are getting
filtered out by the condition resolver in {{ConditionTask}} in Java 8
compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9220) HIVE-9109 missed updating result of list_bucket_dml_10
[
https://issues.apache.org/jira/browse/HIVE-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260453#comment-14260453
]
Mohit Sabharwal commented on HIVE-9220:
---
Thanks for catching this, [~navis]! In this case, we have to rename
{{list_bucket_dml_10.q.out}} to {{list_bucket_dml_10.java1.7.q.out}}. And also
create the java 1.8 version {{list_bucket_dml_10.java1.8.q.out}}. I have
included all these changes in HIVE-9222 patch.
If it's ok with you, should we use the changes in HIVE-9222 instead of this
patch ?
HIVE-9109 missed updating result of list_bucket_dml_10
--
Key: HIVE-9220
URL: https://issues.apache.org/jira/browse/HIVE-9220
Project: Hive
Issue Type: Sub-task
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-9109.1.patch.txt
list_bucket_dml_10.q.java1.7.out is missing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9206) Fix Desc Formatted related Java 8 ordering differences
[
https://issues.apache.org/jira/browse/HIVE-9206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9206:
--
Attachment: HIVE-9206.1.patch
Fix Desc Formatted related Java 8 ordering differences
--
Key: HIVE-9206
URL: https://issues.apache.org/jira/browse/HIVE-9206
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9206.1.patch, HIVE-9206.patch
This patch fixes the following tests for Java 8:
(1) list_bucket_dml_*.q
{{DESC FORMATTED}} calls {{StorageDescriptor.getSkewedInfo()}} HMS API, which
returns a thrift (unordered) map. Generate java version specific out file for
these tests.
(2) partitions_json.q
{{SHOW PARTITIONS}} uses {{MapBuilder}} via {{JsonMetaDataFormatter}} which
uses {{HashMap}}. Changed it to ordered map.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9206) Fix Desc Formatted related Java 8 ordering differences
[
https://issues.apache.org/jira/browse/HIVE-9206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258377#comment-14258377
]
Mohit Sabharwal commented on HIVE-9206:
---
Updated patch with fix for TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
(previous version had output generated via {{TestCliDriver}} instead
{{TestMinimrCliDriver}})
Fix Desc Formatted related Java 8 ordering differences
--
Key: HIVE-9206
URL: https://issues.apache.org/jira/browse/HIVE-9206
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9206.1.patch, HIVE-9206.patch
This patch fixes the following tests for Java 8:
(1) list_bucket_dml_*.q
{{DESC FORMATTED}} calls {{StorageDescriptor.getSkewedInfo()}} HMS API, which
returns a thrift (unordered) map. Generate java version specific out file for
these tests.
(2) partitions_json.q
{{SHOW PARTITIONS}} uses {{MapBuilder}} via {{JsonMetaDataFormatter}} which
uses {{HashMap}}. Changed it to ordered map.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9193) Fix ordering differences due to Java 8 (Part 3)
[
https://issues.apache.org/jira/browse/HIVE-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9193:
--
Attachment: HIVE-9193.1.patch
Fix ordering differences due to Java 8 (Part 3)
---
Key: HIVE-9193
URL: https://issues.apache.org/jira/browse/HIVE-9193
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9193.1.patch, HIVE-9193.patch
This patch fixes following q-test failures in Java 8:
(1) avro_* q-tests with map data type.
Select operator uses {{LazySimpleSerDe}}/{{StandMapObjectInspector}}, which
is used all over for serializing results back to the end user for queries
with maps. To avoid adding the additional memory cost associated with
{{LinkedHashMap}}, we generate java version specific out files for these
tests.
(2) plan_json.q, join0.q, input4.q, authorization_explain.q
{{EXPLAIN FORMATTED}} outputs string representation of {{JSONObject}}, which
internally is a {{HashMap}}. Again, we generate java version specific out
files for these tests.
(3) explain_dependency.q
{{EXPLAIN DEPENDENCY}} outputs string representation of
{{ReadEntity.parents}}. Changed that to an ordered map.
(4) list_bucket_dml_6.q
{{DESC FORMATTED}} calls {{StorageDescriptor.getSkewedInfo()}} HMS API, which
returns a thrift (unordered) map. Generate java version specific out file for
this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9193) Fix ordering differences due to Java 8 (Part 3)
[
https://issues.apache.org/jira/browse/HIVE-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257402#comment-14257402
]
Mohit Sabharwal commented on HIVE-9193:
---
Oops, I only renamed the old out files to 1.7, but needed to regenerate -- to
account for java_version_specific hint.
Attaching patch with the fix.
Fix ordering differences due to Java 8 (Part 3)
---
Key: HIVE-9193
URL: https://issues.apache.org/jira/browse/HIVE-9193
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9193.1.patch, HIVE-9193.patch
This patch fixes following q-test failures in Java 8:
(1) avro_* q-tests with map data type.
Select operator uses {{LazySimpleSerDe}}/{{StandMapObjectInspector}}, which
is used all over for serializing results back to the end user for queries
with maps. To avoid adding the additional memory cost associated with
{{LinkedHashMap}}, we generate java version specific out files for these
tests.
(2) plan_json.q, join0.q, input4.q, authorization_explain.q
{{EXPLAIN FORMATTED}} outputs string representation of {{JSONObject}}, which
internally is a {{HashMap}}. Again, we generate java version specific out
files for these tests.
(3) explain_dependency.q
{{EXPLAIN DEPENDENCY}} outputs string representation of
{{ReadEntity.parents}}. Changed that to an ordered map.
(4) list_bucket_dml_6.q
{{DESC FORMATTED}} calls {{StorageDescriptor.getSkewedInfo()}} HMS API, which
returns a thrift (unordered) map. Generate java version specific out file for
this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9206) Fix Desc Formatted related Java 8 ordering differences
Mohit Sabharwal created HIVE-9206:
-
Summary: Fix Desc Formatted related Java 8 ordering differences
Key: HIVE-9206
URL: https://issues.apache.org/jira/browse/HIVE-9206
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
This patch fixes the following tests for Java 8:
(1) list_bucket_dml_*.q
{{DESC FORMATTED}} calls {{StorageDescriptor.getSkewedInfo()}} HMS API, which
returns a thrift (unordered) map. Generate java version specific out file for
these tests.
(2) partitions_json.q
{{SHOW PARTITIONS}} uses {{MapBuilder}} via {{JsonMetaDataFormatter}} which
uses {{HashMap}}. Changed it to ordered map.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9206) Fix Desc Formatted related Java 8 ordering differences
[
https://issues.apache.org/jira/browse/HIVE-9206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9206:
--
Attachment: HIVE-9206.patch
Fix Desc Formatted related Java 8 ordering differences
--
Key: HIVE-9206
URL: https://issues.apache.org/jira/browse/HIVE-9206
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9206.patch
This patch fixes the following tests for Java 8:
(1) list_bucket_dml_*.q
{{DESC FORMATTED}} calls {{StorageDescriptor.getSkewedInfo()}} HMS API, which
returns a thrift (unordered) map. Generate java version specific out file for
these tests.
(2) partitions_json.q
{{SHOW PARTITIONS}} uses {{MapBuilder}} via {{JsonMetaDataFormatter}} which
uses {{HashMap}}. Changed it to ordered map.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9206) Fix Desc Formatted related Java 8 ordering differences
[
https://issues.apache.org/jira/browse/HIVE-9206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9206:
--
Status: Patch Available (was: Open)
Fix Desc Formatted related Java 8 ordering differences
--
Key: HIVE-9206
URL: https://issues.apache.org/jira/browse/HIVE-9206
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9206.patch
This patch fixes the following tests for Java 8:
(1) list_bucket_dml_*.q
{{DESC FORMATTED}} calls {{StorageDescriptor.getSkewedInfo()}} HMS API, which
returns a thrift (unordered) map. Generate java version specific out file for
these tests.
(2) partitions_json.q
{{SHOW PARTITIONS}} uses {{MapBuilder}} via {{JsonMetaDataFormatter}} which
uses {{HashMap}}. Changed it to ordered map.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9193) Fix ordering differences due to Java 8 (Part 3)
[
https://issues.apache.org/jira/browse/HIVE-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9193:
--
Attachment: HIVE-9193.2.patch
Fix ordering differences due to Java 8 (Part 3)
---
Key: HIVE-9193
URL: https://issues.apache.org/jira/browse/HIVE-9193
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9193.1.patch, HIVE-9193.2.patch, HIVE-9193.patch
This patch fixes following q-test failures in Java 8:
(1) avro_* q-tests with map data type.
Select operator uses {{LazySimpleSerDe}}/{{StandMapObjectInspector}}, which
is used all over for serializing results back to the end user for queries
with maps. To avoid adding the additional memory cost associated with
{{LinkedHashMap}}, we generate java version specific out files for these
tests.
(2) plan_json.q, join0.q, input4.q, authorization_explain.q
{{EXPLAIN FORMATTED}} outputs string representation of {{JSONObject}}, which
internally is a {{HashMap}}. Again, we generate java version specific out
files for these tests.
(3) explain_dependency.q
{{EXPLAIN DEPENDENCY}} outputs string representation of
{{ReadEntity.parents}}. Changed that to an ordered map.
(4) list_bucket_dml_6.q
{{DESC FORMATTED}} calls {{StorageDescriptor.getSkewedInfo()}} HMS API, which
returns a thrift (unordered) map. Generate java version specific out file for
this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9193) Fix ordering differences due to Java 8 (Part 3)
Mohit Sabharwal created HIVE-9193:
-
Summary: Fix ordering differences due to Java 8 (Part 3)
Key: HIVE-9193
URL: https://issues.apache.org/jira/browse/HIVE-9193
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
This patch fixes following q-test failures in Java 8:
(1) avro_* q-tests with map data type.
Select operator uses {{LazySimpleSerDe}}/{{StandMapObjectInspector}}, which is
used all over for serializing results back to the end user for queries with
maps. To avoid adding the additional memory cost associated with
{{LinkedHashMap}}, we generate java version specific out files for these tests.
(2) plan_json.q, join0.q, input4.q, authorization_explain.q
{{EXPLAIN FORMATTED}} outputs string representation of {{JSONObject}}, which
internally is a {{HashMap}}. Again, we generate java version specific out files
for these tests.
(3) explain_dependency.q
{{EXPLAIN DEPENDENCY}} outputs string representation of {{ReadEntity.parents}}.
Changed that to an ordered map.
(4) list_bucket_dml_6.q
{{DESC FORMATTED}} calls {{StorageDescriptor.getSkewedInfo()}} HMS API, which
returns a thrift (unordered) map. Generate java version specific out file for
this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9193) Fix ordering differences due to Java 8 (Part 3)
[
https://issues.apache.org/jira/browse/HIVE-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9193:
--
Attachment: HIVE-9193.patch
Fix ordering differences due to Java 8 (Part 3)
---
Key: HIVE-9193
URL: https://issues.apache.org/jira/browse/HIVE-9193
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9193.patch
This patch fixes following q-test failures in Java 8:
(1) avro_* q-tests with map data type.
Select operator uses {{LazySimpleSerDe}}/{{StandMapObjectInspector}}, which
is used all over for serializing results back to the end user for queries
with maps. To avoid adding the additional memory cost associated with
{{LinkedHashMap}}, we generate java version specific out files for these
tests.
(2) plan_json.q, join0.q, input4.q, authorization_explain.q
{{EXPLAIN FORMATTED}} outputs string representation of {{JSONObject}}, which
internally is a {{HashMap}}. Again, we generate java version specific out
files for these tests.
(3) explain_dependency.q
{{EXPLAIN DEPENDENCY}} outputs string representation of
{{ReadEntity.parents}}. Changed that to an ordered map.
(4) list_bucket_dml_6.q
{{DESC FORMATTED}} calls {{StorageDescriptor.getSkewedInfo()}} HMS API, which
returns a thrift (unordered) map. Generate java version specific out file for
this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9193) Fix ordering differences due to Java 8 (Part 3)
[
https://issues.apache.org/jira/browse/HIVE-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9193:
--
Status: Patch Available (was: Open)
Fix ordering differences due to Java 8 (Part 3)
---
Key: HIVE-9193
URL: https://issues.apache.org/jira/browse/HIVE-9193
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9193.patch
This patch fixes following q-test failures in Java 8:
(1) avro_* q-tests with map data type.
Select operator uses {{LazySimpleSerDe}}/{{StandMapObjectInspector}}, which
is used all over for serializing results back to the end user for queries
with maps. To avoid adding the additional memory cost associated with
{{LinkedHashMap}}, we generate java version specific out files for these
tests.
(2) plan_json.q, join0.q, input4.q, authorization_explain.q
{{EXPLAIN FORMATTED}} outputs string representation of {{JSONObject}}, which
internally is a {{HashMap}}. Again, we generate java version specific out
files for these tests.
(3) explain_dependency.q
{{EXPLAIN DEPENDENCY}} outputs string representation of
{{ReadEntity.parents}}. Changed that to an ordered map.
(4) list_bucket_dml_6.q
{{DESC FORMATTED}} calls {{StorageDescriptor.getSkewedInfo()}} HMS API, which
returns a thrift (unordered) map. Generate java version specific out file for
this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8773) Fix TestWebHCatE2e#getStatus for Java8
[ https://issues.apache.org/jira/browse/HIVE-8773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-8773: -- Attachment: HIVE-8773.1.patch Fix TestWebHCatE2e#getStatus for Java8 -- Key: HIVE-8773 URL: https://issues.apache.org/jira/browse/HIVE-8773 Project: Hive Issue Type: Sub-task Components: Tests Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8773.1.patch, HIVE-8773.patch [HttpMethod.getResponseBodyAsString|https://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/HttpMethod.html#getResponseBodyAsString()] returns response body in different order for Java8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9181) Fix SkewJoinOptimizer related Java 8 ordering differences
[
https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255244#comment-14255244
]
Mohit Sabharwal commented on HIVE-9181:
---
Test failures are present in other recent patch runs as well (like HIVE-8938),
so these are unrelated to patch.
Fix SkewJoinOptimizer related Java 8 ordering differences
-
Key: HIVE-9181
URL: https://issues.apache.org/jira/browse/HIVE-9181
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9181.1.patch, HIVE-9181.patch, HIVE-9181.patch
(1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different
order in Java 8 vs Java 7. This leads to differences in EXPLAIN output
related to {{MapWork}}.
(2) Dbproperties order for a desc database query is non-deterministic at
the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are
un-ordered).
{code}
create database test_db with dbproperties ('key1' = 'value1', 'key2' =
'value2');
desc database extended test_db;
{code}
At client side, we serialize the dbproperties map returned from HMS as string
and write it to a temp file. The {{FetchTask}} then reads the temp file and
returns the string back to the user. In test environment, for consistent
ordering, we should sort the map before it is serialized to the temp file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9181) Fix SkewJoinOptimizer related Java 8 ordering differences
[
https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255030#comment-14255030
]
Mohit Sabharwal commented on HIVE-9181:
---
Attaching patch with added check for when database params is null.
The nonmr_fetch.q and virtual_column.q failures are unrelated (these failing
for other patches as well).
Fix SkewJoinOptimizer related Java 8 ordering differences
-
Key: HIVE-9181
URL: https://issues.apache.org/jira/browse/HIVE-9181
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9181.patch, HIVE-9181.patch
(1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different
order in Java 8 vs Java 7. This leads to differences in EXPLAIN output
related to {{MapWork}}.
(2) Dbproperties order for a desc database query is non-deterministic at
the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are
un-ordered).
{code}
create database test_db with dbproperties ('key1' = 'value1', 'key2' =
'value2');
desc database extended test_db;
{code}
At client side, we serialize the dbproperties map returned from HMS as string
and write it to a temp file. The {{FetchTask}} then reads the temp file and
returns the string back to the user. In test environment, for consistent
ordering, we should sort the map before it is serialized to the temp file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9181) Fix SkewJoinOptimizer related Java 8 ordering differences
[
https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9181:
--
Attachment: HIVE-9181.1.patch
Fix SkewJoinOptimizer related Java 8 ordering differences
-
Key: HIVE-9181
URL: https://issues.apache.org/jira/browse/HIVE-9181
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9181.1.patch, HIVE-9181.patch, HIVE-9181.patch
(1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different
order in Java 8 vs Java 7. This leads to differences in EXPLAIN output
related to {{MapWork}}.
(2) Dbproperties order for a desc database query is non-deterministic at
the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are
un-ordered).
{code}
create database test_db with dbproperties ('key1' = 'value1', 'key2' =
'value2');
desc database extended test_db;
{code}
At client side, we serialize the dbproperties map returned from HMS as string
and write it to a temp file. The {{FetchTask}} then reads the temp file and
returns the string back to the user. In test environment, for consistent
ordering, we should sort the map before it is serialized to the temp file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9161) Fix ordering differences on UDF functions due to Java8
[ https://issues.apache.org/jira/browse/HIVE-9161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14254383#comment-14254383 ] Mohit Sabharwal commented on HIVE-9161: --- Another request, [~spena]. Could you add a comment like // Must be deterministic order map for consistent q-test output across Java versions - see HIVE-9161 ? This is so that these changes are not inadvertently reverted. Thank you! Fix ordering differences on UDF functions due to Java8 -- Key: HIVE-9161 URL: https://issues.apache.org/jira/browse/HIVE-9161 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.1 Reporter: Sergio Peña Assignee: Sergio Peña Attachments: HIVE-9161.1.patch, HIVE-9161.2.patch Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See Java8 vs Java7) This part is related to UDF functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9181) Fix SkewJoinOptimizer related Java 8 ordering differences
Mohit Sabharwal created HIVE-9181:
-
Summary: Fix SkewJoinOptimizer related Java 8 ordering differences
Key: HIVE-9181
URL: https://issues.apache.org/jira/browse/HIVE-9181
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
(1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different
order in Java 8 vs Java 7. This leads to differences in EXPLAIN output related
to {{MapWork}}.
(2) Dbproperties order for a desc database query is non-deterministic at the
HiveMetastore end between Java 8 and Java 7 (because Thrift maps are
un-ordered).
{code}
create database test_db with dbproperties ('key1' = 'value1', 'key2' =
'value2');
desc database extended test_db;
{code}
At client side, we serialize the dbproperties map returned from HMS as string
and write it to a temp file. The {{FetchTask}} then reads the temp file and
returns the string back to the user. In test environment, for consistent
ordering, we should sort the map before it is serialized to the temp file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9181) Fix SkewJoinOptimizer related Java 8 ordering differences
[
https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9181:
--
Attachment: HIVE-9181.patch
Fix SkewJoinOptimizer related Java 8 ordering differences
-
Key: HIVE-9181
URL: https://issues.apache.org/jira/browse/HIVE-9181
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9181.patch
(1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different
order in Java 8 vs Java 7. This leads to differences in EXPLAIN output
related to {{MapWork}}.
(2) Dbproperties order for a desc database query is non-deterministic at
the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are
un-ordered).
{code}
create database test_db with dbproperties ('key1' = 'value1', 'key2' =
'value2');
desc database extended test_db;
{code}
At client side, we serialize the dbproperties map returned from HMS as string
and write it to a temp file. The {{FetchTask}} then reads the temp file and
returns the string back to the user. In test environment, for consistent
ordering, we should sort the map before it is serialized to the temp file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9181) Fix SkewJoinOptimizer related Java 8 ordering differences
[
https://issues.apache.org/jira/browse/HIVE-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9181:
--
Status: Patch Available (was: Open)
Fix SkewJoinOptimizer related Java 8 ordering differences
-
Key: HIVE-9181
URL: https://issues.apache.org/jira/browse/HIVE-9181
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9181.patch
(1) {{SkewJoinOptimizer}} adds items to {{ParseContext.topOps}} is different
order in Java 8 vs Java 7. This leads to differences in EXPLAIN output
related to {{MapWork}}.
(2) Dbproperties order for a desc database query is non-deterministic at
the HiveMetastore end between Java 8 and Java 7 (because Thrift maps are
un-ordered).
{code}
create database test_db with dbproperties ('key1' = 'value1', 'key2' =
'value2');
desc database extended test_db;
{code}
At client side, we serialize the dbproperties map returned from HMS as string
and write it to a temp file. The {{FetchTask}} then reads the temp file and
returns the string back to the user. In test environment, for consistent
ordering, we should sort the map before it is serialized to the temp file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9161) Fix ordering differences on UDF functions due to Java8
[
https://issues.apache.org/jira/browse/HIVE-9161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252268#comment-14252268
]
Mohit Sabharwal commented on HIVE-9161:
---
Hi [~spena], [~brocknoland] and I were discussing this patch offline. Since
{{StandardMapObjectInspector}} map can get arbitrarily large and is extensively
used for serializing/deserializing results back to the end user for queries
with maps, it might be better to avoid adding the additional memory cost
associated with {{LinkedHashMap}}. To provide an indirect way of fixing such
test failures, we added Java version specific q-test support in HIVE-9109.
(That jira also addresses {{varchar_udf1.q}} test failure.)
As such, I think we can remove {{StandardMapObjectInspector}} related changes
from this patch. What do you think ?
Fix ordering differences on UDF functions due to Java8
--
Key: HIVE-9161
URL: https://issues.apache.org/jira/browse/HIVE-9161
Project: Hive
Issue Type: Sub-task
Affects Versions: 0.13.1
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-9161.1.patch, HIVE-9161.2.patch
Java 8 uses a different hash function for HashMap, which is leading to
iteration order differences in several cases. (See Java8 vs Java7)
This part is related to UDF functions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9109) Add support for Java 8 specific q-test out files
Mohit Sabharwal created HIVE-9109:
-
Summary: Add support for Java 8 specific q-test out files
Key: HIVE-9109
URL: https://issues.apache.org/jira/browse/HIVE-9109
Project: Hive
Issue Type: Sub-task
Components: Testing Infrastructure
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Hash function differences between Java 7 and Java 8 lead to result order
differences. While we have been able to fix a good number by converting hash
maps to insert order hash maps, there are several cases where doing so is
either not possible (because changes originate in external APIs) or change
leads to even more out file differences.
For example:
(1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
{code}
select
str_to_map('a:1,b:2,c:3',',',':'),
str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
from varchar_udf_1 limit 1;”)
{code}
the {{StandardMapObjectInspector}} used in {{LazySimpleSerDe}} to serialize the
final output uses a {{HashMap}}. Changing it to {{LinkedHashMap}} will lead to
several other q-test output differences.
(2) In TestCliDriver.testCliDriver_parquet_map_null, data with {{map}} column
is read from an Avro table into a Parquet table. Avro API, specifically
{{GenericData.Record}} uses {{HashMap}} and returns data in different order.
This patch adds supports to specify a hint called
{{JAVA_VERSION_SPECIFIC_OUTPUT}} which may be added to a q-test, only if
different outputs are expected for Java 7 and Java 8.
Under Java 7, test output file has (original) .out extension.
Under Java 8, test output file has .java8.out extension.
If hint is not added, we continue to generate a single .out file for the test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9109) Add support for Java 8 specific q-test out files
[
https://issues.apache.org/jira/browse/HIVE-9109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9109:
--
Attachment: HIVE-9109.patch
Add support for Java 8 specific q-test out files
Key: HIVE-9109
URL: https://issues.apache.org/jira/browse/HIVE-9109
Project: Hive
Issue Type: Sub-task
Components: Testing Infrastructure
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9109.patch
Hash function differences between Java 7 and Java 8 lead to result order
differences. While we have been able to fix a good number by converting hash
maps to insert order hash maps, there are several cases where doing so is
either not possible (because changes originate in external APIs) or change
leads to even more out file differences.
For example:
(1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
{code}
select
str_to_map('a:1,b:2,c:3',',',':'),
str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
from varchar_udf_1 limit 1;”)
{code}
the {{StandardMapObjectInspector}} used in {{LazySimpleSerDe}} to serialize
the final output uses a {{HashMap}}. Changing it to {{LinkedHashMap}} will
lead to several other q-test output differences.
(2) In TestCliDriver.testCliDriver_parquet_map_null, data with {{map}} column
is read from an Avro table into a Parquet table. Avro API, specifically
{{GenericData.Record}} uses {{HashMap}} and returns data in different order.
This patch adds supports to specify a hint called
{{JAVA_VERSION_SPECIFIC_OUTPUT}} which may be added to a q-test, only if
different outputs are expected for Java 7 and Java 8.
Under Java 7, test output file has (original) .out extension.
Under Java 8, test output file has .java8.out extension.
If hint is not added, we continue to generate a single .out file for the
test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9109) Add support for Java 8 specific q-test out files
[
https://issues.apache.org/jira/browse/HIVE-9109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9109:
--
Status: Patch Available (was: Open)
Add support for Java 8 specific q-test out files
Key: HIVE-9109
URL: https://issues.apache.org/jira/browse/HIVE-9109
Project: Hive
Issue Type: Sub-task
Components: Testing Infrastructure
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9109.patch
Hash function differences between Java 7 and Java 8 lead to result order
differences. While we have been able to fix a good number by converting hash
maps to insert order hash maps, there are several cases where doing so is
either not possible (because changes originate in external APIs) or change
leads to even more out file differences.
For example:
(1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
{code}
select
str_to_map('a:1,b:2,c:3',',',':'),
str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
from varchar_udf_1 limit 1;”)
{code}
the {{StandardMapObjectInspector}} used in {{LazySimpleSerDe}} to serialize
the final output uses a {{HashMap}}. Changing it to {{LinkedHashMap}} will
lead to several other q-test output differences.
(2) In TestCliDriver.testCliDriver_parquet_map_null, data with {{map}} column
is read from an Avro table into a Parquet table. Avro API, specifically
{{GenericData.Record}} uses {{HashMap}} and returns data in different order.
This patch adds supports to specify a hint called
{{JAVA_VERSION_SPECIFIC_OUTPUT}} which may be added to a q-test, only if
different outputs are expected for Java 7 and Java 8.
Under Java 7, test output file has (original) .out extension.
Under Java 8, test output file has .java8.out extension.
If hint is not added, we continue to generate a single .out file for the
test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9109) Add support for Java 8 specific q-test out files
[
https://issues.apache.org/jira/browse/HIVE-9109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14247361#comment-14247361
]
Mohit Sabharwal commented on HIVE-9109:
---
Attached patch adds support for {{JAVA_VERSION_SPECIFIC_OUTPUT}} and adds new
Java8 specific out file for {{varchar_udf1.q}} q-test.
Add support for Java 8 specific q-test out files
Key: HIVE-9109
URL: https://issues.apache.org/jira/browse/HIVE-9109
Project: Hive
Issue Type: Sub-task
Components: Testing Infrastructure
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9109.patch
Hash function differences between Java 7 and Java 8 lead to result order
differences. While we have been able to fix a good number by converting hash
maps to insert order hash maps, there are several cases where doing so is
either not possible (because changes originate in external APIs) or change
leads to even more out file differences.
For example:
(1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
{code}
select
str_to_map('a:1,b:2,c:3',',',':'),
str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
from varchar_udf_1 limit 1;”)
{code}
the {{StandardMapObjectInspector}} used in {{LazySimpleSerDe}} to serialize
the final output uses a {{HashMap}}. Changing it to {{LinkedHashMap}} will
lead to several other q-test output differences.
(2) In TestCliDriver.testCliDriver_parquet_map_null, data with {{map}} column
is read from an Avro table into a Parquet table. Avro API, specifically
{{GenericData.Record}} uses {{HashMap}} and returns data in different order.
This patch adds supports to specify a hint called
{{JAVA_VERSION_SPECIFIC_OUTPUT}} which may be added to a q-test, only if
different outputs are expected for Java 7 and Java 8.
Under Java 7, test output file has (original) .out extension.
Under Java 8, test output file has .java8.out extension.
If hint is not added, we continue to generate a single .out file for the
test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9109) Add support for Java 8 specific q-test out files
[
https://issues.apache.org/jira/browse/HIVE-9109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9109:
--
Attachment: HIVE-9109.1.patch
Add support for Java 8 specific q-test out files
Key: HIVE-9109
URL: https://issues.apache.org/jira/browse/HIVE-9109
Project: Hive
Issue Type: Sub-task
Components: Testing Infrastructure
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9109.1.patch, HIVE-9109.patch
Hash function differences between Java 7 and Java 8 lead to result order
differences. While we have been able to fix a good number by converting hash
maps to insert order hash maps, there are several cases where doing so is
either not possible (because changes originate in external APIs) or change
leads to even more out file differences.
For example:
(1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
{code}
select
str_to_map('a:1,b:2,c:3',',',':'),
str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
from varchar_udf_1 limit 1;”)
{code}
the {{StandardMapObjectInspector}} used in {{LazySimpleSerDe}} to serialize
the final output uses a {{HashMap}}. Changing it to {{LinkedHashMap}} will
lead to several other q-test output differences.
(2) In TestCliDriver.testCliDriver_parquet_map_null, data with {{map}} column
is read from an Avro table into a Parquet table. Avro API, specifically
{{GenericData.Record}} uses {{HashMap}} and returns data in different order.
This patch adds supports to specify a hint called
{{JAVA_VERSION_SPECIFIC_OUTPUT}} which may be added to a q-test, only if
different outputs are expected for Java 7 and Java 8.
Under Java 7, test output file has (original) .out extension.
Under Java 8, test output file has .java8.out extension.
If hint is not added, we continue to generate a single .out file for the
test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9109) Add support for Java 8 specific q-test out files
[
https://issues.apache.org/jira/browse/HIVE-9109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9109:
--
Description:
Hash function differences between Java 7 and Java 8 lead to result order
differences. While we have been able to fix a good number by converting hash
maps to insert order hash maps, there are several cases where doing so is
either not possible (because changes originate in external APIs) or change
leads to even more out file differences.
For example:
(1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
{code}
select
str_to_map('a:1,b:2,c:3',',',':'),
str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
from varchar_udf_1 limit 1;”)
{code}
the {{StandardMapObjectInspector}} used in {{LazySimpleSerDe}} to serialize the
final output uses a {{HashMap}}. Changing it to {{LinkedHashMap}} will lead to
several other q-test output differences.
(2) In TestCliDriver.testCliDriver_parquet_map_null, data with {{map}} column
is read from an Avro table into a Parquet table. Avro API, specifically
{{GenericData.Record}} uses {{HashMap}} and returns data in different order.
This patch adds supports to specify a hint called
{{JAVA_VERSION_SPECIFIC_OUTPUT}} which may be added to a q-test, only if
different outputs are expected for Java versions.
For example:
Under Java 7, test output file has .java1.7.out extension.
Under Java 8, test output file has .java1.8.out extension.
If hint is not added, we continue to generate a single .out file for the test.
was:
Hash function differences between Java 7 and Java 8 lead to result order
differences. While we have been able to fix a good number by converting hash
maps to insert order hash maps, there are several cases where doing so is
either not possible (because changes originate in external APIs) or change
leads to even more out file differences.
For example:
(1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
{code}
select
str_to_map('a:1,b:2,c:3',',',':'),
str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
from varchar_udf_1 limit 1;”)
{code}
the {{StandardMapObjectInspector}} used in {{LazySimpleSerDe}} to serialize the
final output uses a {{HashMap}}. Changing it to {{LinkedHashMap}} will lead to
several other q-test output differences.
(2) In TestCliDriver.testCliDriver_parquet_map_null, data with {{map}} column
is read from an Avro table into a Parquet table. Avro API, specifically
{{GenericData.Record}} uses {{HashMap}} and returns data in different order.
This patch adds supports to specify a hint called
{{JAVA_VERSION_SPECIFIC_OUTPUT}} which may be added to a q-test, only if
different outputs are expected for Java 7 and Java 8.
Under Java 7, test output file has (original) .out extension.
Under Java 8, test output file has .java8.out extension.
If hint is not added, we continue to generate a single .out file for the test.
Add support for Java 8 specific q-test out files
Key: HIVE-9109
URL: https://issues.apache.org/jira/browse/HIVE-9109
Project: Hive
Issue Type: Sub-task
Components: Testing Infrastructure
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9109.1.patch, HIVE-9109.patch
Hash function differences between Java 7 and Java 8 lead to result order
differences. While we have been able to fix a good number by converting hash
maps to insert order hash maps, there are several cases where doing so is
either not possible (because changes originate in external APIs) or change
leads to even more out file differences.
For example:
(1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
{code}
select
str_to_map('a:1,b:2,c:3',',',':'),
str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
from varchar_udf_1 limit 1;”)
{code}
the {{StandardMapObjectInspector}} used in {{LazySimpleSerDe}} to serialize
the final output uses a {{HashMap}}. Changing it to {{LinkedHashMap}} will
lead to several other q-test output differences.
(2) In TestCliDriver.testCliDriver_parquet_map_null, data with {{map}} column
is read from an Avro table into a Parquet table. Avro API, specifically
{{GenericData.Record}} uses {{HashMap}} and returns data in different order.
This patch adds supports to specify a hint called
{{JAVA_VERSION_SPECIFIC_OUTPUT}} which may be added to a q-test, only if
different outputs are expected for Java versions.
For example:
Under Java 7, test output file has .java1.7.out extension.
Under Java 8, test output file has .java1.8.out extension.
If hint is not added, we continue to generate a single .out file for the
test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9109) Add support for Java 8 specific q-test out files
[
https://issues.apache.org/jira/browse/HIVE-9109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14247616#comment-14247616
]
Mohit Sabharwal commented on HIVE-9109:
---
Attached patch addresses [~brocknoland]'s review comments.
Out file name now includes java version when {{JAVA_VERSION_SPECIFIC_OUTPUT}}
is specified in q file.
Add support for Java 8 specific q-test out files
Key: HIVE-9109
URL: https://issues.apache.org/jira/browse/HIVE-9109
Project: Hive
Issue Type: Sub-task
Components: Testing Infrastructure
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9109.1.patch, HIVE-9109.patch
Hash function differences between Java 7 and Java 8 lead to result order
differences. While we have been able to fix a good number by converting hash
maps to insert order hash maps, there are several cases where doing so is
either not possible (because changes originate in external APIs) or change
leads to even more out file differences.
For example:
(1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
{code}
select
str_to_map('a:1,b:2,c:3',',',':'),
str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
from varchar_udf_1 limit 1;”)
{code}
the {{StandardMapObjectInspector}} used in {{LazySimpleSerDe}} to serialize
the final output uses a {{HashMap}}. Changing it to {{LinkedHashMap}} will
lead to several other q-test output differences.
(2) In TestCliDriver.testCliDriver_parquet_map_null, data with {{map}} column
is read from an Avro table into a Parquet table. Avro API, specifically
{{GenericData.Record}} uses {{HashMap}} and returns data in different order.
This patch adds supports to specify a hint called
{{JAVA_VERSION_SPECIFIC_OUTPUT}} which may be added to a q-test, only if
different outputs are expected for Java versions.
For example:
Under Java 7, test output file has .java1.7.out extension.
Under Java 8, test output file has .java1.8.out extension.
If hint is not added, we continue to generate a single .out file for the
test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8131) Support timestamp in Avro
[ https://issues.apache.org/jira/browse/HIVE-8131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243819#comment-14243819 ] Mohit Sabharwal commented on HIVE-8131: --- [~Ferd], will take a look today. Support timestamp in Avro - Key: HIVE-8131 URL: https://issues.apache.org/jira/browse/HIVE-8131 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Ferdinand Xu Attachments: HIVE-8131.patch, HIVE-8131.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7444) Update supported operating systems requirements in wikidoc
[
https://issues.apache.org/jira/browse/HIVE-7444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238979#comment-14238979
]
Mohit Sabharwal commented on HIVE-7444:
---
That's right, test failures relate to result ordering differences only. There
is no impact to users.
Update supported operating systems requirements in wikidoc
Key: HIVE-7444
URL: https://issues.apache.org/jira/browse/HIVE-7444
Project: Hive
Issue Type: Bug
Components: Documentation
Reporter: Lefty Leverenz
The first sentence of Getting Started is outdated:
{quote}
DISCLAIMER: Hive has only been tested on Unix (Linux) and Mac systems using
Java 1.6 for now – although it may very well work on other similar platforms.
It does not work on Cygwin.
{quote}
The Requirements section also needs updating:
{quote}
Requirements
* Java 1.6
* Hadoop 0.20.x, 0.23.x, or 2.0.x-alpha
{quote}
Quick reference:
* [Getting Started |
https://cwiki.apache.org/confluence/display/Hive/GettingStarted]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8374) schematool fails on Postgres versions 9.2
[
https://issues.apache.org/jira/browse/HIVE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230826#comment-14230826
]
Mohit Sabharwal commented on HIVE-8374:
---
Thanks, [~leftylev]. Yes, this a bug fix, so no documentation is needed for it.
The -dbOpts option was added for future use. However, the option does appear
in the schematool usage output:
{code}
-dbOpts databaseOpts Backend DB specific options
{code}
I'd just document it when we add options for this arg. What do you think?
schematool fails on Postgres versions 9.2
---
Key: HIVE-8374
URL: https://issues.apache.org/jira/browse/HIVE-8374
Project: Hive
Issue Type: Bug
Components: Database/Schema
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Fix For: 0.15.0, 0.14.1
Attachments: HIVE-8374.1.patch, HIVE-8374.2.patch, HIVE-8374.3.patch,
HIVE-8374.patch
The upgrade script for HIVE-5700 creates an UDF with language 'plpgsql',
which is available by default only for Postgres 9.2+.
For older Postgres versions, the language must be explicitly created,
otherwise schematool fails with the error:
{code}
Error: ERROR: language plpgsql does not exist
Hint: Use CREATE LANGUAGE to load the language into the database.
(state=42704,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8916) Handle user@domain username under LDAP authentication
[
https://issues.apache.org/jira/browse/HIVE-8916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230846#comment-14230846
]
Mohit Sabharwal commented on HIVE-8916:
---
We could add documentation to [Configuration Properties --
hive.server2.authentication.ldap.Domain |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.authentication.ldap.Domain]
In case of LDAP authentication, {{hive.server2.authentication.ldap.Domain}}, if
configured, is appended to the LDAP username passed in the client connection.
This is because LDAP providers like Active Directory expect a fully qualified
username that includes the domain.
Starting 0.15.0 (HIVE-8916), if the username passed in the client connection
already includes the domain, any value configured in
{{hive.server2.authentication.ldap.Domain}} is not appended to the username.
Handle user@domain username under LDAP authentication
-
Key: HIVE-8916
URL: https://issues.apache.org/jira/browse/HIVE-8916
Project: Hive
Issue Type: Bug
Components: Authentication
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Fix For: 0.15.0
Attachments: HIVE-8916.2.patch, HIVE-8916.3.patch, HIVE-8916.patch
If LDAP is configured with multiple domains for authentication, users can be
in different domains.
Currently, LdapAuthenticationProviderImpl blindly appends the domain
configured hive.server2.authentication.ldap.Domain to the username, which
limits user to that domain. However, under multi-domain authentication, the
username may already include the domain (ex: u...@domain.foo.com). We should
not append a domain if one is already present.
Also, if username already includes the domain, rest of Hive and authorization
providers still expects the short name (user and not
u...@domain.foo.com) for looking up privilege rules, etc. As such, any
domain info in the username should be stripped off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7073) Implement Binary in ParquetSerDe
[ https://issues.apache.org/jira/browse/HIVE-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222535#comment-14222535 ] Mohit Sabharwal commented on HIVE-7073: --- Thanks [~Ferd]. Left couple more comments on RB to make test more readable. Implement Binary in ParquetSerDe Key: HIVE-7073 URL: https://issues.apache.org/jira/browse/HIVE-7073 Project: Hive Issue Type: Sub-task Reporter: David Chen Assignee: Ferdinand Xu Attachments: HIVE-7073.1.patch, HIVE-7073.2.patch, HIVE-7073.3.patch, HIVE-7073.patch The ParquetSerDe currently does not support the BINARY data type. This ticket is to implement the BINARY data type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7073) Implement Binary in ParquetSerDe
[ https://issues.apache.org/jira/browse/HIVE-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222694#comment-14222694 ] Mohit Sabharwal commented on HIVE-7073: --- +1 (non-binding) Implement Binary in ParquetSerDe Key: HIVE-7073 URL: https://issues.apache.org/jira/browse/HIVE-7073 Project: Hive Issue Type: Sub-task Reporter: David Chen Assignee: Ferdinand Xu Attachments: HIVE-7073.1.patch, HIVE-7073.2.patch, HIVE-7073.3.patch, HIVE-7073.patch The ParquetSerDe currently does not support the BINARY data type. This ticket is to implement the BINARY data type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8916) Handle user@domain username under LDAP authentication
[ https://issues.apache.org/jira/browse/HIVE-8916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222011#comment-14222011 ] Mohit Sabharwal commented on HIVE-8916: --- Both tests pass for me locally. Handle user@domain username under LDAP authentication - Key: HIVE-8916 URL: https://issues.apache.org/jira/browse/HIVE-8916 Project: Hive Issue Type: Bug Components: Authentication Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8916.2.patch, HIVE-8916.3.patch, HIVE-8916.patch If LDAP is configured with multiple domains for authentication, users can be in different domains. Currently, LdapAuthenticationProviderImpl blindly appends the domain configured hive.server2.authentication.ldap.Domain to the username, which limits user to that domain. However, under multi-domain authentication, the username may already include the domain (ex: u...@domain.foo.com). We should not append a domain if one is already present. Also, if username already includes the domain, rest of Hive and authorization providers still expects the short name (user and not u...@domain.foo.com) for looking up privilege rules, etc. As such, any domain info in the username should be stripped off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8707) Fix ordering differences due to Java 8 HashMap function
[ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-8707: -- Attachment: HIVE-8707.3.patch Fix ordering differences due to Java 8 HashMap function --- Key: HIVE-8707 URL: https://issues.apache.org/jira/browse/HIVE-8707 Project: Hive Issue Type: Sub-task Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.3.patch, HIVE-8707.patch Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29]) For example: - In Explain output: Stages are numbered differently - In PostExecutePrinter hook, table order is different in lineage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8707) Fix ordering differences due to Java 8 HashMap function
[ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221047#comment-14221047 ] Mohit Sabharwal commented on HIVE-8707: --- Updated patch with regenerated TestMiniTezCliDriver.testCliDriver_optimize_nullscan output to address ordering issue. Fix ordering differences due to Java 8 HashMap function --- Key: HIVE-8707 URL: https://issues.apache.org/jira/browse/HIVE-8707 Project: Hive Issue Type: Sub-task Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.3.patch, HIVE-8707.patch Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29]) For example: - In Explain output: Stages are numbered differently - In PostExecutePrinter hook, table order is different in lineage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8707) Fix ordering differences due to Java 8 HashMap function
[ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-8707: -- Attachment: HIVE-8707.2.patch Fix ordering differences due to Java 8 HashMap function --- Key: HIVE-8707 URL: https://issues.apache.org/jira/browse/HIVE-8707 Project: Hive Issue Type: Sub-task Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.patch Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29]) For example: - In Explain output: Stages are numbered differently - In PostExecutePrinter hook, table order is different in lineage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8707) Fix ordering differences due to Java 8 HashMap function
[ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14219740#comment-14219740 ] Mohit Sabharwal commented on HIVE-8707: --- Updated patch: - Added comment where we create the Map saying must be deterministic and referencing this JIRA. - Regenerated q-test output files Fix ordering differences due to Java 8 HashMap function --- Key: HIVE-8707 URL: https://issues.apache.org/jira/browse/HIVE-8707 Project: Hive Issue Type: Sub-task Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8707.1.patch, HIVE-8707.2.patch, HIVE-8707.patch Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29]) For example: - In Explain output: Stages are numbered differently - In PostExecutePrinter hook, table order is different in lineage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8928) Move username format detection to authentication handlers
Mohit Sabharwal created HIVE-8928: - Summary: Move username format detection to authentication handlers Key: HIVE-8928 URL: https://issues.apache.org/jira/browse/HIVE-8928 Project: Hive Issue Type: Bug Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal In HIVE-8916, [~prasadm] suggested moving the username format detection and replacement methods to PasswdAuthenticationProvider and PlainServerCallbackHandler. This way the user name format can be managed by individual authentication handlers rather than high level HS2 RPC processor. Also, explore using a regex to detect ldap domains in usernames as suggested by [~szehon], though I suspect the current way is cleaner to handle multiple matches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8916) Handle user@domain username under LDAP authentication
[ https://issues.apache.org/jira/browse/HIVE-8916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-8916: -- Attachment: HIVE-8916.2.patch Handle user@domain username under LDAP authentication - Key: HIVE-8916 URL: https://issues.apache.org/jira/browse/HIVE-8916 Project: Hive Issue Type: Bug Components: Authentication Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8916.2.patch, HIVE-8916.patch If LDAP is configured with multiple domains for authentication, users can be in different domains. Currently, LdapAuthenticationProviderImpl blindly appends the domain configured hive.server2.authentication.ldap.Domain to the username, which limits user to that domain. However, under multi-domain authentication, the username may already include the domain (ex: u...@domain.foo.com). We should not append a domain if one is already present. Also, if username already includes the domain, rest of Hive and authorization providers still expects the short name (user and not u...@domain.foo.com) for looking up privilege rules, etc. As such, any domain info in the username should be stripped off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8916) Handle user@domain username under LDAP authentication
[ https://issues.apache.org/jira/browse/HIVE-8916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-8916: -- Attachment: HIVE-8916.3.patch Handle user@domain username under LDAP authentication - Key: HIVE-8916 URL: https://issues.apache.org/jira/browse/HIVE-8916 Project: Hive Issue Type: Bug Components: Authentication Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8916.2.patch, HIVE-8916.3.patch, HIVE-8916.patch If LDAP is configured with multiple domains for authentication, users can be in different domains. Currently, LdapAuthenticationProviderImpl blindly appends the domain configured hive.server2.authentication.ldap.Domain to the username, which limits user to that domain. However, under multi-domain authentication, the username may already include the domain (ex: u...@domain.foo.com). We should not append a domain if one is already present. Also, if username already includes the domain, rest of Hive and authorization providers still expects the short name (user and not u...@domain.foo.com) for looking up privilege rules, etc. As such, any domain info in the username should be stripped off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8916) Handle user@domain username under LDAP authentication
[ https://issues.apache.org/jira/browse/HIVE-8916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14220095#comment-14220095 ] Mohit Sabharwal commented on HIVE-8916: --- Fix tests by addressing null case. Created HIVE-8928 to explore moving format detection to authentication handlers. Handle user@domain username under LDAP authentication - Key: HIVE-8916 URL: https://issues.apache.org/jira/browse/HIVE-8916 Project: Hive Issue Type: Bug Components: Authentication Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8916.2.patch, HIVE-8916.3.patch, HIVE-8916.patch If LDAP is configured with multiple domains for authentication, users can be in different domains. Currently, LdapAuthenticationProviderImpl blindly appends the domain configured hive.server2.authentication.ldap.Domain to the username, which limits user to that domain. However, under multi-domain authentication, the username may already include the domain (ex: u...@domain.foo.com). We should not append a domain if one is already present. Also, if username already includes the domain, rest of Hive and authorization providers still expects the short name (user and not u...@domain.foo.com) for looking up privilege rules, etc. As such, any domain info in the username should be stripped off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7073) Implement Binary in ParquetSerDe
[ https://issues.apache.org/jira/browse/HIVE-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14220631#comment-14220631 ] Mohit Sabharwal commented on HIVE-7073: --- Do we need really to cast cbinary? (i.e. hex(cast(cbinary as binary))). Not sure how 306230303030303131313131313030303030 is hex of binary 0b0110. Maybe I'm missing something here ? Implement Binary in ParquetSerDe Key: HIVE-7073 URL: https://issues.apache.org/jira/browse/HIVE-7073 Project: Hive Issue Type: Sub-task Reporter: David Chen Assignee: Ferdinand Xu Attachments: HIVE-7073.1.patch, HIVE-7073.2.patch, HIVE-7073.patch The ParquetSerDe currently does not support the BINARY data type. This ticket is to implement the BINARY data type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8916) Handle user@domain username under LDAP authentication
Mohit Sabharwal created HIVE-8916: - Summary: Handle user@domain username under LDAP authentication Key: HIVE-8916 URL: https://issues.apache.org/jira/browse/HIVE-8916 Project: Hive Issue Type: Bug Components: Authentication Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal If LDAP is configured with multiple domains for authentication, users can be in different domains. Currently, LdapAuthenticationProviderImpl blindly appends the domain configured hive.server2.authentication.ldap.Domain to the username, which limits user to that domain. However, under multi-domain authentication, the username may already include the domain (ex: u...@domain.foo.com). We should not append a domain if one is already present. Also, if username already includes the domain, rest of Hive and authorization providers still expects the short name (user and not u...@domain.foo.com) for looking up privilege rules, etc. As such, any domain info in the username should be stripped off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8916) Handle user@domain username under LDAP authentication
[ https://issues.apache.org/jira/browse/HIVE-8916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-8916: -- Attachment: HIVE-8916.patch Handle user@domain username under LDAP authentication - Key: HIVE-8916 URL: https://issues.apache.org/jira/browse/HIVE-8916 Project: Hive Issue Type: Bug Components: Authentication Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8916.patch If LDAP is configured with multiple domains for authentication, users can be in different domains. Currently, LdapAuthenticationProviderImpl blindly appends the domain configured hive.server2.authentication.ldap.Domain to the username, which limits user to that domain. However, under multi-domain authentication, the username may already include the domain (ex: u...@domain.foo.com). We should not append a domain if one is already present. Also, if username already includes the domain, rest of Hive and authorization providers still expects the short name (user and not u...@domain.foo.com) for looking up privilege rules, etc. As such, any domain info in the username should be stripped off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8916) Handle user@domain username under LDAP authentication
[ https://issues.apache.org/jira/browse/HIVE-8916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-8916: -- Status: Patch Available (was: Open) Handle user@domain username under LDAP authentication - Key: HIVE-8916 URL: https://issues.apache.org/jira/browse/HIVE-8916 Project: Hive Issue Type: Bug Components: Authentication Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8916.patch If LDAP is configured with multiple domains for authentication, users can be in different domains. Currently, LdapAuthenticationProviderImpl blindly appends the domain configured hive.server2.authentication.ldap.Domain to the username, which limits user to that domain. However, under multi-domain authentication, the username may already include the domain (ex: u...@domain.foo.com). We should not append a domain if one is already present. Also, if username already includes the domain, rest of Hive and authorization providers still expects the short name (user and not u...@domain.foo.com) for looking up privilege rules, etc. As such, any domain info in the username should be stripped off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8609) Move beeline to jline2
[ https://issues.apache.org/jira/browse/HIVE-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215069#comment-14215069 ] Mohit Sabharwal commented on HIVE-8609: --- We probably want to fix the typo in the license, but otherwise LGTM. +1 (non-binding) Move beeline to jline2 -- Key: HIVE-8609 URL: https://issues.apache.org/jira/browse/HIVE-8609 Project: Hive Issue Type: Task Reporter: Brock Noland Assignee: Ferdinand Xu Priority: Blocker Attachments: HIVE-8609.1.patch, HIVE-8609.2.patch, HIVE-8609.3.patch, HIVE-8609.4.patch, HIVE-8609.5.patch, HIVE-8609.6.patch, HIVE-8609.patch We found a serious bug in jline in HIVE-8565. We should move to jline2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8122) Make use of SearchArgument classes for Parquet SERDE
[ https://issues.apache.org/jira/browse/HIVE-8122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215529#comment-14215529 ] Mohit Sabharwal commented on HIVE-8122: --- +1 (non-binding) Make use of SearchArgument classes for Parquet SERDE Key: HIVE-8122 URL: https://issues.apache.org/jira/browse/HIVE-8122 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Ferdinand Xu Attachments: HIVE-8122.1.patch, HIVE-8122.2.patch, HIVE-8122.3.patch, HIVE-8122.4.patch, HIVE-8122.patch ParquetSerde could be much cleaner if we used SearchArgument and associated classes like ORC does: https://github.com/apache/hive/blob/trunk/serde/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgument.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8707) Fix ordering differences due to Java 8 HashMap function
[ https://issues.apache.org/jira/browse/HIVE-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-8707: -- Attachment: HIVE-8707.1.patch Fix ordering differences due to Java 8 HashMap function --- Key: HIVE-8707 URL: https://issues.apache.org/jira/browse/HIVE-8707 Project: Hive Issue Type: Sub-task Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8707.1.patch, HIVE-8707.patch Java 8 uses a different hash function for HashMap, which is leading to iteration order differences in several cases. (See [Java8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/HashMap.java?av=f#336] vs [Java7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/java/util/HashMap.java#HashMap.hash%28int%29]) For example: - In Explain output: Stages are numbered differently - In PostExecutePrinter hook, table order is different in lineage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
