Repository: hive Updated Branches: refs/heads/master 442a1d728 -> b267500fa
http://git-wip-us.apache.org/repos/asf/hive/blob/b267500f/RELEASE_NOTES.txt ---------------------------------------------------------------------- diff --git a/RELEASE_NOTES.txt b/RELEASE_NOTES.txt index 1e92566..ba85af7 100644 --- a/RELEASE_NOTES.txt +++ b/RELEASE_NOTES.txt @@ -1,5870 +1,629 @@ -Release Notes - Hive - Version 1.2.1 +Release Notes - Hive - Version 2.1.0 ** Sub-task - * [HIVE-10629] - Dropping table in an encrypted zone does not drop warehouse directory - * [HIVE-10630] - Renaming tables across encryption zones renames table even though the operation throws error - * [HIVE-10658] - Insert with values clause may expose data that should be encrypted - * [HIVE-10747] - Enable the cleanup of side effect for the Encryption related qfile test - * [HIVE-10910] - Alter table drop partition queries in encrypted zone failing to remove data from HDFS - - -** Bug - * [HIVE-4577] - hive CLI can't handle hadoop dfs command with space and quotes. - * [HIVE-8931] - Test TestAccumuloCliDriver is not completing - * [HIVE-9069] - Simplify filter predicates for CBO - * [HIVE-9828] - Semantic analyzer does not capture view parent entity for tables referred in view with union all - * [HIVE-9842] - Enable session/operation timeout by default in HiveServer2 - * [HIVE-10107] - Union All : Vertex missing stats resulting in OOM and in-efficient plans - * [HIVE-10244] - Vectorization : TPC-DS Q80 fails with java.lang.ClassCastException when hive.vectorized.execution.reduce.enabled is enabled - * [HIVE-10453] - HS2 leaking open file descriptors when using UDFs - * [HIVE-10528] - Hiveserver2 in HTTP mode is not applying auth_to_local rules - * [HIVE-10563] - MiniTezCliDriver tests ordering issues - * [HIVE-10605] - Make hive version number update automatically in webhcat-default.xml during hive tar generation - * [HIVE-10606] - Divide by zero error in HybridHashTableContainer - * [HIVE-10627] - Queries fail with Failed to breakup Windowing invocations into Groups - * [HIVE-10628] - Incorrect result when vectorized native mapjoin is enabled using null safe operators <=> - * [HIVE-10659] - Beeline command which contains semi-colon as a non-command terminator will fail - * [HIVE-10664] - Unit tests run fail in windows because of illegal escape character in file path - * [HIVE-10672] - Analyze command on a table using row format serde JsonSerDe fails with NoClassDefFoundError - * [HIVE-10674] - jars should not be checked in to the source control repo - * [HIVE-10675] - Provide option to skip Accumulo related Hive tests in itests directory - * [HIVE-10677] - hive.exec.parallel=true has problem when it is used for analyze table column stats - * [HIVE-10678] - update sql standard authorization configuration whitelist - more optimization flags - * [HIVE-10679] - JsonSerde ignores varchar and char size limit specified during table creation - * [HIVE-10684] - Fix the unit test failures for HIVE-7553 after HIVE-10674 removed the binary jar files - * [HIVE-10685] - Alter table concatenate oparetor will cause duplicate data - * [HIVE-10686] - java.lang.IndexOutOfBoundsException for query with rank() over(partition ...) - * [HIVE-10688] - constant folding is broken for case-when udf - * [HIVE-10689] - HS2 metadata api calls should use HiveAuthorizer interface for authorization - * [HIVE-10690] - ArrayIndexOutOfBounds exception in MetaStoreDirectSql.aggrColStatsForPartitions() - * [HIVE-10696] - TestAddResource tests are non-portable - * [HIVE-10704] - Errors in Tez HashTableLoader when estimated table size is 0 - * [HIVE-10711] - Tez HashTableLoader attempts to allocate more memory than available when HIVECONVERTJOINNOCONDITIONALTASKTHRESHOLD exceeds process max mem - * [HIVE-10719] - Hive metastore failure when alter table rename is attempted. - * [HIVE-10724] - WebHCat e2e test TestStreaming_5 fails on Windows - * [HIVE-10727] - Import throws error message "org.apache.thrift.protocol.TProtocolException: Required field 'filesAdded' is unset!" - * [HIVE-10735] - Cached plan race condition - VectorMapJoinCommonOperator has no closeOp() - * [HIVE-10736] - HiveServer2 shutdown of cached tez app-masters is not clean - * [HIVE-10741] - count distinct rewrite is not firing - * [HIVE-10745] - Better null handling by Vectorizer - * [HIVE-10746] - Hive 1.2.0+Tez produces 1-byte FileSplits from mapred.TextInputFormat - * [HIVE-10753] - hs2 jdbc url - wrong connection string cause error on beeline/jdbc/odbc client, misleading message - * [HIVE-10760] - Templeton: HCatalog Get Column for Non-existent column returns Server Error (500) rather than Not Found(404) - * [HIVE-10768] - In QTestGenTask.execute() we should not throw an exception right away if we are unable to clean any old files - * [HIVE-10776] - Schema on insert for bucketed tables throwing NullPointerException - * [HIVE-10789] - union distinct query with NULL constant on both the sides throws "Unsuported vector output type: void" error - * [HIVE-10792] - PPD leads to wrong answer when mapper scans the same table with multiple aliases - * [HIVE-10801] - 'drop view' fails throwing java.lang.NullPointerException - * [HIVE-10805] - OOM in vectorized reduce - * [HIVE-10811] - RelFieldTrimmer throws NoSuchElementException in some cases - * [HIVE-10813] - Fix current test failures after HIVE-8769 - * [HIVE-10819] - SearchArgumentImpl for Timestamp is broken by HIVE-10286 - * [HIVE-10828] - Insert with schema and dynamic partitions NullPointerException - * [HIVE-10829] - ATS hook fails for explainTask - * [HIVE-10839] - TestHCatLoaderEncryption.* tests fail in windows because of path related issues - * [HIVE-10840] - NumberFormatException while running analyze table partition compute statics query - * [HIVE-10841] - [WHERE col is not null] does not work sometimes for queries with many JOIN statements - * [HIVE-10843] - desc database and show tables commands don't pass db to HiveAuthorizer check - * [HIVE-10845] - TezJobMonitor uses killedTaskCount instead of killedTaskAttemptCount - * [HIVE-10857] - Accumulo storage handler fail throwing java.lang.IllegalArgumentException: Cannot determine SASL mechanism for token class: class org.apache.accumulo.core.client.security.tokens.PasswordToken - * [HIVE-10858] - WebHCat specific resources should be added to HADOOP_CLASSPATH first - * [HIVE-10862] - TestHiveAuthorizerShowFilters tests fail when run in sequence - * [HIVE-10874] - Fail in TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2.q due to duplicate column name - * [HIVE-10875] - Select query with view in subquery adds underlying table as direct input - * [HIVE-10877] - TestUtil class name confuses ptest2 - * [HIVE-10885] - with vectorization enabled join operation involving interval_day_time fails - * [HIVE-10887] - TestCliDriver tests ordering issues with Mac and CentOS - * [HIVE-10892] - TestHCatClient should not accept external metastore param from -Dhive.metastore.uris - * [HIVE-10907] - Hive on Tez: Classcast exception in some cases with SMB joins - * [HIVE-10919] - Windows: create table with JsonSerDe failed via beeline unless you add hcatalog core jar to classpath - * [HIVE-10922] - In HS2 doAs=false mode, file system related errors in one query causes other failures - * [HIVE-10925] - Non-static threadlocals in metastore code can potentially cause memory leak - * [HIVE-10929] - In Tez mode,dynamic partitioning query with union all fails at moveTask,Invalid partition key & values - * [HIVE-10934] - Restore support for DROP PARTITION PURGE - * [HIVE-10941] - Provide option to disable spark tests outside itests - * [HIVE-10949] - Disable hive-minikdc tests in Windows - * [HIVE-10957] - QueryPlan's start time is incorrect in certain cases - * [HIVE-10958] - Centos: TestMiniTezCliDriver.testCliDriver_mergejoin fails - * [HIVE-10959] - webhcat launcher job should reconnect to the running child job on task retry - * [HIVE-10965] - direct SQL for stats fails in 0-column case - * [HIVE-10966] - direct SQL for stats has a cast exception on some databases - * [HIVE-10967] - add mapreduce.job.tags to sql std authorization config whitelist - * [HIVE-10968] - Windows: analyze json table via beeline failed throwing Class org.apache.hive.hcatalog.data.JsonSerDe not found - * [HIVE-10971] - count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true - * [HIVE-10992] - WebHCat should not create delegation tokens when Kerberos is not enabled - * [HIVE-10994] - Hive.moveFile should not fail on a no-op move - * [HIVE-11001] - HS2 http cookie mode does not honor doAs url parameter - * [HIVE-11006] - improve logging wrt ACID module - * [HIVE-11010] - Accumulo storage handler queries via HS2 fail - * [HIVE-11023] - Disable directSQL if datanucleus.identifierFactory = datanucleus2 - * [HIVE-11027] - Hive on tez: Bucket map joins fail when hashcode goes negative - * [HIVE-11031] - ORC concatenation of old files can fail while merging column statistics - * [HIVE-11033] - BloomFilter index is not honored by ORC reader - * [HIVE-11035] - PPD: Orc Split elimination fails because filterColumns=[-1] - * [HIVE-11040] - Change Derby dependency version to 10.10.2.0 - * [HIVE-11041] - Update tests for HIVE-9302 after removing binaries - * [HIVE-11047] - Update versions of branch-1.2 to 1.2.1 - - -** Improvement - * [HIVE-8769] - Physical optimizer : Incorrect CE results in a shuffle join instead of a Map join (PK/FK pattern not detected) - * [HIVE-10812] - Scaling PK/FK's selectivity for stats annotation - * [HIVE-10900] - Fix the indeterministic stats for some hive queries - - -** New Feature - * [HIVE-10636] - CASE comparison operator rotation optimization - * [HIVE-10716] - Fold case/when udf for expression involving nulls in filter operator. - - -** Task - * [HIVE-10327] - Remove ExprNodeNullDesc - * [HIVE-11018] - Turn on cbo in more q files - - -** Test - * [HIVE-10869] - fold_case.q failing on trunk - * [HIVE-10909] - Make TestFilterHooks robust - * [HIVE-10939] - Make TestFileDump robust - * [HIVE-10955] - CliDriver leaves tables behind at end of test run - * [HIVE-11026] - Make vector_outer_join* test more robust - - - -Release Notes - Hive - Version 1.2.0 - -** Sub-task - * [HIVE-8119] - Implement Date in ParquetSerde - * [HIVE-8164] - Adding in a ReplicationTask that converts a Notification Event to actionable tasks - * [HIVE-8165] - Annotation changes for replication - * [HIVE-8379] - NanoTimeUtils performs some work needlessly - * [HIVE-8696] - HCatClientHMSImpl doesn't use a Retrying-HiveMetastoreClient. - * [HIVE-8817] - Create unit test where we insert into an encrypted table and then read from it with pig - * [HIVE-8818] - Create unit test where we insert into an encrypted table and then read from it with hcatalog mapreduce - * [HIVE-9009] - order by (limit) meaning for the last subquery of union in Hive is different from other main stream RDBMS - * [HIVE-9253] - MetaStore server should support timeout for long running requests - * [HIVE-9271] - Add ability for client to request metastore to fire an event - * [HIVE-9273] - Add option to fire metastore event on insert - * [HIVE-9327] - CBO (Calcite Return Path): Removing Row Resolvers from ParseContext - * [HIVE-9333] - Move parquet serialize implementation to DataWritableWriter to improve write speeds - * [HIVE-9432] - CBO (Calcite Return Path): Removing QB from ParseContext - * [HIVE-9501] - DbNotificationListener doesn't include dbname in create database notification and does not include tablename in create table notification - * [HIVE-9508] - MetaStore client socket connection should have a lifetime - * [HIVE-9516] - Enable CBO related tests [Spark Branch] - * [HIVE-9550] - ObjectStore.getNextNotification() can return events inside NotificationEventResponse as null which conflicts with its thrift "required" tag - * [HIVE-9558] - [Parquet] support HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable in vectorized mode - * [HIVE-9561] - SHUFFLE_SORT should only be used for order by query [Spark Branch] - * [HIVE-9563] - CBO(Calcite Return Path): Translate GB to Hive OP [CBO branch] - * [HIVE-9569] - Enable more unit tests for UNION ALL [Spark Branch] - * [HIVE-9571] - CBO (Calcite Return Path): Generate FileSink Op [CBO branch] - * [HIVE-9582] - HCatalog should use IMetaStoreClient interface - * [HIVE-9585] - AlterPartitionMessage should return getKeyValues instead of getValues - * [HIVE-9601] - New Beeline queries will hang If Beeline terminates in-properly [Spark Branch] - * [HIVE-9657] - Use new parquet Types API builder to construct data types - * [HIVE-9659] - 'Error while trying to create table container' occurs during hive query case execution when hive.optimize.skewjoin set to 'true' [Spark Branch] - * [HIVE-9666] - Improve some qtests - * [HIVE-9671] - Support Impersonation [Spark Branch] - * [HIVE-9690] - Refactoring for non-numeric arithmetic operations - * [HIVE-9696] - Address RB comments for HIVE-9425 [Spark Branch] - * [HIVE-9698] - Merge trunk to Spark branch 2/15/2015 [Spark Branch] - * [HIVE-9726] - Upgrade to spark 1.3 [Spark Branch] - * [HIVE-9750] - avoid log locks in operators - * [HIVE-9781] - Utilize spark.kryo.classesToRegister [Spark Branch] - * [HIVE-9792] - Support interval type in expressions/predicates - * [HIVE-9810] - prep object registry for multi threading - * [HIVE-9812] - Merge trunk to Spark branch 02/27/2015 [Spark Branch] - * [HIVE-9819] - Add timeout check inside the HMS server - * [HIVE-9823] - Load spark-defaults.conf from classpath [Spark Branch] - * [HIVE-9824] - LLAP: Native Vectorization of Map Join - * [HIVE-9871] - Print spark job id in history file [spark branch] - * [HIVE-9882] - Add jar/file doesn't work with yarn-cluster mode [Spark Branch] - * [HIVE-9889] - Merge trunk to Spark branch 3/6/2015 [Spark Branch] - * [HIVE-9894] - Use new parquet Types API builder to construct DATE data type - * [HIVE-9906] - Add timeout mechanism in RawStoreProxy - * [HIVE-9937] - LLAP: Vectorized Field-By-Field Serialize / Deserialize to support new Vectorized Map Join - * [HIVE-9969] - Avoid Utilities.getMapRedWork for spark [Spark Branch] - * [HIVE-9982] - CBO (Calcite Return Path): Prune TS Relnode schema - * [HIVE-9998] - Vectorization support for interval types - * [HIVE-10006] - RSC has memory leak while execute multi queries.[Spark Branch] - * [HIVE-10037] - JDBC support for interval expressions - * [HIVE-10044] - Allow interval params for year/month/day/hour/minute/second functions - * [HIVE-10053] - Override new init API fom ReadSupport instead of the deprecated one - * [HIVE-10071] - CBO (Calcite Return Path): Join to MultiJoin rule - * [HIVE-10076] - Bump up parquet-hadoop-bundle and parquet-column to the version of 1.6.0rc6 - * [HIVE-10130] - Merge trunk to Spark branch 03/27/2015 [Spark Branch] - * [HIVE-10131] - LLAP: BytesBytesMultiHashMap and mapjoin container should reuse refs - * [HIVE-10134] - Fix test failures after HIVE-10130 [Spark Branch] - * [HIVE-10227] - Concrete implementation of Export/Import based ReplicationTaskFactory - * [HIVE-10228] - Changes to Hive Export/Import/DropTable/DropPartition to support replication semantics - * [HIVE-10243] - CBO (Calcite Return Path): Introduce JoinAlgorithm Interface - * [HIVE-10252] - Make PPD work for Parquet in row group level - * [HIVE-10262] - CBO (Calcite Return Path): Temporarily disable Aggregate check input for bucketing - * [HIVE-10263] - CBO (Calcite Return Path): Aggregate checking input for bucketing should be conditional - * [HIVE-10291] - Hive on Spark job configuration needs to be logged [Spark Branch] - * [HIVE-10326] - CBO (Calcite Return Path): Invoke Hive's Cumulative Cost - * [HIVE-10329] - Hadoop reflectionutils has issues - * [HIVE-10343] - CBO (Calcite Return Path): Parameterize algorithm cost model - * [HIVE-10347] - Merge spark to trunk 4/15/2015 - * [HIVE-10350] - CBO: Use total size instead of bucket count to determine number of splits & parallelism - * [HIVE-10369] - CBO: Don't use HiveDefaultCostModel when With Tez and hive.cbo.costmodel.extended enabled - * [HIVE-10375] - CBO (Calcite Return Path): disable the identity project remover for some union operators - * [HIVE-10386] - CBO (Calcite Return Path): Disable Trivial Project Removal on ret path - * [HIVE-10391] - CBO (Calcite Return Path): HiveOpConverter always assumes that HiveFilter does not include a partition column - * [HIVE-10400] - CBO (Calcite Return Path): Exception when column name contains dot or colon characters - * [HIVE-10413] - [CBO] Return path assumes distinct column cant be same as grouping column - * [HIVE-10416] - CBO (Calcite Return Path): Fix return columns if Sort operator is on top of plan returned by Calcite - * [HIVE-10426] - Rework/simplify ReplicationTaskFactory instantiation - * [HIVE-10455] - CBO (Calcite Return Path): Different data types at Reducer before JoinOp - * [HIVE-10462] - CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered - * [HIVE-10493] - Merge multiple joins when join keys are the same - * [HIVE-10506] - CBO (Calcite Return Path): Disallow return path to be enable if CBO is off - * [HIVE-10512] - CBO (Calcite Return Path): SMBJoin conversion throws ClassCastException - * [HIVE-10520] - LLAP: Must reset small table result columns for Native Vectorization of Map Join - * [HIVE-10522] - CBO (Calcite Return Path): fix the wrong needed column names when TS is created - * [HIVE-10526] - CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account - * [HIVE-10547] - CBO (Calcite Return Path) : genFileSinkPlan uses wrong partition col to create FS - * [HIVE-10549] - CBO (Calcite Return Path): Enable NonBlockingOpDeDupProc - * [HIVE-10804] - CBO: Calcite Operator To Hive Operator (Calcite Return Path): optimizer for limit 0 does not work - -** Bug - * [HIVE-3454] - Problem with CAST(BIGINT as TIMESTAMP) - * [HIVE-4625] - HS2 should not attempt to get delegation token from metastore if using embedded metastore - * [HIVE-5545] - HCatRecord getInteger method returns String when used on Partition columns of type INT - * [HIVE-5672] - Insert with custom separator not supported for non-local directory - * [HIVE-5733] - Publish hive-exec artifact without all the dependencies - * [HIVE-6069] - Improve error message in GenericUDFRound - * [HIVE-6099] - Multi insert does not work properly with distinct count - * [HIVE-6950] - Parsing Error in GROUPING SETS - * [HIVE-7351] - ANALYZE TABLE statement fails on postgres metastore - * [HIVE-7641] - INSERT ... SELECT with no source table leads to NPE - * [HIVE-8524] - When table is renamed stats are lost as changes are not propagated to metastore tables TAB_COL_STATS and PART_COL_STATS - * [HIVE-8626] - Extend HDFS super-user checks to dropPartitions - * [HIVE-8746] - ORC timestamp columns are sensitive to daylight savings time - * [HIVE-8890] - HiveServer2 dynamic service discovery: use persistent ephemeral nodes curator recipe - * [HIVE-8915] - Log file explosion due to non-existence of COMPACTION_QUEUE table - * [HIVE-9002] - union all does not generate correct result for order by and limit - * [HIVE-9023] - HiveHistoryImpl relies on removed counters to print num rows - * [HIVE-9073] - NPE when using custom windowing UDAFs - * [HIVE-9083] - New metastore API to support to purge partition-data directly in dropPartitions(). - * [HIVE-9086] - Add language support to PURGE data while dropping partitions. - * [HIVE-9115] - Hive build failure on hadoop-2.7 due to HADOOP-11356 - * [HIVE-9118] - Support auto-purge for tables, when dropping tables/partitions. - * [HIVE-9151] - Checking s against null in TezJobMonitor#getNameWithProgress() should be done earlier - * [HIVE-9228] - Problem with subquery using windowing functions - * [HIVE-9303] - Parquet files are written with incorrect definition levels - * [HIVE-9322] - Make null-checks consistent for MapObjectInspector subclasses. - * [HIVE-9350] - Add ability for HiveAuthorizer implementations to filter out results of 'show tables', 'show databases' - * [HIVE-9397] - SELECT max(bar) FROM foo is broken after ANALYZE ... FOR COLUMNS - * [HIVE-9430] - NullPointerException on ALTER TABLE ADD PARTITION if no value given - * [HIVE-9438] - The standalone-jdbc jar missing some jars - * [HIVE-9456] - Make Hive support unicode with MSSQL as Metastore backend - * [HIVE-9468] - Test groupby3_map_skew.q fails due to decimal precision difference - * [HIVE-9471] - Bad seek in uncompressed ORC, at row-group boundary. - * [HIVE-9472] - Implement 7 simple UDFs added to Hive - * [HIVE-9474] - truncate table changes permissions on the target - * [HIVE-9481] - allow column list specification in INSERT statement - * [HIVE-9482] - Hive parquet timestamp compatibility - * [HIVE-9484] - ThriftCLIService#getDelegationToken does case sensitive comparison - * [HIVE-9486] - Use session classloader instead of application loader - * [HIVE-9489] - add javadoc for UDFType annotation - * [HIVE-9496] - Slf4j warning in hive command - * [HIVE-9507] - Make "LATERAL VIEW inline(expression) mytable" tolerant to nulls - * [HIVE-9509] - Restore partition spec validation removed by HIVE-9445 - * [HIVE-9512] - HIVE-9327 causing regression in stats annotation - * [HIVE-9513] - NULL POINTER EXCEPTION - * [HIVE-9526] - ClassCastException thrown by HiveStatement - * [HIVE-9529] - "alter table .. concatenate" under Tez mode should create TezTask - * [HIVE-9539] - Wrong check of version format in TestWebHCatE2e.getHiveVersion() - * [HIVE-9553] - Fix log-line in Partition Pruner - * [HIVE-9555] - assorted ORC refactorings for LLAP on trunk - * [HIVE-9560] - When hive.stats.collect.rawdatasize=true, 'rawDataSize' for an ORC table will result in value '0' after running 'analyze table TABLE_NAME compute statistics;' - * [HIVE-9565] - Minor cleanup in TestMetastoreExpr. - * [HIVE-9567] - JSON SerDe not escaping special chars when writing char/varchar data - * [HIVE-9580] - Server returns incorrect result from JOIN ON VARCHAR columns - * [HIVE-9587] - UDF decode should accept STRING_GROUP types for the second parameter - * [HIVE-9588] - Reimplement HCatClientHMSImpl.dropPartitions() with HMSC.dropPartitions() - * [HIVE-9592] - fix ArrayIndexOutOfBoundsException in date_add and date_sub initialize - * [HIVE-9609] - AddPartitionMessage.getPartitions() can return null - * [HIVE-9612] - Turn off DEBUG logging for Lazy Objects for tests - * [HIVE-9613] - Left join query plan outputs wrong column when using subquery - * [HIVE-9617] - UDF from_utc_timestamp throws NPE if the second argument is null - * [HIVE-9619] - Uninitialized read of numBitVectors in NumDistinctValueEstimator - * [HIVE-9620] - Cannot retrieve column statistics using HMS API if column name contains uppercase characters - * [HIVE-9622] - Getting NPE when trying to restart HS2 when metastore is configured to use org.apache.hadoop.hive.thrift.DBTokenStore - * [HIVE-9623] - NullPointerException in MapJoinOperator.processOp(MapJoinOperator.java:253) for TPC-DS Q75 against un-partitioned schema - * [HIVE-9624] - NullPointerException in MapJoinOperator.processOp(MapJoinOperator.java:253) for TPC-DS Q75 against un-partitioned schema - * [HIVE-9628] - HiveMetaStoreClient.dropPartitions(...List<ObjectPair<Integer,byte[]>>...) doesn't take (boolean needResult) - * [HIVE-9633] - Add HCatClient.dropPartitions() overload to skip deletion of partition-directories. - * [HIVE-9644] - Fold case & when udfs - * [HIVE-9645] - Constant folding case NULL equality - * [HIVE-9647] - Discrepancy in cardinality estimates between partitioned and un-partitioned tables - * [HIVE-9648] - Null check key provider before doing set - * [HIVE-9652] - Tez in place updates should detect redirection of STDERR - * [HIVE-9655] - Dynamic partition table insertion error - * [HIVE-9665] - Parallel move task optimization causes race condition - * [HIVE-9667] - Disable ORC bloom filters for ORC v11 output-format - * [HIVE-9674] - *DropPartitionEvent should handle partition-sets. - * [HIVE-9679] - Remove redundant null-checks from DbNotificationListener. - * [HIVE-9680] - GlobalLimitOptimizer is not checking filters correctly - * [HIVE-9681] - Extend HiveAuthorizationProvider to support partition-sets. - * [HIVE-9706] - HBase handler support for snapshots should confirm properties before use - * [HIVE-9711] - ORC Vectorization DoubleColumnVector.isRepeating=false if all entries are NaN - * [HIVE-9716] - Map job fails when table's LOCATION does not have scheme - * [HIVE-9717] - The max/min function used by AggrStats for decimal type is not what we expected - * [HIVE-9718] - Insert into dynamic partitions with same column structure in the "distibute by" clause barfs - * [HIVE-9720] - Metastore does not properly migrate column stats when renaming a table across databases. - * [HIVE-9721] - Hadoop23Shims.setFullFileStatus should check for null - * [HIVE-9727] - GroupingID translation from Calcite - * [HIVE-9731] - WebHCat MapReduce Streaming Job does not allow StreamXmlRecordReader to be specified - * [HIVE-9734] - Correlating expression cannot contain unqualified column references - * [HIVE-9735] - aggregate ( smalllint ) fails when ORC file used ava.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Short - * [HIVE-9743] - Incorrect result set for vectorized left outer join - * [HIVE-9749] - ObjectStore schema verification logic is incorrect - * [HIVE-9754] - rename GenericUDFLevenstein to GenericUDFLevenshtein - * [HIVE-9755] - Hive built-in "ngram" UDAF fails when a mapper has no matches. - * [HIVE-9767] - Fixes in Hive UDF to be usable in Pig - * [HIVE-9770] - Beeline ignores --showHeader for non-tablular output formats i.e csv,tsv,dsv - * [HIVE-9772] - Hive parquet timestamp conversion doesn't work with new Parquet - * [HIVE-9779] - ATSHook does not log the end user if doAs=false (it logs the hs2 server user) - * [HIVE-9791] - insert into table throws NPE - * [HIVE-9797] - Need update some spark tests for java 8 - * [HIVE-9803] - SparkClientImpl should not attempt impersonation in CLI mode [Spark Branch] - * [HIVE-9813] - Hive JDBC - DatabaseMetaData.getColumns method cannot find classes added with "add jar" command - * [HIVE-9817] - fix DateFormat pattern in hive-exec - * [HIVE-9826] - Firing insert event fails on temporary table - * [HIVE-9830] - Map join could dump a small table multiple times [Spark Branch] - * [HIVE-9831] - HiveServer2 should use ConcurrentHashMap in ThreadFactory - * [HIVE-9832] - Merge join followed by union and a map join in hive on tez fails. - * [HIVE-9834] - VectorGroupByOperator logs too much - * [HIVE-9836] - Hive on tez: fails when virtual columns are present in the join conditions (for e.g. partition columns) - * [HIVE-9839] - HiveServer2 leaks OperationHandle on async queries which fail at compile phase - * [HIVE-9841] - IOException thrown by ORC should include the path of processing file - * [HIVE-9845] - HCatSplit repeats information making input split data size huge - * [HIVE-9847] - Hive should not allow additional attemps when RSC fails [Spark Branch] - * [HIVE-9848] - readlink -f is GNU coreutils only (used in bin/hive) - * [HIVE-9851] - org.apache.hadoop.hive.serde2.avro.AvroSerializer should use org.apache.avro.generic.GenericData.Array when serializing a list - * [HIVE-9855] - Runtime skew join doesn't work when skewed data only exists in big table - * [HIVE-9860] - MapredLocalTask/SecureCmdDoAs leaks local files - * [HIVE-9866] - Changing a column's type doesn't change column stats type in metastore - * [HIVE-9869] - Trunk doesn't build with hadoop-1 - * [HIVE-9872] - Making yarn-cluster default for Hive on Spark [Spark Branch] - * [HIVE-9873] - Hive on MR throws DeprecatedParquetHiveInput exception - * [HIVE-9877] - Beeline cannot run multiple statements in the same row - * [HIVE-9886] - Hive on tez: NPE when converting join to SMB in sub-query - * [HIVE-9892] - various MSSQL upgrade scripts don't work - * [HIVE-9902] - Map join small table files need more replications [Spark Branch] - * [HIVE-9908] - vectorization error binary type not supported, group by with binary columns - * [HIVE-9915] - Allow specifying file format for managed tables - * [HIVE-9918] - Spark branch build is failing due to unknown url [Spark Branch] - * [HIVE-9919] - upgrade scripts don't work on some auto-created DBs due to absence of tables - * [HIVE-9920] - DROP DATABASE IF EXISTS throws exception if database does not exist - * [HIVE-9923] - No clear message when "from" is missing - * [HIVE-9929] - StatsUtil#getAvailableMemory could return negative value - * [HIVE-9930] - fix QueryPlan.makeQueryId time format - * [HIVE-9932] - DDLTask.conf hides base class Task.conf - * [HIVE-9934] - Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to "none", allowing authentication without password - * [HIVE-9935] - Fix tests for java 1.8 [Spark Branch] - * [HIVE-9936] - fix potential NPE in DefaultUDAFEvaluatorResolver - * [HIVE-9939] - Code cleanup for redundant if check in ExplainTask [Spark Branch] - * [HIVE-9944] - Convert array[] to string properly in log messages - * [HIVE-9945] - FunctionTask.conf hides Task.conf field - * [HIVE-9947] - ScriptOperator replaceAll uses unescaped dot and result is not assigned - * [HIVE-9948] - SparkUtilities.getFileName passes File.separator to String.split() method - * [HIVE-9950] - fix rehash in CuckooSetBytes and CuckooSetLong - * [HIVE-9951] - VectorizedRCFileRecordReader creates Exception but does not throw it - * [HIVE-9952] - fix NPE in CorrelationUtilities - * [HIVE-9953] - fix NPE in WindowingTableFunction - * [HIVE-9954] - UDFJson uses the == operator to compare Strings - * [HIVE-9955] - TestVectorizedRowBatchCtx compares byte[] using equals() method - * [HIVE-9956] - use BigDecimal.valueOf instead of new in TestFileDump - * [HIVE-9957] - Hive 1.1.0 not compatible with Hadoop 2.4.0 - * [HIVE-9961] - HookContext for view should return a table type of VIRTUAL_VIEW - * [HIVE-9971] - Clean up operator class - * [HIVE-9975] - Renaming a nonexisting partition should not throw out NullPointerException - * [HIVE-9976] - Possible race condition in DynamicPartitionPruner for <200ms tasks - * [HIVE-9977] - Compactor not running on partitions after dynamic partitioned insert - * [HIVE-9984] - JoinReorder's getOutputSize is exponential - * [HIVE-9991] - Cannot do a SELECT on external tables that are on S3 due to Encryption error - * [HIVE-9993] - Retrying task could use cached bad operators [Spark Branch] - * [HIVE-9994] - Hive query plan returns sensitive data to external applications - * [HIVE-9997] - minor tweaks for bytes mapjoin hash table - * [HIVE-10001] - SMB join in reduce side - * [HIVE-10009] - LazyObjectInspectorFactory is not thread safe [Spark Branch] - * [HIVE-10017] - SparkTask log improvement [Spark Branch] - * [HIVE-10023] - Fix more cache related concurrency issue [Spark Branch] - * [HIVE-10032] - Remove HCatalog broken java file from source code - * [HIVE-10042] - clean up TreeReaders - ORC refactoring for LLAP on trunk - * [HIVE-10050] - Support overriding memory configuration for AM launched for TempletonControllerJob - * [HIVE-10059] - Make udaf_percentile_approx_23.q test more stable - * [HIVE-10062] - HiveOnTez: Union followed by Multi-GB followed by Multi-insert loses data - * [HIVE-10066] - Hive on Tez job submission through WebHCat doesn't ship Tez artifacts - * [HIVE-10073] - Runtime exception when querying HBase with Spark [Spark Branch] - * [HIVE-10074] - Ability to run HCat Client Unit tests in a system test setting - * [HIVE-10078] - Optionally allow logging of records processed in fixed intervals - * [HIVE-10083] - SMBJoin fails in case one table is uninitialized - * [HIVE-10085] - Lateral view on top of a view throws RuntimeException - * [HIVE-10086] - Hive throws error when accessing Parquet file schema using field name match - * [HIVE-10087] - Beeline's --silent option should suppress query from being echoed when running with -f option - * [HIVE-10093] - Unnecessary HMSHandler initialization for default MemoryTokenStore on HS2 - * [HIVE-10095] - format_number udf throws NPE - * [HIVE-10098] - HS2 local task for map join fails in KMS encrypted cluster - * [HIVE-10106] - Regression : Dynamic partition pruning not working after HIVE-9976 - * [HIVE-10108] - Index#getIndexTableName() returns db.index_table_name - * [HIVE-10122] - Hive metastore filter-by-expression is broken for non-partition expressions - * [HIVE-10123] - Hybrid grace Hash join : Use estimate key count from stats to initialize BytesBytesMultiHashMap - * [HIVE-10128] - BytesBytesMultiHashMap does not allow concurrent read-only access - * [HIVE-10136] - BaseWork.vectorMode hides AbstractOperatorDesc.vectorMode - * [HIVE-10143] - HS2 fails to clean up Spark client state on timeout [Spark Branch] - * [HIVE-10145] - set Tez ACLs appropriately in hive - * [HIVE-10148] - update of bucketing column should not be allowed - * [HIVE-10150] - delete from acidTbl where a in(select a from nonAcidOrcTbl) fails - * [HIVE-10151] - insert into A select from B is broken when both A and B are Acid tables and bucketed the same way - * [HIVE-10152] - ErrorMsg.formatToErrorMsgMap has bad regex - * [HIVE-10167] - HS2 logs the server started only before the server is shut down - * [HIVE-10172] - Fix performance regression caused by HIVE-8122 for ORC - * [HIVE-10178] - DateWritable incorrectly calculates daysSinceEpoch for negative Unix time - * [HIVE-10186] - Hive does not log Tez diagnostics on errors - * [HIVE-10197] - Add lockId to all ACID log statements - * [HIVE-10202] - Beeline outputs prompt+query on standard output when used in non-interactive mode - * [HIVE-10208] - templeton.hive.extra.files should be commented out in webhcat-default.xml - * [HIVE-10209] - FetchTask with VC may fail because ExecMapper.done is true - * [HIVE-10213] - MapReduce jobs using dynamic-partitioning fail on commit. - * [HIVE-10225] - CLI JLine does not flush history on quit/Ctrl-C - * [HIVE-10226] - Column stats for Date columns not supported - * [HIVE-10229] - Set conf and processor context in the constructor instead of init - * [HIVE-10231] - Compute partition column stats fails if partition col type is date - * [HIVE-10242] - ACID: insert overwrite prevents create table command - * [HIVE-10265] - Hive CLI crashes on != inequality - * [HIVE-10267] - HIVE-9664 makes hive depend on ivysettings.xml : trivial breakage fix - * [HIVE-10269] - HiveMetaStore.java:[6089,29] cannot find symbol class JvmPauseMonitor - * [HIVE-10270] - Cannot use Decimal constants less than 0.1BD - * [HIVE-10271] - remove hive.server2.thrift.http.min/max.worker.threads properties - * [HIVE-10272] - Some HCat tests fail under windows - * [HIVE-10273] - Union with partition tables which have no data fails with NPE - * [HIVE-10274] - Send context and description to tez via dag info - * [HIVE-10275] - GenericUDF getTimestampValue should return Timestamp instead of Date - * [HIVE-10284] - enable container reuse for grace hash join - * [HIVE-10285] - Incorrect endFunction call in HiveMetaStore - * [HIVE-10286] - SARGs: Type Safety via PredicateLeaf.type - * [HIVE-10288] - Cannot call permanent UDFs - * [HIVE-10290] - Add negative test case to modify a non-existent config value when hive security authorization is enabled. - * [HIVE-10303] - HIVE-9471 broke forward compatibility of ORC files - * [HIVE-10305] - TestOrcFile has a mistake that makes metadata test ineffective - * [HIVE-10306] - We need to print tez summary when hive.server2.logging.level >= PERFORMANCE. - * [HIVE-10309] - TestJdbcWithMiniHS2.java broken because of the removal of hive.server2.thrift.http.max.worker.threads - * [HIVE-10312] - SASL.QOP in JDBC URL is ignored for Delegation token Authentication - * [HIVE-10313] - Literal Decimal ExprNodeConstantDesc should contain value of HiveDecimal instead of String - * [HIVE-10318] - The HMS upgrade test does not test patches that affect the upgrade test scripts - * [HIVE-10323] - Tez merge join operator does not honor hive.join.emit.interval - * [HIVE-10324] - Hive metatool should take table_param_key to allow for changes to avro serde's schema url key - * [HIVE-10331] - ORC : Is null SARG filters out all row groups written in old ORC format - * [HIVE-10340] - Enable ORC test for timezone reading from old format - * [HIVE-10346] - Tez on HBase has problems with settings again - * [HIVE-10364] - The HMS upgrade script test does not publish results when prepare.sh fails. - * [HIVE-10367] - org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.* fails in Windows - * [HIVE-10370] - Hive does not compile with -Phadoop-1 option - * [HIVE-10372] - Update parquet version to 1.6.0 - * [HIVE-10374] - Revert HIVE-10304 deprecation message to HiveCLI - * [HIVE-10384] - RetryingMetaStoreClient does not retry wrapped TTransportExceptions - * [HIVE-10407] - separate out the timestamp ranges for testing purposes - * [HIVE-10409] - Webhcat tests need to be updated, to accomodate HADOOP-10193 - * [HIVE-10421] - DROP TABLE with qualified table name ignores database name when checking partitions - * [HIVE-10428] - NPE in RegexSerDe using HCat - * [HIVE-10431] - HIVE-9555 broke hadoop-1 build - * [HIVE-10437] - NullPointerException on queries where map/reduce is not involved on tables with partitions - * [HIVE-10441] - Fix confusing log statement in SessionState about hive.execution.engine setting - * [HIVE-10442] - HIVE-10098 broke hadoop-1 build - * [HIVE-10443] - HIVE-9870 broke hadoop-1 build - * [HIVE-10444] - HIVE-10223 breaks hadoop-1 build - * [HIVE-10446] - Hybrid Hybrid Grace Hash Join : java.lang.IllegalArgumentException in Kryo while spilling big table - * [HIVE-10447] - Beeline JDBC Driver to support 2 way SSL - * [HIVE-10450] - More than one TableScan in MapWork not supported in Vectorization -- causes query to fail during vectorization - * [HIVE-10451] - PTF deserializer fails if values are not used in reducer - * [HIVE-10456] - Grace Hash Join should not load spilled partitions on abort - * [HIVE-10465] - whitelist restrictions don't get initialized in new copy of HiveConf - * [HIVE-10472] - Jenkins HMS upgrade test is not publishing results because JIRAService class is not found. - * [HIVE-10477] - Provide option to disable Spark tests - * [HIVE-10481] - ACID table update finishes but values not really updated if column names are not all lower case - * [HIVE-10483] - insert overwrite partition deadlocks on itself with DbTxnManager - * [HIVE-10484] - Vectorization : RuntimeException "Big Table Retained Mapping duplicate column" - * [HIVE-10490] - HBase Snapshot IF fails at run time with missing dependency of MetricsRegistry - * [HIVE-10499] - Ensure Session/ZooKeeperClient instances are closed - * [HIVE-10500] - Repeated deadlocks in underlying RDBMS cause transaction or lock failure - * [HIVE-10507] - Expose RetryingMetastoreClient to other external users of metastore client like Flume and Storm. - * [HIVE-10508] - Strip out password information from config passed to Tez/MR in cases where password encryption is not used - * [HIVE-10510] - Change 1.2.0-SNAPSHOT to 1.2.0 in branch-1.2 - * [HIVE-10513] - [CBO] return path : Fix create_func1.q for return path - * [HIVE-10514] - Fix MiniCliDriver tests failure - * [HIVE-10517] - HCatPartition should not be created with "" as location in tests - * [HIVE-10521] - TxnHandler.timeOutTxns only times out some of the expired transactions - * [HIVE-10529] - Remove references to tez task context before storing operator plan in object cache - * [HIVE-10530] - Aggregate stats cache: bug fixes for RDBMS path - * [HIVE-10538] - Fix NPE in FileSinkOperator from hashcode mismatch - * [HIVE-10539] - set default value of hive.repl.task.factory - * [HIVE-10542] - Full outer joins in tez produce incorrect results in certain cases - * [HIVE-10543] - improve error message in MetaStoreAuthzAPIAuthorizerEmbedOnly - * [HIVE-10544] - Beeline/Hive JDBC Driver fails in HTTP mode on Windows with java.lang.NoSuchFieldError: INSTANCE - * [HIVE-10548] - Remove dependency to s3 repository in root pom - * [HIVE-10564] - webhcat should use webhcat-site.xml properties for controller job submission - * [HIVE-10572] - Improve Hive service test to check empty string - * [HIVE-10576] - add jar command does not work with Windows OS - * [HIVE-10578] - update sql standard authorization configuration whitelist - * [HIVE-10579] - Fix -Phadoop-1 build - * [HIVE-10595] - Dropping a table can cause NPEs in the compactor - * [HIVE-10603] - increase default permgen space for HS2 on windows - * [HIVE-10604] - update webhcat-default.xml with 1.2 version numbers - * [HIVE-10607] - Combination of ReducesinkDedup + TopN optimization yields incorrect result if there are multiple GBY in reducer - * [HIVE-10609] - Vectorization : Q64 fails with ClassCastException - * [HIVE-10610] - hive command fails to get hadoop version - * [HIVE-10612] - HIVE-10578 broke TestSQLStdHiveAccessControllerHS2 tests - * [HIVE-10614] - schemaTool upgrade from 0.14.0 to 1.3.0 causes failure - * [HIVE-10638] - HIVE-9736 introduces issues with Hadoop23Shims.checkFileAccess - * [HIVE-10685] - Alter table concatenate oparetor will cause duplicate data - * [HIVE-10715] - RAT failures - many files do not have ASF licenses - -** Improvement - * [HIVE-1575] - get_json_object does not support JSON array at the root level - * [HIVE-2573] - Create per-session function registry - * [HIVE-3185] - allow union set operation without ALL keyword (align to ISO-SQL 2011 specification) - * [HIVE-5472] - support a simple scalar which returns the current timestamp - * [HIVE-7175] - Provide password file option to beeline - * [HIVE-8385] - UNION Operator in Hive - * [HIVE-9138] - Add some explain to PTF operator - * [HIVE-9143] - select user(), current_user() - * [HIVE-9298] - Support reading alternate timestamp formats - * [HIVE-9470] - Use a generic writable object to run ColumnaStorageBench write/read tests - * [HIVE-9480] - Build UDF TRUNC to implement FIRST_DAY as compared with LAST_DAY - * [HIVE-9500] - Support nested structs over 24 levels. - * [HIVE-9518] - Implement MONTHS_BETWEEN aligned with Oracle one - * [HIVE-9521] - Drop support for Java6 - * [HIVE-9556] - create UDF to calculate the Levenshtein distance between two strings - * [HIVE-9564] - Extend HIVE-9298 for JsonSerDe - * [HIVE-9590] - add qtests for ADD_MONTHS UDF - * [HIVE-9594] - Add qtests for LAST_DAY udf - * [HIVE-9596] - move standard getDisplayString impl to GenericUDF - * [HIVE-9607] - Remove unnecessary attach-jdbc-driver execution from package/pom.xml - * [HIVE-9618] - Deduplicate RS keys for ptf/windowing - * [HIVE-9664] - Hive "add jar" command should be able to download and add jars from a repository - * [HIVE-9673] - Set operationhandle in ATS entities for lookups - * [HIVE-9699] - Extend PTFs to provide referenced columns for CP - * [HIVE-9709] - Hive should support replaying cookie from JDBC driver for beeline - * [HIVE-9710] - HiveServer2 should support cookie based authentication, when using HTTP transport. - * [HIVE-9738] - create SOUNDEX udf - * [HIVE-9744] - Move common arguments validation and value extraction code to GenericUDF - * [HIVE-9766] - Add JavaConstantXXXObjectInspector - * [HIVE-9788] - Make double quote optional in tsv/csv/dsv output - * [HIVE-9793] - Remove hard coded paths from cli driver tests - * [HIVE-9800] - Create scripts to do metastore upgrade tests on Jenkins - * [HIVE-9804] - Turn on some kryo settings by default for Spark [Spark Branch] - * [HIVE-9857] - Create Factorial UDF - * [HIVE-9858] - Create cbrt (cube root) UDF - * [HIVE-9859] - Create bitwise left/right shift UDFs - * [HIVE-9870] - Add JvmPauseMonitor threads to HMS and HS2 daemons - * [HIVE-9909] - Specify hive branch to use on jenkins hms tests - * [HIVE-9914] - Post success comments on Jira from Jenkins metastore upgrades scripts - * [HIVE-9917] - After HIVE-3454 is done, make int to timestamp conversion configurable - * [HIVE-9981] - Avoid throwing many exceptions when attempting to create new hdfs encryption shim - * [HIVE-10007] - Support qualified table name in analyze table compute statistics for columns - * [HIVE-10027] - Use descriptions from Avro schema files in column comments - * [HIVE-10072] - Add vectorization support for Hybrid Grace Hash Join - * [HIVE-10114] - Split strategies for ORC - * [HIVE-10119] - Allow Log verbosity to be set in hiveserver2 session - * [HIVE-10120] - Disallow create table with dot/colon in column name - * [HIVE-10146] - Not count session as idle if query is running - * [HIVE-10160] - Give a warning when grouping or ordering by a constant column - * [HIVE-10177] - Enable constant folding for char & varchar - * [HIVE-10206] - Improve Alter Table to not initialize Serde unnecessarily - * [HIVE-10214] - log metastore call timing information aggregated at query level - * [HIVE-10216] - log hive cli classpath at debug level - * [HIVE-10223] - Consolidate several redundant FileSystem API calls. - * [HIVE-10276] - Implement date_format(timestamp, fmt) UDF - * [HIVE-10307] - Support to use number literals in partition column - * [HIVE-10339] - Allow JDBC Driver to pass HTTP header Key/Value pairs - * [HIVE-10382] - Aggregate stats cache for RDBMS based metastore codepath - * [HIVE-10403] - Add n-way join support for Hybrid Grace Hash Join - * [HIVE-10467] - Switch to GIT repository on Jenkins precommit tests - * [HIVE-10546] - genFileSinkPlan should use the generated SEL's RR for the partition col of FS - * [HIVE-10568] - Select count(distinct()) can have more optimal execution plan - -** New Feature - * [HIVE-7998] - Enhance JDBC Driver to not require class specification - * [HIVE-9039] - Support Union Distinct - * [HIVE-9188] - BloomFilter support in ORC - * [HIVE-9277] - Hybrid Hybrid Grace Hash Join - * [HIVE-9302] - Beeline add commands to register local jdbc driver names and jars - * [HIVE-9780] - Add another level of explain for RDBMS audience - * [HIVE-10038] - Add Calcite's ProjectMergeRule. - * [HIVE-10099] - Enable constant folding for Decimal - * [HIVE-10591] - Support limited integer type promotion in ORC - -** Task - * [HIVE-6617] - Reduce ambiguity in grammar - * [HIVE-8898] - Remove HIVE-8874 once HBASE-12493 is fixed - * [HIVE-9015] - Constant Folding optimizer doesn't handle expressions involving null - * [HIVE-9411] - Improve error messages in TestMultiOutputFormat - * [HIVE-9416] - Get rid of Extract Operator - * [HIVE-9485] - Update trunk to 1.2.0-SNAPSHOT - * [HIVE-9498] - Update golden files of join38 & subquery_in on trunk due to 9327 - * [HIVE-9520] - Create NEXT_DAY UDF - * [HIVE-9572] - Merge from Spark branch to trunk 02/03/2015 - * [HIVE-9703] - Merge from Spark branch to trunk 02/16/2015 - * [HIVE-9861] - Add spark-assembly on Hive's classpath [Spark Branch] - * [HIVE-9867] - Migrate usage of deprecated Calcite methods - * [HIVE-9903] - Update calcite version - * [HIVE-9966] - Get rid of customBucketMapJoin field from MapJoinDesc - * [HIVE-10019] - Configure jenkins precommit jobs to run HMS upgrade tests - * [HIVE-10222] - Upgrade Calcite dependency to newest version - * [HIVE-10268] - Merge cbo branch into trunk - * [HIVE-10325] - Remove ExprNodeNullEvaluator - * [HIVE-10676] - Update Hive's README to mention spark, and to remove jdk1.6 - -** Test - * [HIVE-7948] - Add an E2E test to verify fix for HIVE-7155 - * [HIVE-8949] - Test for PARQUET-26 - * [HIVE-9399] - ppd_multi_insert.q generate same output in different order, when mapred.reduce.tasks is set to larger than 1 - * [HIVE-9525] - Enable constant propagation optimization in few existing tests where it was disabled. - * [HIVE-9924] - Fix union12 and union31 for spark [Spark Branch] - * [HIVE-10135] - Add qtest to access struct<> data type with parquet format after parquet column index access enabled - * [HIVE-10168] - make groupby3_map.q more stable - * [HIVE-10396] - decimal_precision2.q test is failing on trunk - - -Release Notes - Hive - Version 1.1.0 - -** Sub-task - * [HIVE-7073] - Implement Binary in ParquetSerDe - * [HIVE-7331] - Create SparkCompiler [Spark Branch] - * [HIVE-7334] - Create SparkShuffler, shuffling data between map-side data processing and reduce-side processing [Spark Branch] - * [HIVE-7335] - Create SparkPlan, DAG representation of a Spark job [Spark Branch] - * [HIVE-7336] - Create MapFunction [Spark Branch] - * [HIVE-7337] - Create ReduceFunction [Spark Branch] - * [HIVE-7338] - Create SparkPlanGenerator [Spark Branch] - * [HIVE-7382] - Create a MiniSparkCluster and set up a testing framework [Spark Branch] - * [HIVE-7384] - Research into reduce-side join [Spark Branch] - * [HIVE-7410] - Spark 1.0.1 is released, stop using SNAPSHOT [Spark Branch] - * [HIVE-7411] - Exclude hadoop 1 from spark dep [Spark Branch] - * [HIVE-7436] - Load Spark configuration into Hive driver [Spark Branch] - * [HIVE-7439] - Spark job monitoring and error reporting [Spark Branch] - * [HIVE-7465] - Implement pre-commit testing [Spark Branch] - * [HIVE-7492] - Enhance SparkCollector [Spark Branch] - * [HIVE-7503] - Support Hive's multi-table insert query with Spark [Spark Branch] - * [HIVE-7527] - Support order by and sort by on Spark [Spark Branch] - * [HIVE-7528] - Support cluster by and distributed by [Spark Branch] - * [HIVE-7541] - Support union all on Spark [Spark Branch] - * [HIVE-7560] - StarterProject: Fix exception handling in POC code [Spark Branch] - * [HIVE-7561] - StarterProject: Move from assert to Guava Preconditions.* in Hive on Spark [Spark Branch] - * [HIVE-7580] - Support dynamic partitioning [Spark Branch] - * [HIVE-7597] - Support analyze table [Spark Branch] - * [HIVE-7613] - Research optimization of auto convert join to map join [Spark branch] - * [HIVE-7665] - Create TestSparkCliDriver to run test in spark local mode [Spark Branch] - * [HIVE-7674] - Update to Spark 1.2 [Spark Branch] - * [HIVE-7677] - Implement native HiveReduceFunction [Spark Branch] - * [HIVE-7702] - Start running .q file tests on spark [Spark Branch] - * [HIVE-7708] - Fix qtest-spark pom.xml reference to test properties [Spark Branch] - * [HIVE-7709] - Create SparkReporter [Spark Branch] - * [HIVE-7717] - Add .q tests coverage for "union all" [Spark Branch] - * [HIVE-7728] - Enable q-tests for TABLESAMPLE feature [Spark Branch] - * [HIVE-7729] - Enable q-tests for ANALYZE TABLE feature [Spark Branch] - * [HIVE-7731] - Incorrect result returned when a map work has multiple downstream reduce works [Spark Branch] - * [HIVE-7746] - Cleanup SparkClient and make refreshLocalResources method synchronized [Spark Branch] - * [HIVE-7748] - Add qfile_regex to qtest-spark pom [Spark Branch] - * [HIVE-7749] - Enable timestamp.* tests [Spark Branch] - * [HIVE-7755] - Enable avro* tests [Spark Branch] - * [HIVE-7758] - PTest2 separates test files with spaces while QTestGen uses commas [Spark Branch] - * [HIVE-7766] - Cleanup Reduce operator code [Spark Branch] - * [HIVE-7767] - hive.optimize.union.remove does not work properly [Spark Branch] - * [HIVE-7773] - Union all query finished with errors [Spark Branch] - * [HIVE-7775] - enable sample8.q.[Spark Branch] - * [HIVE-7776] - enable sample10.q.[Spark Branch] - * [HIVE-7781] - Enable windowing and analytic function qtests [Spark Branch] - * [HIVE-7791] - Enable tests on Spark branch (1) [Sparch Branch] - * [HIVE-7792] - Enable tests on Spark branch (2) [Sparch Branch] - * [HIVE-7793] - Enable tests on Spark branch (3) [Sparch Branch] - * [HIVE-7794] - Enable tests on Spark branch (4) [Sparch Branch] - * [HIVE-7810] - Insert overwrite table query has strange behavior when set hive.optimize.union.remove=true [Spark Branch] - * [HIVE-7815] - Reduce Side Join with single reducer [Spark Branch] - * [HIVE-7816] - Enable map-join tests which Tez executes [Spark Branch] - * [HIVE-7821] - StarterProject: enable groupby4.q [Spark Branch] - * [HIVE-7839] - Update union_null results now that it's deterministic [Spark Branch] - * [HIVE-7842] - Enable qtest load_dyn_part1.q [Spark Branch] - * [HIVE-7843] - orc_analyze.q fails due to random mapred.task.id in FileSinkOperator [Spark Branch] - * [HIVE-7844] - optimize_nullscan.q fails due to differences in explain plan [Spark Branch] - * [HIVE-7848] - Refresh SparkContext when spark configuration changes [Spark Branch] - * [HIVE-7856] - Enable parallelism in Reduce Side Join [Spark Branch] - * [HIVE-7861] - Enable reduce-side join tests (1) [Spark Branch] - * [HIVE-7866] - Merge from trunk (1) [Spark Branch] - * [HIVE-7867] - Re-order spark.query.files in sorted order [Spark Branch] - * [HIVE-7870] - Insert overwrite table query does not generate correct task plan [Spark Branch] - * [HIVE-7873] - Re-enable lazy HiveBaseFunctionResultList [Spark Branch] - * [HIVE-7880] - Support subquery [Spark Branch] - * [HIVE-7881] - enable Qtest scriptfile1.q [Spark Branch] - * [HIVE-7922] - Merge from trunk (2) [Spark Branch] - * [HIVE-7952] - Investigate query failures (1) - * [HIVE-7954] - Investigate query failures (3) - * [HIVE-8015] - Merge from trunk (3) [Spark Branch] - * [HIVE-8017] - Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark Branch] - * [HIVE-8054] - Disable hive.optimize.union.remove when hive.execution.engine=spark [Spark Branch] - * [HIVE-8055] - Code cleanup after HIVE-8054 [Spark Branch] - * [HIVE-8074] - Merge trunk into spark 9/12/2014 - * [HIVE-8106] - Enable vectorization for spark [spark branch] - * [HIVE-8121] - Create micro-benchmarks for ParquetSerde and evaluate performance - * [HIVE-8122] - Make use of SearchArgument classes for Parquet SERDE - * [HIVE-8130] - Support Date in Avro - * [HIVE-8131] - Support timestamp in Avro - * [HIVE-8140] - Remove obsolete code from SparkWork [Spark Branch] - * [HIVE-8141] - Refactor the GraphTran code by moving union handling logic to UnionTran [Spark Branch] - * [HIVE-8202] - Support SMB Join for Hive on Spark [Spark Branch] - * [HIVE-8206] - Merge from trunk to spark 9/20/14 - * [HIVE-8275] - Introduce MapInput encapsulating a Hadoop RDD [Spark Branch] - * [HIVE-8276] - Separate shuffle from ReduceTran and so create ShuffleTran [Spark Branch] - * [HIVE-8278] - Restoring a graph representation of SparkPlan [Spark Branch] - * [HIVE-8302] - GroupByShuffler.java missing apache license header [Spark Branch] - * [HIVE-8303] - Merge from trunk to spark 9/29/14 - * [HIVE-8352] - Enable windowing.q for spark [Spark Branch] - * [HIVE-8353] - Merge trunk into spark 10/4/2015 [Spark Branch] - * [HIVE-8356] - Fix fs_default_name2.q on spark [Spark Branch] - * [HIVE-8362] - Investigate flaky test parallel.q - * [HIVE-8405] - Research Bucket Map Join [Spark Branch] - * [HIVE-8412] - Make reduce side join work for all join queries [Spark Branch] - * [HIVE-8422] - Turn on all join .q tests [Spark Branch] - * [HIVE-8436] - Modify SparkWork to split works with multiple child works [Spark Branch] - * [HIVE-8437] - Modify SparkPlan generation to set toCache flag to SparkTrans where caching is needed [Spark Branch] - * [HIVE-8438] - Clean up code introduced by HIVE-7503 and such [Spark Plan] - * [HIVE-8455] - Print Spark job progress format info on the console[Spark Branch] - * [HIVE-8456] - Support Hive Counter to collect spark job metric[Spark Branch] - * [HIVE-8477] - Update timestamp in status console [Spark Branch] - * [HIVE-8486] - TPC-DS Query 96 parallelism is not set correcly - * [HIVE-8502] - Merge trunk into spark 10/17/14 [Spark Branch] - * [HIVE-8507] - UT: fix rcfile_bigdata test [Spark Branch] - * [HIVE-8508] - UT: fix bucketsort_insert tests - related to SMBMapJoinOperator - * [HIVE-8509] - UT: fix list_bucket_dml_2 test [Spark Branch] - * [HIVE-8528] - Add remote Spark client to Hive [Spark Branch] - * [HIVE-8539] - Enable collect table statistics based on SparkCounter[Spark Branch] - * [HIVE-8540] - HivePairFlatMapFunction.java missing license header [Spark Branch] - * [HIVE-8599] - Add InterfaceAudience annotations to spark-client [Spark Branch] - * [HIVE-8616] - convert joinOp to MapJoinOp and generate MapWorks only [Spark Branch] - * [HIVE-8621] - Dump small table join data for map-join [Spark Branch] - * [HIVE-8622] - Split map-join plan into 2 SparkTasks in 3 stages [Spark Branch] - * [HIVE-8623] - Implement HashTableLoader for Spark map-join [Spark Branch] - * [HIVE-8638] - Implement bucket map join optimization [Spark Branch] - * [HIVE-8639] - Convert SMBJoin to MapJoin [Spark Branch] - * [HIVE-8640] - Support hints of SMBJoin [Spark Branch] - * [HIVE-8649] - Increase level of parallelism in reduce phase [Spark Branch] - * [HIVE-8651] - CBO: sort column changed in infer_bucket_sort test - * [HIVE-8670] - Combine Hive Operator statistic and Spark Metric to an uniformed query statistic.[Spark Branch] - * [HIVE-8674] - Fix tests after merge [Spark Branch] - * [HIVE-8682] - Enable table statistic collection on counter for CTAS query[Spark Branch] - * [HIVE-8700] - Replace ReduceSink to HashTableSink (or equi.) for small tables [Spark Branch] - * [HIVE-8707] - Fix ordering differences due to Java 8 HashMap function - * [HIVE-8718] - Refactoring: move mapLocalWork field from MapWork to BaseWork - * [HIVE-8725] - spark-client build failed sometimes.[Spark Branch] - * [HIVE-8726] - Collect Spark TaskMetrics and build job statistic[Spark Branch] - * [HIVE-8756] - numRows and rawDataSize are not collected by the Spark stats [Spark Branch] - * [HIVE-8758] - Fix hadoop-1 build [Spark Branch] - * [HIVE-8773] - Fix TestWebHCatE2e#getStatus for Java8 - * [HIVE-8775] - Merge from trunk 11/6/14 [SPARK BRANCH] - * [HIVE-8776] - Generate MapredLocalWork in SparkMapJoinResolver [Spark Brach] - * [HIVE-8777] - Should only register used counters in SparkCounters[Spark Branch] - * [HIVE-8780] - insert1.q and ppd_join4.q hangs with hadoop-1 [Spark Branch] - * [HIVE-8783] - Create some tests that use Spark counter for stats collection [Spark Branch] - * [HIVE-8787] - UT: update hive-site.xml for spark UTs to add hive_admin_user to admin role - * [HIVE-8788] - UT: fix partition test case [Spark Branch] - * [HIVE-8793] - Refactor to make splitting SparkWork a physical resolver [Spark Branch] - * [HIVE-8810] - Make HashTableSinkOperator works for Spark Branch [Spark Branch] - * [HIVE-8833] - Unify spark client API and implement remote spark client.[Spark Branch] - * [HIVE-8834] - enable job progress monitoring of Remote Spark Context [Spark Branch] - * [HIVE-8835] - identify dependency scope for Remote Spark Context.[Spark Branch] - * [HIVE-8836] - Enable automatic tests with remote spark client [Spark Branch] - * [HIVE-8840] - Print prettier Spark work graph after HIVE-8793 [Spark Branch] - * [HIVE-8841] - Make RDD caching work for multi-insert after HIVE-8793 when map join is involved [Spark Branch] - * [HIVE-8842] - auto_join2.q produces incorrect tree [Spark Branch] - * [HIVE-8843] - Release RDD cache when Hive query is done [Spark Branch] - * [HIVE-8844] - Choose a persisent policy for RDD caching [Spark Branch] - * [HIVE-8852] - Update new spark progress API for local submitted job monitoring [Spark Branch] - * [HIVE-8853] - Make vectorization work with Spark [Spark Branch] - * [HIVE-8854] - Guava dependency conflict between hive driver and remote spark context[Spark Branch] - * [HIVE-8855] - Automatic calculate reduce number for spark job [Spark Branch] - * [HIVE-8859] - ColumnStatsTask fails because of SparkMapJoinResolver [Spark Branch] - * [HIVE-8860] - Populate ExecMapperContext in SparkReduceRecordHandler [Spark Branch] - * [HIVE-8862] - Fix ordering diferences on TestParse tests due to Java8 - * [HIVE-8865] - Needs to set hashTableMemoryUsage for MapJoinDesc [Spark Branch] - * [HIVE-8867] - Investigate test failure on mapjoin_filter_on_outerjoin.q [Spark Branch] - * [HIVE-8868] - SparkSession and SparkClient mapping[Spark Branch] - * [HIVE-8882] - Fix test TestHiveKVResultCache [Spark Branch] - * [HIVE-8883] - Investigate test failures on auto_join30.q [Spark Branch] - * [HIVE-8885] - Investigate test failure on auto_join13.q [Spark Branch] - * [HIVE-8887] - Investigate test failures on auto_join6, auto_join7, auto_join18, auto_join18_multi_distinct [Spark Branch] - * [HIVE-8892] - Use MEMORY_AND_DISK for RDD caching [Spark Branch] - * [HIVE-8899] - Merge from trunk to spark [Spark Branch] - * [HIVE-8905] - Servlet classes signer information does not match [Spark branch] - * [HIVE-8908] - Investigate test failure on join34.q [Spark Branch] - * [HIVE-8911] - Enable mapjoin hints [Spark Branch] - * [HIVE-8920] - IOContext problem with multiple MapWorks cloned for multi-insert [Spark Branch] - * [HIVE-8921] - Investigate test failure on auto_join2.q [Spark Branch] - * [HIVE-8922] - CBO: assorted date and timestamp issues - * [HIVE-8923] - HIVE-8512 needs to be fixed also for CBO - * [HIVE-8924] - Investigate test failure for join_empty.q [Spark Branch] - * [HIVE-8934] - Investigate test failure on bucketmapjoin10.q and bucketmapjoin11.q [Spark Branch] - * [HIVE-8936] - Add SORT_QUERY_RESULTS for join tests that do not guarantee order - * [HIVE-8942] - Investigate test failure on skewjoin.q [Spark Branch] - * [HIVE-8943] - Fix memory limit check for combine nested mapjoins [Spark Branch] - * [HIVE-8946] - Enable Map Join [Spark Branch] - * [HIVE-8951] - Spark remote context doesn't work with local-cluster [Spark Branch] - * [HIVE-8956] - Hive hangs while some error/exception happens beyond job execution [Spark Branch] - * [HIVE-8959] - SparkSession is not closed until JVM exit.[Spark Branch] - * [HIVE-8961] - Remove unnecessary dependency collection task [Spark Branch] - * [HIVE-8962] - Add SORT_QUERY_RESULTS for join tests that do not guarantee order #2 - * [HIVE-8963] - Investigate test failure on bucketmapjoin7.q [Spark Branch] - * [HIVE-8970] - Enable map join optimization only when hive.auto.convert.join is true [Spark Branch] - * [HIVE-8972] - Implement more fine-grained remote client-level events [Spark Branch] - * [HIVE-8982] - IndexOutOfBounds exception in mapjoin [Spark Branch] - * [HIVE-8984] - Enable job metrics/statistics gathering for remote spark context [Spark Branch] - * [HIVE-8991] - Fix custom_input_output_format [Spark Branch] - * [HIVE-8992] - Fix bucket related test failure: parquet_join.q [Spark Branch] - * [HIVE-8993] - Make sure Spark + HS2 work [Spark Branch] - * [HIVE-8994] - Merge from trunk Nov 28 2014 - * [HIVE-8995] - Find thread leak in RSC Tests [Spark Branch] - * [HIVE-8998] - Logging is not configured in spark-submit sub-process - * [HIVE-9007] - Hive may generate wrong plan for map join queries due to IdentityProjectRemover [Spark Branch] - * [HIVE-9016] - SparkCounter display name is not set correctly[Spark Branch] - * [HIVE-9019] - Avoid using SPARK_JAVA_OPTS [Spark Branch] - * [HIVE-9026] - Re-enable remaining tests after HIVE-8970 [Spark Branch] - * [HIVE-9027] - Enable ppd_join4 [Spark Branch] - * [HIVE-9030] - CBO: Plans with comparison of values with different types - * [HIVE-9033] - Fix ordering differences due to Java8 (part 2) - * [HIVE-9034] - CBO: type change in literal_ints.q - * [HIVE-9035] - CBO: Disable PPD when functions are non-deterministic (ppd_random.q - non-deterministic udf rand() pushed above join) - * [HIVE-9036] - Replace akka for remote spark client RPC [Spark Branch] - * [HIVE-9040] - Spark Memory can be formatted string [Spark Branch] - * [HIVE-9041] - Generate better plan for queries containing both union and multi-insert [Spark Branch] - * [HIVE-9042] - Support multiple mapjoin operators in one work [Spark Branch] - * [HIVE-9043] - HiveException: Conflict on row inspector for {table} - * [HIVE-9054] - Add additional logging to SetSparkReducerParallelism [Spark Branch] - * [HIVE-9059] - Remove wrappers for SparkJobInfo and SparkStageInfo [Spark Branch] - * [HIVE-9063] - NPE in RemoteSparkJobStatus.getSparkStatistics [Spark Branch] - * [HIVE-9066] - temporarily disable CBO for non-deterministic functions - * [HIVE-9078] - Hive should not submit second SparkTask while previous one has failed.[Spark Branch] - * [HIVE-9079] - Hive hangs while failed to get executorCount[Spark Branch] - * [HIVE-9081] - Bucket mapjoin should use the new alias in posToAliasMap [Spark Branch] - * [HIVE-9085] - Spark Client RPC should have larger default max message size [Spark Branch] - * [HIVE-9088] - Spark counter serialization error in spark.log [Spark Branch] - * [HIVE-9094] - TimeoutException when trying get executor count from RSC [Spark Branch] - * [HIVE-9098] - Check cross product for conditional task [Spark Branch] - * [HIVE-9099] - infer_bucket_sort_convert_join.q and mapjoin_hook.q failed.[Spark Branch] - * [HIVE-9101] - bucket_map_join_spark4.q failed due to NPE.[Spark Branch] - * [HIVE-9104] - windowing.q failed when mapred.reduce.tasks is set to larger than one - * [HIVE-9109] - Add support for Java 8 specific q-test out files - * [HIVE-9110] - Performance of SELECT COUNT(*) FROM store_sales WHERE ss_item_sk IS NOT NULL [Spark Branch] - * [HIVE-9116] - Add unit test for multi sessions.[Spark Branch] - * [HIVE-9121] - Enable beeline query progress information for Spark job[Spark Branch] - * [HIVE-9127] - Improve CombineHiveInputFormat.getSplit performance - * [HIVE-9133] - CBO (Calcite Return Path): Refactor Semantic Analyzer to Move CBO code out - * [HIVE-9135] - Cache Map and Reduce works in RSC [Spark Branch] - * [HIVE-9136] - Profile query compiler [Spark Branch] - * [HIVE-9153] - Perf enhancement on CombineHiveInputFormat and HiveInputFormat - * [HIVE-9157] - Merge from trunk to spark 12/26/2014 [Spark Branch] - * [HIVE-9161] - Fix ordering differences on UDF functions due to Java8 - * [HIVE-9164] - Profile query compiler #2 [Spark Branch] - * [HIVE-9170] - UT: udf_in_file fails with filenotfoundexception [Spark Branch] - * [HIVE-9174] - Enable queuing of HCatalog notification events in metastore DB - * [HIVE-9175] - Add alters to list of events handled by NotificationListener - * [HIVE-9178] - Create a separate API for remote Spark Context RPC other than job submission [Spark Branch] - * [HIVE-9179] - Add listeners on JobHandle so job status change can be notified to the client [Spark Branch] - * [HIVE-9181] - Fix SkewJoinOptimizer related Java 8 ordering differences - * [HIVE-9184] - Modify HCatClient to support new notification methods in HiveMetaStoreClient - * [HIVE-9192] - One-pass SMB Optimizations [Spark Branch] - * [HIVE-9193] - Fix ordering differences due to Java 8 (Part 3) - * [HIVE-9194] - Support select distinct * - * [HIVE-9200] - CBO (Calcite Return Path): Inline Join, Properties - * [HIVE-9202] - Choose Kryo as the serializer for pTest [Spark Branch] - * [HIVE-9206] - Fix Desc Formatted related Java 8 ordering differences - * [HIVE-9207] - Add more log information for debug RSC[Spark Branch] - * [HIVE-9209] - Spark branch compile failed on hadoop-1[Spark Branch] - * [HIVE-9211] - Research on build mini HoS cluster on YARN for unit test[Spark Branch] - * [HIVE-9216] - Avoid redundant clone of JobConf [Spark Branch] - * [HIVE-9218] - Remove authorization_admin_almighty1 from spark tests [Spark Branch] - * [HIVE-9222] - Fix ordering differences due to Java 8 (Part 4) - * [HIVE-9224] - CBO (Calcite Return Path): Inline Table, Properties - * [HIVE-9230] - Followup for HIVE-9125, update ppd_join4.q.out for Spark [Spark Branch] - * [HIVE-9239] - Fix ordering differences due to Java 8 (Part 5) - * [HIVE-9241] - Fix TestCliDriver.testCliDriver_subquery_multiinsert - * [HIVE-9246] - Remove tabs from spark code [Spark Branch] - * [HIVE-9251] - SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch] - * [HIVE-9256] - Merge trunk to spark 1/5/2015 [Spark Branch] - * [HIVE-9257] - Merge from spark to trunk January 2015 - * [HIVE-9259] - Fix ClassCastException when CBO is enabled for HOS [Spark Branch] - * [HIVE-9264] - Merge encryption branch to trunk - * [HIVE-9267] - Ensure custom UDF works with Spark [Spark Branch] - * [HIVE-9281] - Code cleanup [Spark Branch] - * [HIVE-9288] - TODO cleanup task1.[Spark Branch] - * [HIVE-9292] - CBO (Calcite Return Path): Inline GroupBy, Properties - * [HIVE-9305] - Set default miniClusterType back to none in QTestUtil.[Spark branch] - * [HIVE-9306] - Let Context.isLocalOnlyExecutionMode() return false if execution engine is Spark [Spark Branch] - * [HIVE-9315] - CBO (Calcite Return Path): Inline FileSinkOperator, Properties - * [HIVE-9319] - Cleanup Modified Files [Spark Branch] - * [HIVE-9321] - Notification message size can be arbitrarily long, DbNotificationListener limits to 1024 - * [HIVE-9323] - Merge from trunk to spark 1/8/2015 - * [HIVE-9326] - BaseProtocol.Error failed to deserialization due to NPE.[Spark Branch] - * [HIVE-9335] - Address review items on HIVE-9257 [Spark Branch] - * [HIVE-9337] - Move more hive.spark.* configurations to HiveConf [Spark Branch] - * [HIVE-9342] - add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode [Spark Branch] - * [HIVE-9352] - Merge from spark to trunk (follow-up of HIVE-9257) - * [HIVE-9370] - SparkJobMonitor timeout as sortByKey would launch extra Spark job before original job get submitted [Spark Branch] - * [HIVE-9379] - Fix tests with some versions of Spark + Snappy [Spark Branch] - * [HIVE-9387] - Shutting down cli takes quite some time [Spark Branch] - * [HIVE-9395] - Make WAIT_SUBMISSION_TIMEOUT configuable and check timeout in SparkJobMonitor level.[Spark Branch] - * [HIVE-9409] - Avoid ser/de loggers as logging framework can be incompatible on driver and workers - * [HIVE-9410] - ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch] - * [HIVE-9425] - Add jar/file doesn't work with yarn-cluster mode [Spark Branch] - * [HIVE-9426] - Merge trunk to spark 1/21/2015 - * [HIVE-9428] - LocalSparkJobStatus may return failed job as successful [Spark Branch] - * [HIVE-9431] - CBO (Calcite Return Path): Removing AST from ParseContext - * [HIVE-9434] - Shim the method Path.getPathWithoutSchemeAndAuthority - * [HIVE-9444] - CBO (Calcite Return Path): Rewrite GlobalLimitOptimizer - * [HIVE-9449] - Push YARN configuration to Spark while deply Spark on YARN[Spark Branch] - * [HIVE-9450] - [Parquet] Check all data types work for Parquet in Group By operator - * [HIVE-9477] - No error thrown when global limit optimization failed to find enough number of rows [Spark Branch] - * [HIVE-9487] - Make Remote Spark Context secure [Spark Branch] - * [HIVE-9493] - Failed job may not throw exceptions [Spark Branch] - * [HIVE-9517] - UNION ALL query failed with ArrayIndexOutOfBoundsException [Spark Branch] - * [HIVE-9552] - Merge trunk to Spark branch 2/2/2015 [Spark Branch] - * [HIVE-9568] - Revert changes in two test configuration files accidently brought in by HIVE-9552 [Spark Branch] - * [HIVE-9574] - Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark Branch] - -** Bug - * [HIVE-1344] - error in select disinct - * [HIVE-1654] - select distinct should allow column name regex - * [HIVE-1869] - TestMTQueries failing on jenkins - * [HIVE-3781] - Index related events should be delivered to metastore event listener - * [HIVE-4009] - CLI Tests fail randomly due to MapReduce LocalJobRunner race condition - * [HIVE-5536] - Incorrect Operation Name is passed to hookcontext - * [HIVE-5865] - AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8' - * [HIVE-6165] - Unify HivePreparedStatement from jdbc:hive and jdbc:hive2 - * [HIVE-6308] - COLUMNS_V2 Metastore table not populated for tables created without an explicit column list. - * [HIVE-6421] - abs() should preserve precision/scale of decimal input - * [HIVE-6623] - Add "owner" tag to ptest2 created instances - * [HIVE-6683] - Beeline does not accept comments at end of line - * [HIVE-6914] - parquet-hive cannot write nested map (map value is map) - * [HIVE-7024] - Escape control characters for explain result - * [HIVE-7069] - Zookeeper connection leak - * [HIVE-7431] - When run on spark cluster, some spark tasks may fail - * [HIVE-7489] - Change ql/pom.xml to fix mvn project setup [Spark Branch] - * [HIVE-7523] - Fix spark build after merge merging trunk - * [HIVE-7530] - Go thru the common code to find references to HIVE_EXECUCTION_ENGINE to make sure conditions works with Spark [Spark Branch] - * [HIVE-7540] - NotSerializableException encountered when using sortByKey transformation - * [HIVE-7556] - Fix code style, license header, tabs, etc. [Spark Branch] - * [HIVE-7591] - GenMapRedUtils::addStatsTask only assumes either MapredWork or TezWork - * [HIVE-7624] - Reduce operator initialization failed when running multiple MR query on spark - * [HIVE-7626] - Add jar through CLI did not loaded by Spark executor[Spark Branck] - * [HIVE-7627] - FSStatsPublisher does fit into Spark multi-thread task mode[Spark Branch] - * [HIVE-7642] - Set hive input format by configuration.[Spark Branch] - * [HIVE-7643] - ExecMapper static states lead to unpredictable query result.[Spark Branch] - * [HIVE-7652] - Check OutputCollector after closing ExecMapper/ExecReducer - * [HIVE-7745] - NullPointerException when turn on hive.optimize.union.remove, hive.merge.mapfiles and hive.merge.mapredfiles [Spark Branch] - * [HIVE-7747] - Submitting a query to Spark from HiveServer2 fails [Spark Branch] - * [HIVE-7763] - Failed to query TABLESAMPLE on empty bucket table [Spark Branch] - * [HIVE-7795] - Enable ptf.q and ptf_streaming.q.[Spark Branch] - * [HIVE-7799] - TRANSFORM failed in transform_ppr1.q[Spark Branch] - * [HIVE-7909] - Fix sample8.q automatic test failure[Spark Branch] - * [HIVE-7930] - enable vectorization_short_regress.q, vector_string_concat.q [Spark Branch] - * [HIVE-7932] - It may cause NP exception when add accessed columns to ReadEntity - * [HIVE-7951] - InputFormats implementing (Job)Configurable should not be cached - * [HIVE-7997] - Potential null pointer reference in ObjectInspectorUtils#compareTypes() - * [HIVE-8118] - Support work that have multiple child works to work around SPARK-3622 [Spark Branch] - * [HIVE-8180] - Update SparkReduceRecordHandler for processing the vectors [spark branch] - * [HIVE-8182] - beeline fails when executing multiple-line queries with trailing spaces - * [HIVE-8257] - Accumulo introduces old hadoop-client dependency - * [HIVE-8266] - create function using <resource> statement compilation should include resource URI entity - * [HIVE-8284] - Equality comparison is done between two floating point variables in HiveRelMdUniqueKeys#getUniqueKeys() - * [HIVE-8300] - Missing guava lib causes IllegalStateException when deserializing a task [Spark Branch] - * [HIVE-8308] - Acid related table properties should be defined in one place and should be case insensitive - * [HIVE-8317] - WebHCat pom should explicitly depend on jersey-core - * [HIVE-8326] - Using DbTxnManager with concurrency off results in run time error - * [HIVE-8330] - HiveResultSet.findColumn() parameters are case sensitive - * [HIVE-8338] - Add ip and command to semantic analyzer hook context - * [HIVE-8345] - q-test for Avro date support - * [HIVE-8359] - Map containing null values are not correctly written in Parquet files - * [HIVE-8381] - Update hive version on trunk to 0.15 - * [HIVE-8387] - add retry logic to ZooKeeperStorage in WebHCat - * [HIVE-8425] - enable vectorized_mapjoin.q,vectorized_nested_mapjoin.q [Sparch Branch] - * [HIVE-8426] - paralle.q assert failed.[Spark Branch] - * [HIVE-8448] - Union All might not work due to the type conversion issue - * [HIVE-8450] - Create table like does not copy over table properties - * [HIVE-8457] - MapOperator initialization fails when multiple Spark threads is enabled [Spark Branch] - * [HIVE-8491] - Fix build name in ptest pre-commit message - * [HIVE-8496] - Re-enable statistics [Spark Branch] - * [HIVE-8500] - beeline does not need to set hive.aux.jars.path - * [HIVE-8512] - queries with star and gby produce incorrect results - * [HIVE-8518] - Compile time skew join optimization returns duplicated results - * [HIVE-8523] - Potential null dereference in DDLSemanticAnalyzer#addInputsOutputsAlterTable() - * [HIVE-8542] - Enable groupby_map_ppr.q and groupby_map_ppr_multi_distinct.q [Spark Branch] - * [HIVE-8545] - Exception when casting Text to BytesWritable [Spark Branch] - * [HIVE-8556] - introduce overflow control and sanity check to BytesBytesMapJoin - * [HIVE-8564] - DROP TABLE IF EXISTS throws exception if the table does not exist. - * [HIVE-8565] - beeline may go into an infinite loop when using EOF - * [HIVE-8576] - Guaranteed NPE in StatsRulesProcFactory - * [HIVE-8594] - Wrong condition in SettableConfigUpdater#setHiveConfWhiteList() - * [HIVE-8600] - Add option to log explain output for query - * [HIVE-8610] - Compile time skew join optimization doesn't work with auto map join - * [HIVE-8611] - grant/revoke syntax should support additional objects for authorization plugins - * [HIVE-8612] - Support metadata result filter hooks - * [HIVE-8613] - percentile_approx raise a comparator error - * [HIVE-8627] - Compute stats on a table from impala caused the table to be corrupted - * [HIVE-8634] - HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly - * [HIVE-8636] - CBO: split cbo_correctness test - * [HIVE-8666] - hive.metastore.server.max.threads default is too high - * [HIVE-8680] - Set Max Message for Binary Thrift endpoints - * [HIVE-8693] - Separate out fair scheduler dependency from hadoop 0.23 shim - * [HIVE-8708] - Add query id to explain log option - * [HIVE-8720] - Update orc_merge tests to make it consistent across OS'es - * [HIVE-8728] - Fix ptf.q determinism - * [HIVE-8730] - schemaTool failure when date partition has non-date value - * [HIVE-8736] - add ordering to cbo_correctness to make result consistent - * [HIVE-8743] - Disable MapJoin [Spark Branch] - * [HIVE-8757] - YARN dep in scheduler shim should be optional - * [HIVE-8762] - HiveMetaStore.BooleanPointer should be replaced with an AtomicBoolean - * [HIVE-8791] - Hive permission inheritance throws exception S3 - * [HIVE-8796] - TestCliDriver acid tests with decimal needs benchmark to be updated - * [HIVE-8797] - Simultaneous dynamic inserts can result in "partition already exists" error - * [HIVE-8803] - DESC SCHEMA <DATABASE-NAME> is not working - * [HIVE-8808] - HiveInputFormat caching cannot work with all input formats - * [HIVE-8812] - TestMinimrCliDriver failure if run in the same command as TestHBaseNegativeCliDriver - * [HIVE-8825] - SQLCompletor catches Throwable and ignores it - * [HIVE-8847] - Fix bugs in jenkins scripts - * [HIVE-8848] - data loading from text files or text file processing doesn't handle nulls correctly - * [HIVE-8850] - ObjectStore:: rollbackTransaction() needs to be looked into further. - * [HIVE-8863] - Cannot drop table with uppercase name after "compute statistics for columns" - * [HIVE-8869] - RowSchema not updated for some ops when columns are pruned - * [HIVE-8872] - Hive view of HBase range scan intermittently returns incorrect data. - * [HIVE-8874] - Error Accessing HBase from Hive via Oozie on Kerberos 5.0.1 cluster - * [HIVE-8875] - hive.optimize.sort.dynamic.partition should be turned off for ACID - * [HIVE-8877] - improve context logging during job submission via WebHCat - * [HIVE-8879] - Upgrade derby version to address race candition - * [HIVE-8881] - Receiving json "{"error":"Could not find job job_1415748506143_0002"}" when web client tries to fetch all jobs from webhcat where HDFS does not have the data. - * [HIVE-8889] - JDBC Driver ResultSet.getXXXXXX(String columnLabel) methods Broken - * [HIVE-8891] - Another possible cause to NucleusObjectNotFoundException from drops/rollback - * [HIVE-8893] - Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode - * [HIVE-8901] - increase retry attempt, interval on metastore database errors - * [HIVE-8909] - Hive doesn't correctly read Parquet nested types - * [HIVE-8914] - HDFSCleanup thread holds reference to FileSystem - * [HIVE-8916] - Handle user@domain username under LDAP authentication - * [HIVE-8917] - HIVE-5679 adds two thread safety problems - * [HIVE-8926] - Projections that only swap input columns are identified incorrectly as identity projections - * [HIVE-8938] - Compiler should save the transform URI as input entity - * [HIVE-8944] - TestCompactor fails with IncompatibleClassChangeError - * [HIVE-8948] - TestStreaming is flaky - * [HIVE-8964] - Some TestMiniTezCliDriver tests taking two hours - * [HIVE-8965] - Enhance PTest to kill all processes between tests and to report when a TEST*.xml file is not generated - * [HIVE-8967] - Fix bucketmapjoin7.q determinism - * [HIVE-8975] - Possible performance regression on bucket_map_join_tez2.q - * [HIVE-8978] - Fix test determinism issue for qfile: smb_mapjoin_1.q etc - * [HIVE-8986] - Add limit_partition_metadataonly.q in testconfiguration.properties [Spark Branch] - * [HIVE-8990] - mapjoin_mapjoin.q is failing on Tez (missed golden file update) - * [HIVE-9001] - Ship with log4j.properties file that has a reliable time based rolling policy - * [HIVE-9006] - hiveserver thrift api version is still 6 - * [HIVE-9011] - Fix parquet_join.q determinism - * [HIVE-9024] - NullPointerException when starting webhcat server if templeton.hive.properties is not set - * [HIVE-9032] - Help for orcfiledump script does not reflect new options - * [HIVE-9048] - Hive build failed on hadoop-1 after HIVE-8828. - * [HIVE-9055] - Tez: union all followed by group by followed by another union all gives error - * [HIVE-9060] - Fix child operator references after NonBlockingOpDeDupProc - * [HIVE-9077] - Set completer in CliDriver is not working - * [HIVE-9096] - GenericUDF may be left unclosed in PartitionPrune#visitCall() - * [HIVE-9113] - Explain on query failed with NPE - * [HIVE-9120] - Hive Query log does not work when hive.exec.parallel is true - * [HIVE-9122] - Need to remove additional references to hive-shims-common-secure, hive-shims-0.20 - * [HIVE-9129] - Migrate to newer Calcite snapshot, where ByteString is now in org.apache.calcite.avatica.util - * [HIVE-9130] - vector_partition_diff_num_cols result is not updated after CBO upgrade - * [HIVE-9131] - MiniTez optimize_nullscan test is unstable - * [HIVE-9149] - Add unit test to test implicit conversion during dynamic partitioning/distribute by - * [HIVE-9150] - Unrelated types are compared in GenTezWork#getFollowingWorkIndex() - * [HIVE-9154] - Cache pathToPartitionInfo in context aware record reader - * [HIVE-9177] - Fix child operator references after NonBlockingOpDeDupProc (II) - * [HIVE-9195] - CBO changes constant to column type - * [HIVE-9197] - fix lvj_mapjoin.q diff in trunk - * [HIVE-9199] - Excessive exclusive lock used in some DDLs with DummyTxnManager - * [HIVE-9203] - CREATE TEMPORARY FUNCTION hangs trying to acquire lock - * [HIVE-9215] - Some mapjoin queries broken with IdentityProjectRemover with PPD - * [HIVE-9221] - Remove deprecation warning for hive.metastore.local - * [HIVE-9242] - Many places in CBO code eat exceptions - * [HIVE-9243] - Static Map in IOContext is not thread safe - * [HIVE-9255] - Fastpath for limited fetches from unpartitioned tables - * [HIVE-9289] - TODO : Store user name in session [Spark Branch] - * [HIVE-9296] - Need to add schema upgrade changes for queueing events in the database - * [HIVE-9299] - Reuse Configuration in AvroSerdeUtils - * [HIVE-9300] - Make TCompactProtocol configurable - * [HIVE-9301] - Potential null dereference in MoveTask#createTargetPath() - * [HIVE-9309] - schematool fails on Postgres 8.1 - * [HIVE-9310] - CLI JLine does not flush history back to ~/.hivehistory - * [HIVE-9316] - TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs - * [HIVE-9330] - DummyTxnManager will throw NPE if WriteEntity writeType has not been set - * [HIVE-9331] - get rid of pre-optimized-hashtable memory optimizations - * [HIVE-9344] - Fix flaky test optimize_nullscan - * [HIVE-9347] - Bug with max() together with rank() and grouping sets - * [HIVE-9351] - Running Hive Jobs with Tez cause templeton to never report percent complete - * [HIVE-9353] - make TABLE keyword optional in INSERT INTO TABLE foo... - * [HIVE-9366] - wrong date in description annotation in date_add() and date_sub() udf - * [HIVE-9369] - fix arguments length checking in Upper and Lower UDF - * [HIVE-9377] - UDF in_file() in WHERE predicate causes NPE. - * [HIVE-9381] - HCatalog hardcodes maximum append limit to 1000. - * [HIVE-9382] - Query got rerun with Global Limit optimization on and Fetch optimization off - * [HIVE-9386] - FileNotFoundException when using in_file() - * [HIVE-9393] - reduce noisy log level of ColumnarSerDe.java:116 from INFO to DEBUG - * [HIVE-9396] - date_add()/date_sub() should allow tinyint/smallint/bigint arguments in addition to int - * [HIVE-9414] - Fixup post HIVE-9264 - Merge encryption branch to trunk - * [HIVE-9437] - Beeline does not add any existing HADOOP_CLASSPATH - * [HIVE-9440] - Folders may not be pruned for Hadoop 2 - * [HIVE-9441] - Remove call to deprecated Calcite method - * [HIVE-9443] - ORC PPD - fix fuzzy case evaluation of IS_NULL - * [HIVE-9445] - Revert HIVE-5700 - enforce single date format for partition column storage - * [HIVE-9446] - JDBC DatabaseMetadata.getColumns() does not work for temporary tables - * [HIVE-9448] - Merge spark to trunk 1/23/15 - * [HIVE-9454] - Test failures due to new Calcite version - * [HIVE-9462] - HIVE-8577 - breaks type evolution - * [HIVE-9475] - HiveMetastoreClient.tableExists does not work - * [HIVE-9476] - Beeline fails to start on trunk - * [HIVE-9502] - Parquet cannot read Map types from files written with Hive <= 0.12 - * [HIVE-9515] - Check hive.vectorized.execution.reduce.enabled [Spark Branch] - * [HIVE-9519] - Bump up spark client connection timeout - * [HIVE-9593] - <TRUNCATED>
