[jira] [Updated] (HIVE-5209) JDBC support for varchar

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-5209:
-

Attachment: HIVE-5209.2.patch

new patch HIVE-5209.2.patch, rebasing against dependent patches.

> JDBC support for varchar
> 
>
> Key: HIVE-5209
> URL: https://issues.apache.org/jira/browse/HIVE-5209
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC, Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-5209.1.patch, HIVE-5209.2.patch, 
> HIVE-5209.D12705.1.patch
>
>
> Support returning varchar length in result set metadata
> NO PRECOMMIT TESTS - dependent on HIVE-4844

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4492) Revert HIVE-4322

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766167#comment-13766167
 ] 

Thejas M Nair commented on HIVE-4492:
-

This looks like an api backward compatibility fix that we should include in 
0.12 . [~ashutoshc] Do you agree ?


> Revert HIVE-4322
> 
>
> Key: HIVE-4492
> URL: https://issues.apache.org/jira/browse/HIVE-4492
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Thrift API
>Reporter: Samuel Yuan
>Assignee: Samuel Yuan
> Fix For: 0.13.0
>
> Attachments: HIVE-4492.1.patch.txt, HIVE-4492.patch
>
>
> See HIVE-4432 and HIVE-4433. It's possible to work around these issues but a 
> better solution is probably to roll back the "fix" and change the API to use 
> a primitive type as the map key (in a backwards-compatible manner).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4583) Make Hive compile and run with JDK7

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765618#comment-13765618
 ] 

Xuefu Zhang commented on HIVE-4583:
---

Given that the last dependent JIRA HIVE-5263 is closed, I think we are ready to 
close this JIRA as well. I ran a build based on trunk with JDK7, and all Hive 
tests passed. There were a few hcat test case failures, but I believe they are 
not related to JDK7, but are transient failures due to recent hcat rebase.

Please raise any concern you might have. Otherwise, the JIRA will be closed as 
resolved in a day or two. 

> Make Hive compile and run with JDK7
> ---
>
> Key: HIVE-4583
> URL: https://issues.apache.org/jira/browse/HIVE-4583
> Project: Hive
>  Issue Type: Task
>Affects Versions: 0.11.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>
> This is an umbrella ticket to cover all issues related to supporting JDK 7 in 
> HIVE. Many such issues are expected. Of course, JDK 6 needs to continue to 
> work.
> The major obstacles on the way supporting JDK7 are:
> 1. JDBC component needs to be upgraded because of JDBC interface changes in 
> JDK7.
> 2. DataNucleus needs to be upgraded because the current version doesn't 
> support JDK7.
> 3. Many test failures needs to be fixed. The majority of the failures are 
> caused by JDK7 subtle behaviour change.
> 4. Build needs to be changed to accommodate JDK7 compiler.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5161) Additional SerDe support for varchar type

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-5161:
-

Attachment: HIVE-5161.3.patch

uploading patch v3:
  - rebase against trunk, dependent patches
  - regenerate OrcProto using protobuf-2.5

> Additional SerDe support for varchar type
> -
>
> Key: HIVE-5161
> URL: https://issues.apache.org/jira/browse/HIVE-5161
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers, Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-5161.1.patch, HIVE-5161.2.patch, HIVE-5161.3.patch
>
>
> Breaking out support for varchar for the various SerDes as an additional task.
> NO PRECOMMIT TESTS - can't run tests until HIVE-4844 is committed

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5161) Additional SerDe support for varchar type

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766172#comment-13766172
 ] 

Jason Dere commented on HIVE-5161:
--

posted review at https://reviews.facebook.net/D12897

> Additional SerDe support for varchar type
> -
>
> Key: HIVE-5161
> URL: https://issues.apache.org/jira/browse/HIVE-5161
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers, Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-5161.1.patch, HIVE-5161.2.patch, HIVE-5161.3.patch
>
>
> Breaking out support for varchar for the various SerDes as an additional task.
> NO PRECOMMIT TESTS - can't run tests until HIVE-4844 is committed

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5281) Constructor for DelegationTokenInformation changed in HDFS-4680 which we use in HiveDelegationTokenSupport.

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765604#comment-13765604
 ] 

Brock Noland commented on HIVE-5281:


[~thejas] after HDFS-4680 delegation tokens will have more information. 
Additionally as of 0.12 we will be storing them in the metastore (HIVE-3255). 
HIVE-2467 in 0.8 put them in ZK but I am sure that is less used than HIVE-3255 
will be. My concern is that we have no way at present to version these things 
but we know more fields will be coming later. I feel we should add versioning 
in the 0.12 release, before we start storing them in the db, since we know they 
will need to be versioned in the future.

> Constructor for DelegationTokenInformation changed in HDFS-4680 which we use 
> in HiveDelegationTokenSupport.
> ---
>
> Key: HIVE-5281
> URL: https://issues.apache.org/jira/browse/HIVE-5281
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Brock Noland
>Assignee: Brock Noland
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5036) [WebHCat] Add cmd script for WebHCat

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766176#comment-13766176
 ] 

Thejas M Nair commented on HIVE-5036:
-

+1


> [WebHCat] Add cmd script for WebHCat
> 
>
> Key: HIVE-5036
> URL: https://issues.apache.org/jira/browse/HIVE-5036
> Project: Hive
>  Issue Type: Sub-task
>  Components: HCatalog
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-5036-1.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4910) Hadoop 2 archives broken

[Vikram Dixit K (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766177#comment-13766177
 ] 

Vikram Dixit K commented on HIVE-4910:
--

Hi Brock,

I had a conversation with the HDFS folks some time back and it seems like there 
is a difference in the way mapreduce forms the URI vs how we form it. Mapreduce 
seems to have been able to form the URI correctly (with the ':' et. al.). 
Ideally, we need to use/replicate what mapreduce does. We could go with this 
fix for now but we may need to revisit this.

Thanks
Vikram.

> Hadoop 2 archives broken
> 
>
> Key: HIVE-4910
> URL: https://issues.apache.org/jira/browse/HIVE-4910
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor, Tests
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Minor
> Fix For: 0.11.1
>
> Attachments: HIVE-4910.patch, HIVE-4910.patch
>
>
> Hadoop 2 archive tests are broken. The issue stems from the fact that har uri 
> construction does not really have a port in the URI when unit tests are run. 
> This means that an invalid uri is constructed resulting in failures.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5264) SQL generated by MetaStoreDirectSql.java not compliant with Postgres.

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-5264:
---

Attachment: HIVE-5264.patch

preliminary patch, let me run the tests against different stores and then 
submit to Hive QA, there may be some bugs

> SQL generated by MetaStoreDirectSql.java not compliant with Postgres.
> -
>
> Key: HIVE-5264
> URL: https://issues.apache.org/jira/browse/HIVE-5264
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.12.0
> Environment: Ubuntu 12.04
> PostgreSQL 9.1.8
>Reporter: Alexander Behm
>Assignee: Sergey Shelukhin
> Attachments: HIVE-5264.patch
>
>
> Some operations against the Hive Metastore seem broken
> against Postgres.
> For example, when using HiveMetastoreClient.listPartitions()
> the Postgres logs show queries such as:
> 2013-09-09 19:10:01 PDT STATEMENT:  select PARTITIONS.PART_ID from
> PARTITIONS  inner join TBLS on PARTITIONS.TBL_ID = TBLS.TBL_ID   inner
> join DBS on TBLS.DB_ID = DBS.DB_ID  where TBLS.TBL_NAME = $1 and
> DBS.NAME = $2 order by PART_NAME asc
> with a somewhat cryptic (but correct) error:
> ERROR:  relation "partitions" does not exist at character 32
> Postgres identifiers are somewhat unusual. Unquoted identifiers are 
> interpreted as lower case (there is no Postgres option to change this). Since 
> the Metastore table schema uses upper case table names, the correct SQL 
> requires escaped identifiers to those tables, i.e.,
> select "PARTITIONS"."PART_ID" from "PARTITIONS"...
> Hive sets metastore.try.direct.sql=true by default, so the above SQL is 
> generated by hive/metastore/MetaStoreDirectSql.java, i.e., this is not a 
> Datanucleus problem.
> When I set metastore.try.direct.sql=false, then the Metastore backed by 
> Postgres works.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5035) [WebHCat] Hardening parameters for Windows

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-5035:


Description: 
everything pass to pig/hive/hadoop command line will be quoted. That include:
mapreducejar:
libjars
arg
define

mapreducestream:
cmdenv
define
arg

pig
arg
execute

hive
arg
define
execute

NO PRECOMMIT TESTS 

  was:
everything pass to pig/hive/hadoop command line will be quoted. That include:
mapreducejar:
libjars
arg
define

mapreducestream:
cmdenv
define
arg

pig
arg
execute

hive
arg
define
execute


> [WebHCat] Hardening parameters for Windows
> --
>
> Key: HIVE-5035
> URL: https://issues.apache.org/jira/browse/HIVE-5035
> Project: Hive
>  Issue Type: Sub-task
>  Components: HCatalog
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-5035-1.patch
>
>
> everything pass to pig/hive/hadoop command line will be quoted. That include:
> mapreducejar:
> libjars
> arg
> define
> mapreducestream:
> cmdenv
> define
> arg
> pig
> arg
> execute
> hive
> arg
> define
> execute
> NO PRECOMMIT TESTS 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5036) [WebHCat] Add cmd script for WebHCat

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-5036:


Description: NO PRECOMMIT TESTS 

> [WebHCat] Add cmd script for WebHCat
> 
>
> Key: HIVE-5036
> URL: https://issues.apache.org/jira/browse/HIVE-5036
> Project: Hive
>  Issue Type: Sub-task
>  Components: HCatalog
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-5036-1.patch
>
>
> NO PRECOMMIT TESTS 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5035) [WebHCat] Hardening parameters for Windows

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766180#comment-13766180
 ] 

Thejas M Nair commented on HIVE-5035:
-

+1 Will run the test manually as its webhcat-only change (running all hive 
tests is going to be an overkill, I think we should look at enabling hcat-only 
pre-commit tests).



> [WebHCat] Hardening parameters for Windows
> --
>
> Key: HIVE-5035
> URL: https://issues.apache.org/jira/browse/HIVE-5035
> Project: Hive
>  Issue Type: Sub-task
>  Components: HCatalog
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-5035-1.patch
>
>
> everything pass to pig/hive/hadoop command line will be quoted. That include:
> mapreducejar:
> libjars
> arg
> define
> mapreducestream:
> cmdenv
> define
> arg
> pig
> arg
> execute
> hive
> arg
> define
> execute
> NO PRECOMMIT TESTS 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5036) [WebHCat] Add cmd script for WebHCat

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766182#comment-13766182
 ] 

Thejas M Nair commented on HIVE-5036:
-

Will run the test manually as its webhcat-only change (running all hive tests 
is going to be an overkill, I think we should look at enabling hcat-only 
pre-commit tests).

> [WebHCat] Add cmd script for WebHCat
> 
>
> Key: HIVE-5036
> URL: https://issues.apache.org/jira/browse/HIVE-5036
> Project: Hive
>  Issue Type: Sub-task
>  Components: HCatalog
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-5036-1.patch
>
>
> NO PRECOMMIT TESTS 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5078) [WebHCat] Fix e2e tests on Windows plus test cases for new features

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766186#comment-13766186
 ] 

Thejas M Nair commented on HIVE-5078:
-

[~daijy] Can you please address Eugene's comment ?


> [WebHCat] Fix e2e tests on Windows plus test cases for new features
> ---
>
> Key: HIVE-5078
> URL: https://issues.apache.org/jira/browse/HIVE-5078
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-5078-1.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4844) Add varchar data type

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-4844:
-

Attachment: HIVE-4844.16.patch

Apparently hadn't completely removed java files for 
GenericUDFConcat/Upper/Lower - they were still in the patch as empty files. 
Adding patch v16 which removes those empty files.

> Add varchar data type
> -
>
> Key: HIVE-4844
> URL: https://issues.apache.org/jira/browse/HIVE-4844
> Project: Hive
>  Issue Type: New Feature
>  Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-4844.10.patch, HIVE-4844.11.patch, 
> HIVE-4844.12.patch, HIVE-4844.13.patch, HIVE-4844.14.patch, 
> HIVE-4844.15.patch, HIVE-4844.16.patch, HIVE-4844.1.patch.hack, 
> HIVE-4844.2.patch, HIVE-4844.3.patch, HIVE-4844.4.patch, HIVE-4844.5.patch, 
> HIVE-4844.6.patch, HIVE-4844.7.patch, HIVE-4844.8.patch, HIVE-4844.9.patch, 
> HIVE-4844.D12699.1.patch, screenshot.png
>
>
> Add new varchar data types which have support for more SQL-compliant 
> behavior, such as SQL string comparison semantics, max length, etc.
> Char type will be added as another task.
> NO PRECOMMIT TESTS - now dependent on HIVE-5203/5204/5206

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4444) [HCatalog] WebHCat Hive should support equivalent parameters as Pig

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-:


Description: 
Currently there is no "files" and "args" parameter in Hive. We shall add them 
to make them similar to Pig.
NO PRECOMMIT TESTS 

  was:Currently there is no "files" and "args" parameter in Hive. We shall add 
them to make them similar to Pig.


> [HCatalog] WebHCat Hive should support equivalent parameters as Pig 
> 
>
> Key: HIVE-
> URL: https://issues.apache.org/jira/browse/HIVE-
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Reporter: Daniel Dai
> Attachments: HIVE--1.patch, HIVE--2.patch
>
>
> Currently there is no "files" and "args" parameter in Hive. We shall add them 
> to make them similar to Pig.
> NO PRECOMMIT TESTS 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4444) [HCatalog] WebHCat Hive should support equivalent parameters as Pig

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766189#comment-13766189
 ] 

Thejas M Nair commented on HIVE-:
-

+1  I will run the hcat tests manually.


> [HCatalog] WebHCat Hive should support equivalent parameters as Pig 
> 
>
> Key: HIVE-
> URL: https://issues.apache.org/jira/browse/HIVE-
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Reporter: Daniel Dai
> Attachments: HIVE--1.patch, HIVE--2.patch
>
>
> Currently there is no "files" and "args" parameter in Hive. We shall add them 
> to make them similar to Pig.
> NO PRECOMMIT TESTS 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4443) [HCatalog] Have an option for GET queue to return all job information in single call

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-4443:


Description: 
Currently do display a summary of all jobs, one has to call GET queue to 
retrieve all the jobids and then call GET queue/:jobid for each job. It would 
be nice to do this in a single call.

I would suggest:
* GET queue - mark deprecate
* GET queue/ - mark deprecate
* DELETE queue/ - mark deprecate
* GET jobs - return the list of JSON objects jobid but no detailed info
* GET jobs/fields=* - return the list of JSON objects containing detailed Job 
info
* GET jobs/ - return the single JSON object containing the detailed Job 
info for the job with the given ID (equivalent to GET queue/)
* DELETE jobs/ - equivalent to DELETE queue/

NO PRECOMMIT TESTS 

  was:
Currently do display a summary of all jobs, one has to call GET queue to 
retrieve all the jobids and then call GET queue/:jobid for each job. It would 
be nice to do this in a single call.

I would suggest:
* GET queue - mark deprecate
* GET queue/ - mark deprecate
* DELETE queue/ - mark deprecate
* GET jobs - return the list of JSON objects jobid but no detailed info
* GET jobs/fields=* - return the list of JSON objects containing detailed Job 
info
* GET jobs/ - return the single JSON object containing the detailed Job 
info for the job with the given ID (equivalent to GET queue/)
* DELETE jobs/ - equivalent to DELETE queue/


> [HCatalog] Have an option for GET queue to return all job information in 
> single call 
> -
>
> Key: HIVE-4443
> URL: https://issues.apache.org/jira/browse/HIVE-4443
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-4443-1.patch, HIVE-4443-2.patch, HIVE-4443-3.patch
>
>
> Currently do display a summary of all jobs, one has to call GET queue to 
> retrieve all the jobids and then call GET queue/:jobid for each job. It would 
> be nice to do this in a single call.
> I would suggest:
> * GET queue - mark deprecate
> * GET queue/ - mark deprecate
> * DELETE queue/ - mark deprecate
> * GET jobs - return the list of JSON objects jobid but no detailed info
> * GET jobs/fields=* - return the list of JSON objects containing detailed Job 
> info
> * GET jobs/ - return the single JSON object containing the detailed 
> Job info for the job with the given ID (equivalent to GET queue/)
> * DELETE jobs/ - equivalent to DELETE queue/
> NO PRECOMMIT TESTS 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4443) [HCatalog] Have an option for GET queue to return all job information in single call

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766192#comment-13766192
 ] 

Thejas M Nair commented on HIVE-4443:
-

+1 . [~daijy] Is there a follow up jira with documentation update ? If not can 
you please create one (or link it with this one?)


> [HCatalog] Have an option for GET queue to return all job information in 
> single call 
> -
>
> Key: HIVE-4443
> URL: https://issues.apache.org/jira/browse/HIVE-4443
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-4443-1.patch, HIVE-4443-2.patch, HIVE-4443-3.patch
>
>
> Currently do display a summary of all jobs, one has to call GET queue to 
> retrieve all the jobids and then call GET queue/:jobid for each job. It would 
> be nice to do this in a single call.
> I would suggest:
> * GET queue - mark deprecate
> * GET queue/ - mark deprecate
> * DELETE queue/ - mark deprecate
> * GET jobs - return the list of JSON objects jobid but no detailed info
> * GET jobs/fields=* - return the list of JSON objects containing detailed Job 
> info
> * GET jobs/ - return the single JSON object containing the detailed 
> Job info for the job with the given ID (equivalent to GET queue/)
> * DELETE jobs/ - equivalent to DELETE queue/
> NO PRECOMMIT TESTS 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4844) Add varchar data type

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765858#comment-13765858
 ] 

Ashutosh Chauhan commented on HIVE-4844:


[~jdere] Left some comments on phabricator.

> Add varchar data type
> -
>
> Key: HIVE-4844
> URL: https://issues.apache.org/jira/browse/HIVE-4844
> Project: Hive
>  Issue Type: New Feature
>  Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-4844.10.patch, HIVE-4844.11.patch, 
> HIVE-4844.12.patch, HIVE-4844.13.patch, HIVE-4844.14.patch, 
> HIVE-4844.1.patch.hack, HIVE-4844.2.patch, HIVE-4844.3.patch, 
> HIVE-4844.4.patch, HIVE-4844.5.patch, HIVE-4844.6.patch, HIVE-4844.7.patch, 
> HIVE-4844.8.patch, HIVE-4844.9.patch, HIVE-4844.D12699.1.patch, screenshot.png
>
>
> Add new varchar data types which have support for more SQL-compliant 
> behavior, such as SQL string comparison semantics, max length, etc.
> Char type will be added as another task.
> NO PRECOMMIT TESTS - now dependent on HIVE-5203/5204/5206

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5084) Fix newline.q on Windows

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-5084:
-

Status: Patch Available  (was: Open)

> Fix newline.q on Windows
> 
>
> Key: HIVE-5084
> URL: https://issues.apache.org/jira/browse/HIVE-5084
> Project: Hive
>  Issue Type: Bug
>  Components: Tests, Windows
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-5084-1.patch
>
>
> Test failed with vague error message:
> [junit] Error during job, obtaining debugging information...
> [junit] junit.framework.AssertionFailedError: Client Execution failed 
> with error code = 2
> hive.log doesn't show something interesting either:
> 2013-08-14 00:47:29,411 DEBUG zookeeper.ClientCnxn 
> (ClientCnxn.java:readResponse(723)) - Got ping response for sessionid: 
> 0x1407a49fc1e0003 after 1ms
> 2013-08-14 00:47:31,391 ERROR exec.Task (SessionState.java:printError(416)) - 
> Execution failed with exit status: 2
> 2013-08-14 00:47:31,391 ERROR exec.Task (SessionState.java:printError(416)) - 
> Obtaining error information
> 2013-08-14 00:47:31,392 ERROR exec.Task (SessionState.java:printError(416)) - 
> Task failed!
> Task ID:
>   Stage-1
> Logs:

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5086) Fix scriptfile1.q on Windows

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-5086:
-

Status: Patch Available  (was: Open)

> Fix scriptfile1.q on Windows
> 
>
> Key: HIVE-5086
> URL: https://issues.apache.org/jira/browse/HIVE-5086
> Project: Hive
>  Issue Type: Bug
>  Components: Tests, Windows
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-5086-1.patch
>
>
> Test failed with error message:
> [junit] Task with the most failures(4): 
> [junit] -
> [junit] Task ID:
> [junit]   task_20130814023904691_0001_m_00
> [junit] 
> [junit] URL:
> [junit]   
> http://localhost:50030/taskdetails.jsp?jobid=job_20130814023904691_0001&tipid=task_20130814023904691_0001_m_00
> [junit] -
> [junit] Diagnostic Messages for this Task:
> [junit] java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row {"key":"238","value":"val_238"}
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
> [junit]   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> [junit]   at 
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> [junit]   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
> [junit]   at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
> [junit]   at java.security.AccessController.doPrivileged(Native Method)
> [junit]   at javax.security.auth.Subject.doAs(Subject.java:396)
> [junit]   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> [junit]   at org.apache.hadoop.mapred.Child.main(Child.java:265)
> [junit] Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive 
> Runtime Error while processing row {"key":"238","value":"val_238"}
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:538)
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
> [junit]   ... 8 more
> [junit] Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> [Error 2]: Unable to initialize custom script.
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:357)
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:848)
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88)
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:848)
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:848)
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:528)
> [junit]   ... 9 more
> [junit] Caused by: java.io.IOException: Cannot run program 
> "D:\tmp\hadoop-Administrator\mapred\local\3_0\taskTracker\Administrator\jobcache\job_20130814023904691_0001\attempt_20130814023904691_0001_m_00_3\work\.\testgrep":
>  CreateProcess error=193, %1 is not a valid Win32 application
> [junit]   at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:316)
> [junit]   ... 18 more
> [junit] Caused by: java.io.IOException: CreateProcess error=193, %1 is 
> not a valid Win32 application
> [junit]   at java.lang.ProcessImpl.create(Native Method)
> [junit]   at java.lang.ProcessImpl.(ProcessImpl.java:81)
> [junit]   at java.lang.ProcessImpl.start(ProcessImpl.java:30)
> [junit]   at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> [junit]   ... 19 more
> [junit] 
> [junit] 
> [junit] Exception: Client Execution failed with error code = 2
> [junit] See build/ql/tmp/hive.log, or try "ant test ... 
> -Dtest.silent=false" to get more logs.
> [junit] junit.framework.AssertionFailedError: Client Execution failed 
> with error code = 2
> [junit] See build/ql/tmp/hive.log, or try "ant test ... 
> -Dtest.silent=false" to get more logs.
> [junit]   at junit.framework.Assert.fail(Assert.java:47)
> [junit]   at 
> org.apache.hadoop.hive.cli.TestMinimrCliDriver.runTest(TestMinimrCliDriver.java:122)
> [junit]   at 
> org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_script

[jira] [Updated] (HIVE-5079) Make Hive compile under Windows

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-5079:
-

Status: Patch Available  (was: Open)

> Make Hive compile under Windows
> ---
>
> Key: HIVE-5079
> URL: https://issues.apache.org/jira/browse/HIVE-5079
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-5079-1.patch, HIVE-5079-2.patch
>
>
> Hive compilation failed under Windows. Error message:
> {code}
> compile:
>  [echo] Project: common
>  [exec] D:\Program Files (x86)\GnuWin32\bin\xargs.exe: md5sum: No such 
> file
> or directory
>  [exec] md5sum: 
> ../serde/src/java/org/apache/hadoop/hive/serde2/io/Timesta:
> No such file or directory
> [javac] Compiling 25 source files to 
> D:\Users\Administrator\hive\build\common\classes
> [javac] 
> D:\Users\Administrator\hive\common\src\gen\org\apache\hive\common\package-info.java:4:
>  unclosed string literal
> [javac] @HiveVersionAnnotation(version="0.12.0-SNAPSHOT", 
> revision="80eadd8fa2af5eeba61f921318ab8b2c19980ab3", branch="trunk
> [javac]
>   ^
> [javac] 
> D:\Users\Administrator\hive\common\src\gen\org\apache\hive\common\package-info.java:5:
>  unclosed string literal
> [javac] ",
> [javac] ^
> [javac] 
> D:\Users\Administrator\hive\common\src\gen\org\apache\hive\common\package-info.java:6:
>  class, interface, or enum expected
> [javac]  user="Administrator
> [javac]  ^
> [javac] 
> D:\Users\Administrator\hive\common\src\gen\org\apache\hive\common\package-info.java:6:
>  unclosed string literal
> [javac]  user="Administrator
> [javac]   ^
> [javac] 
> D:\Users\Administrator\hive\common\src\gen\org\apache\hive\common\package-info.java:10:
>  unclosed string literal
> [javac] ",
> [javac] ^
> [javac] 
> D:\Users\Administrator\hive\common\src\gen\org\apache\hive\common\package-info.java:11:
>  unclosed string literal
> [javac]  
> srcChecksum="aadceb95c37a1704aaf19501f46f6e84
> [javac]  ^
> [javac] 
> D:\Users\Administrator\hive\common\src\gen\org\apache\hive\common\package-info.java:12:
>  unclosed string literal
> [javac] ")
> [javac] ^
> [javac] 7 errors
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5088) Fix udf_translate.q on Windows

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-5088:
-

Status: Patch Available  (was: Open)

> Fix udf_translate.q on Windows
> --
>
> Key: HIVE-5088
> URL: https://issues.apache.org/jira/browse/HIVE-5088
> Project: Hive
>  Issue Type: Bug
>  Components: Tests, Windows
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-5088-1.patch
>
>
> Test failed with message:
> [junit] Begin query: udf_translate.q
> [junit] 13/08/14 03:23:57 FATAL conf.Configuration: error parsing conf 
> file: 
> com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 
> Invalid byte 1 of 1-byte UTF-8 sequence.
> [junit] Exception in thread "main" java.lang.RuntimeException: 
> com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 
> Invalid byte 1 of 1-byte UTF-8 sequence.
> [junit]   at 
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1255)
> [junit]   at 
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1117)
> [junit]   at 
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
> [junit]   at 
> org.apache.hadoop.conf.Configuration.get(Configuration.java:397)
> [junit]   at 
> org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:594)
> [junit]   at 
> org.apache.hadoop.hive.conf.HiveConf.getBoolVar(HiveConf.java:1015)
> [junit]   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:659)
> [junit]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit]   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> [junit]   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> [junit]   at java.lang.reflect.Method.invoke(Method.java:597)
> [junit]   at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> [junit] Caused by: 
> com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 
> Invalid byte 1 of 1-byte UTF-8 sequence.
> [junit]   at 
> com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.invalidByte(UTF8Reader.java:684)
> [junit]   at 
> com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:554)
> [junit]   at 
> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1742)
> [junit]   at 
> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.peekChar(XMLEntityScanner.java:487)
> [junit]   at 
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2688)
> [junit]   at 
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:647)
> [junit]   at 
> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
> [junit]   at 
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
> [junit]   at 
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
> [junit]   at 
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
> [junit]   at 
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
> [junit]   at 
> com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:232)
> [junit]   at 
> com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)
> [junit]   at 
> javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:124)
> [junit]   at 
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1181)
> [junit]   ... 11 more
> [junit] Exception: Client Execution failed with error code = 1
> [junit] See build/ql/tmp/hive.log, or try "ant test ... 
> -Dtest.silent=false" to get more logs.
> [junit] junit.framework.AssertionFailedError: Client Execution failed 
> with error code = 1
> [junit] See build/ql/tmp/hive.log, or try "ant test ... 
> -Dtest.silent=false" to get more logs.
> [junit]   at junit.framework.Assert.fail(Assert.java:47)
> [junit]   at 
> org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:122)
> [junit]   at 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_translate(TestCliDriver.java:104)
> [junit]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit]   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> [junit]   at 
> sun.reflect.DelegatingMethodAccessorImpl.i

[jira] [Updated] (HIVE-5206) Support parameterized primitive types

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-5206:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Jason!

> Support parameterized primitive types
> -
>
> Key: HIVE-5206
> URL: https://issues.apache.org/jira/browse/HIVE-5206
> Project: Hive
>  Issue Type: Improvement
>  Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 0.13.0
>
> Attachments: HIVE-5206.1.patch, HIVE-5206.2.patch, HIVE-5206.3.patch, 
> HIVE-5206.4.patch, HIVE-5206.D12693.1.patch
>
>
> Support for parameterized types is needed for char/varchar/decimal support. 
> This adds a type parameters value to the 
> PrimitiveTypeEntry/PrimitiveTypeInfo/PrimitiveObjectInspector objects. 
> NO PRECOMMIT TESTS - dependent on HIVE-5203/HIVE-5204

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5092) Fix hiveserver2 mapreduce local job on Windows

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-5092:
-

Status: Patch Available  (was: Open)

> Fix hiveserver2 mapreduce local job on Windows
> --
>
> Key: HIVE-5092
> URL: https://issues.apache.org/jira/browse/HIVE-5092
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Windows
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-5092-1.patch
>
>
> Hiveserver2 fail on Mapreduce local job fail. For example:
> {code}
> select /*+ MAPJOIN(v) */ registration from studenttab10k s join votertab10k v 
> on (s.name = v.name);
> {code}
> The root cause is "class not found" in the local hadoop job 
> (MapredLocalTask.execute). HADOOP_CLASSPATH does not include $HIVE_HOME/lib. 
> Set HADOOP_CLASSPATH correctly will fix the issue.
> However, there is one complexity in Windows. We start Hiveserver2 using 
> Windows service console (services.msc), which takes hiveserver2.xml generated 
> by hive.cmd. There is no way to pass environment variable in hiveserver2.xml 
> (weird but reality). I attach a patch which pass it through command line 
> arguments and relay to HADOOP_CLASSPATH in Hive code. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4844) Add varchar data type

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-4844:
-

Description: 
Add new varchar data types which have support for more SQL-compliant behavior, 
such as SQL string comparison semantics, max length, etc.
Char type will be added as another task.


  was:
Add new varchar data types which have support for more SQL-compliant behavior, 
such as SQL string comparison semantics, max length, etc.
Char type will be added as another task.

NO PRECOMMIT TESTS - now dependent on HIVE-5203/5204/5206


> Add varchar data type
> -
>
> Key: HIVE-4844
> URL: https://issues.apache.org/jira/browse/HIVE-4844
> Project: Hive
>  Issue Type: New Feature
>  Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-4844.10.patch, HIVE-4844.11.patch, 
> HIVE-4844.12.patch, HIVE-4844.13.patch, HIVE-4844.14.patch, 
> HIVE-4844.15.patch, HIVE-4844.16.patch, HIVE-4844.1.patch.hack, 
> HIVE-4844.2.patch, HIVE-4844.3.patch, HIVE-4844.4.patch, HIVE-4844.5.patch, 
> HIVE-4844.6.patch, HIVE-4844.7.patch, HIVE-4844.8.patch, HIVE-4844.9.patch, 
> HIVE-4844.D12699.1.patch, screenshot.png
>
>
> Add new varchar data types which have support for more SQL-compliant 
> behavior, such as SQL string comparison semantics, max length, etc.
> Char type will be added as another task.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5278) Move some string UDFs to GenericUDFs, for better varchar support

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-5278:
-

Status: Patch Available  (was: Open)

> Move some string UDFs to GenericUDFs, for better varchar support
> 
>
> Key: HIVE-5278
> URL: https://issues.apache.org/jira/browse/HIVE-5278
> Project: Hive
>  Issue Type: Improvement
>  Components: Types, UDF
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-5278.1.patch
>
>
> To better support varchar/char types in string UDFs, select UDFs should be 
> converted to GenericUDFs. This allows the UDF to return the resulting 
> char/varchar length in the type metadata.
> This work is being split off as a separate task from HIVE-4844. The initial 
> UDFs as part of this work are concat/lower/upper.
> NO PRECOMMIT TESTS - dependent on HIVE-4844

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5278) Move some string UDFs to GenericUDFs, for better varchar support

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-5278:
-

Attachment: HIVE-5278.1.patch

Attaching patch v1
Posted review at https://reviews.facebook.net/D12909

> Move some string UDFs to GenericUDFs, for better varchar support
> 
>
> Key: HIVE-5278
> URL: https://issues.apache.org/jira/browse/HIVE-5278
> Project: Hive
>  Issue Type: Improvement
>  Components: Types, UDF
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-5278.1.patch
>
>
> To better support varchar/char types in string UDFs, select UDFs should be 
> converted to GenericUDFs. This allows the UDF to return the resulting 
> char/varchar length in the type metadata.
> This work is being split off as a separate task from HIVE-4844. The initial 
> UDFs as part of this work are concat/lower/upper.
> NO PRECOMMIT TESTS - dependent on HIVE-4844

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4492) Revert HIVE-4322

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766214#comment-13766214
 ] 

Ashutosh Chauhan commented on HIVE-4492:


Yup. Please apply to 0.12

> Revert HIVE-4322
> 
>
> Key: HIVE-4492
> URL: https://issues.apache.org/jira/browse/HIVE-4492
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Thrift API
>Reporter: Samuel Yuan
>Assignee: Samuel Yuan
> Fix For: 0.13.0
>
> Attachments: HIVE-4492.1.patch.txt, HIVE-4492.patch
>
>
> See HIVE-4432 and HIVE-4433. It's possible to work around these issues but a 
> better solution is probably to roll back the "fix" and change the API to use 
> a primitive type as the map key (in a backwards-compatible manner).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4844) Add varchar data type

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766216#comment-13766216
 ] 

Phabricator commented on HIVE-4844:
---

jdere has commented on the revision "HIVE-4844 [jira] Add varchar data type".

  Most of the SerDe support has been broken out into HIVE-5161.  The SerDe's 
I'm testing there are RegexSerDe, LazyBinarySerDe, LazySimpleSerDe, 
ColumnarSerDe, LazyBinaryColumnarSerDe, OrcSerDe.  I'll need to take a look 
again, but if HBase/Avro SerDe's use reflection-based methods to get their 
Types/ObjectInspectors, then they won't be able to support varchars properly.

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java:629 will 
change in the next patch
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:160 will 
change in next patch
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:169 will 
change in next patch
  ql/src/java/org/apache/hadoop/hive/ql/udf/GenericUDFEncode.java:48 sure, will 
change
  ql/src/java/org/apache/hadoop/hive/ql/udf/GenericUDFEncode.java:56 will 
change this message too
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToVarchar.java:36 
whoops forgot to add that .. will do
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToVarchar.java:39-40
 Actually, my latest patch did mark these fields as transient .. I'll need to 
update this review with the updated patch.
  
serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyHiveVarcharObjectInspector.java:34
 Do we need to serialize object inspectors? my impression is that should always 
have been transient, at least when they're used in UDFs since they can be set 
during initialize().
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java:401
 Yeah, this comment is gone in the updated patch.
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java:474
 2011 SQL Standard (6.13 cast specification) does say that the value should be 
"TRUE"/"FALSE". This also matches what we're doing with the string type.
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantHiveVarcharObjectInspector.java:33
 ok, will change
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableHiveVarcharObjectInspector.java:35
 will change

REVISION DETAIL
  https://reviews.facebook.net/D12699

To: JIRA, jdere
Cc: ashutoshc


> Add varchar data type
> -
>
> Key: HIVE-4844
> URL: https://issues.apache.org/jira/browse/HIVE-4844
> Project: Hive
>  Issue Type: New Feature
>  Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-4844.10.patch, HIVE-4844.11.patch, 
> HIVE-4844.12.patch, HIVE-4844.13.patch, HIVE-4844.14.patch, 
> HIVE-4844.15.patch, HIVE-4844.16.patch, HIVE-4844.1.patch.hack, 
> HIVE-4844.2.patch, HIVE-4844.3.patch, HIVE-4844.4.patch, HIVE-4844.5.patch, 
> HIVE-4844.6.patch, HIVE-4844.7.patch, HIVE-4844.8.patch, HIVE-4844.9.patch, 
> HIVE-4844.D12699.1.patch, screenshot.png
>
>
> Add new varchar data types which have support for more SQL-compliant 
> behavior, such as SQL string comparison semantics, max length, etc.
> Char type will be added as another task.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5260) Introduce HivePassThroughOutputFormat that allows Hive to use general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765664#comment-13765664
 ] 

Hudson commented on HIVE-5260:
--

FAILURE: Integrated in Hive-trunk-hadoop1-ptest #163 (See 
[https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/163/])
HIVE-5260 : Introduce HivePassThroughOutputFormat that allows Hive to use 
general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers 
(Viraj Bhat via Ashutosh Chauhan) (hashutosh: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522574)
* 
/hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
* 
/hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
* /hive/trunk/hbase-handler/src/test/results/positive/external_table_ppd.q.out
* 
/hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java


> Introduce HivePassThroughOutputFormat that allows Hive to use general purpose 
> OutputFormats instead of HiveOutputFormats in StorageHandlers
> ---
>
> Key: HIVE-5260
> URL: https://issues.apache.org/jira/browse/HIVE-5260
> Project: Hive
>  Issue Type: Sub-task
>  Components: HBase Handler, HCatalog
>Affects Versions: 0.12.0
>Reporter: Sushanth Sowmyan
>Assignee: Viraj Bhat
> Fix For: 0.13.0
>
> Attachments: HIVE-5260.2.patch, HIVE-5260.patch
>
>
> This is a task being created to address the hive-side of HIVE-4331.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5261) Make the Hive HBase storage handler work from HCatalog, and use HiveStorageHandlers instead of HCatStorageHandlers

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765663#comment-13765663
 ] 

Hudson commented on HIVE-5261:
--

FAILURE: Integrated in Hive-trunk-hadoop1-ptest #163 (See 
[https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/163/])
HIVE-5261 : Make the Hive HBase storage handler work from HCatalog, and use 
HiveStorageHandlers instead of HCatStorageHandlers (Viraj Bhat via Sushanth 
Sowmyan) (hashutosh: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522583)
* /hive/trunk/hcatalog/build-support/ant/checkstyle.xml
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/mapreduce/HCatStorageHandler.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/CreateTableHook.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/DefaultRecordWriterContainer.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputFormatContainer.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileRecordWriterContainer.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatBaseInputFormat.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatBaseOutputFormat.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatRecordReader.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatStorageHandler.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/InitializeInput.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/PartInfo.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/security/StorageDelegationAuthorizationProvider.java
* 
/hive/trunk/hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/DummyStorageHandler.java
* /hive/trunk/hcatalog/storage-handlers/hbase/pom.xml
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseHCatStorageHandler.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/RevisionManager.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/SkeletonHBaseTest.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseInputFormat.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHCatHBaseInputFormat.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHiveHBaseStorageHandler.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHiveHBaseTableOutputFormat.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestPigHBaseStorageHandler.java


> Make the Hive HBase storage handler work from HCatalog, and use 
> HiveStorageHandlers instead of HCatStorageHandlers
> --
>
> Key: HIVE-5261
> URL: https://issues.apache.org/jira/browse/HIVE-5261
> Project: Hive
>  Issue Type: Sub-task
>  Components: HBase Handler, HCatalog
>Affects Versions: 0.12.0
>Reporter: Sushanth Sowmyan
>Assignee: Viraj Bhat
> Fix For: 0.13.0
>
> Attachments: HIVE-5261.patch
>
>
> This is a task being created for the HCat side of HIVE-4331

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [jira] [Created] (HIVE-5112) Upgrade protobuf to 2.5 from 2.4

[Edward Capriolo]

In conviencing current users to make developers happy of beta software
happy is not a big win. Protobuf 2.4 has been the standard for a long time
now. We and others have considerable amount of udfs that amd other things
that need to be rebuilt.

Forcing 12 to protobuf 2.5 will make it harder for me and others to
upgrade. Really this dep should all be external to hive. There is no reason
that a input or storage handler has to dictate the version of things.

I do not see what the problem of waitng 3 month is. On one hand we are
going to run 2x tests of hcat for 2 releases to guarentee comparability.

On the otherhand we are willimlng to force people to rebuild the kits to be
in sync with non release software,
On Thursday, September 12, 2013, Sushanth Sowmyan (JIRA) 
wrote:
>
> [
https://issues.apache.org/jira/browse/HIVE-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765916#comment-13765916]
>
> Sushanth Sowmyan commented on HIVE-5112:
> 
>
> I'd also much rather have this in 0.12. Without this in 0.12, we either
block HIVE-4388 inclusion in 0.12, or require two versions of HIVE-4388,
one with protobuf 2.4 and one with 2.5. And that jira blocks being able to
work with hbase 0.96, which is in an RC stage, and past beta.
>
> The term beta, imo, is not so much about experimentation nowadays as it
is about being a preview.
>
>> Upgrade protobuf to 2.5 from 2.4
>> 
>>
>> Key: HIVE-5112
>> URL: https://issues.apache.org/jira/browse/HIVE-5112
>> Project: Hive
>>  Issue Type: Improvement
>>Reporter: Brock Noland
>>Assignee: Owen O'Malley
>> Fix For: 0.13.0
>>
>> Attachments: HIVE-5112.2.patch, HIVE-5112.D12429.1.patch
>>
>>
>> Hadoop and Hbase have both upgraded protobuf. We should as well.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>

[jira] [Created] (HIVE-5289) PTest2 should disable checking of libraries during batch exec

[Brock Noland (JIRA)]

Brock Noland created HIVE-5289:
--

 Summary: PTest2 should disable checking of libraries during batch 
exec
 Key: HIVE-5289
 URL: https://issues.apache.org/jira/browse/HIVE-5289
 Project: Hive
  Issue Type: Test
Reporter: Brock Noland
Assignee: Brock Noland


PTest2 has two phases:

1) Build from source
2) Exec in parallel

during phase two we don't want ivy making HTTP requests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5289) PTest2 should disable checking of libraries during batch exec

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-5289:
---

Attachment: HIVE-5289.patch

The attached patch improved execution time quite a bit for traditional unit 
tests.

> PTest2 should disable checking of libraries during batch exec
> -
>
> Key: HIVE-5289
> URL: https://issues.apache.org/jira/browse/HIVE-5289
> Project: Hive
>  Issue Type: Test
>Reporter: Brock Noland
>Assignee: Brock Noland
> Attachments: HIVE-5289.patch
>
>
> PTest2 has two phases:
> 1) Build from source
> 2) Exec in parallel
> during phase two we don't want ivy making HTTP requests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5289) PTest2 should disable checking of libraries during batch exec

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-5289:
---

Fix Version/s: 0.13.0
   Status: Patch Available  (was: Open)

> PTest2 should disable checking of libraries during batch exec
> -
>
> Key: HIVE-5289
> URL: https://issues.apache.org/jira/browse/HIVE-5289
> Project: Hive
>  Issue Type: Test
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.13.0
>
> Attachments: HIVE-5289.patch
>
>
> PTest2 has two phases:
> 1) Build from source
> 2) Exec in parallel
> during phase two we don't want ivy making HTTP requests.
> NO PRECOMMIT TESTS

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5289) PTest2 should disable checking of libraries during batch exec

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-5289:
---

Description: 
PTest2 has two phases:

1) Build from source
2) Exec in parallel

during phase two we don't want ivy making HTTP requests.

NO PRECOMMIT TESTS

  was:
PTest2 has two phases:

1) Build from source
2) Exec in parallel

during phase two we don't want ivy making HTTP requests.


> PTest2 should disable checking of libraries during batch exec
> -
>
> Key: HIVE-5289
> URL: https://issues.apache.org/jira/browse/HIVE-5289
> Project: Hive
>  Issue Type: Test
>Reporter: Brock Noland
>Assignee: Brock Noland
> Attachments: HIVE-5289.patch
>
>
> PTest2 has two phases:
> 1) Build from source
> 2) Exec in parallel
> during phase two we don't want ivy making HTTP requests.
> NO PRECOMMIT TESTS

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5127) Upgrade xerces and xalan for WebHCat

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766241#comment-13766241
 ] 

Eugene Koifman commented on HIVE-5127:
--

no, the patch is fine.  There was a block of text commented out.  Now it just 
adds a latest version of Xerces as a dependency.

> Upgrade xerces and xalan for WebHCat
> 
>
> Key: HIVE-5127
> URL: https://issues.apache.org/jira/browse/HIVE-5127
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.12.0
>
> Attachments: HIVE-5127.patch
>
>
> Currently webhcat log files are full of exceptions like this, which obscures 
> the real output and may cause perf issues.
> Upgrading to more recent versions of xerces/xalan fixes this.
> Add the following to hive/hcatalog/webhcat/svr/pom.xml
> 
> xerces
> xercesImpl
> 2.9.1
> 
> 
> xalan
> xalan
> 2.7.1
> 
> 13/08/20 16:54:04 ERROR conf.Configuration: Failed to set 
> setXIncludeAware(true) for parser 
> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl@48dbb335:java.lang.UnsupportedOperationException:
>  This parser does not support specification "null" version "null"
> java.lang.UnsupportedOperationException: This parser does not support 
> specification "null" version "null"
> at 
> javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
> at 
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1892)
> at 
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1861)
> at 
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1778)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:870)
> at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:171)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:305)
> at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:288)
> at 
> org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:383)
> at 
> org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:281)
> at 
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:422)
> at 
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:168)
> at 
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:151)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at 
> org.apache.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:99)
> at 
> org.apache.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:95)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
> at 
> org.apache.hcatalog.templeton.LauncherDelegator.queueAsUser(LauncherDelegator.java:95)
> at 
> org.apache.hcatalog.templeton.LauncherDelegator.enqueueController(LauncherDelegator.java:77)
> at 
> org.apache.hcatalog.templeton.JarDelegator.run(JarDelegator.java:52)
> at 
> org.apache.hcatalog.templeton.StreamingDelegator.run(StreamingDelegator.java:53)
> at 
> org.apache.hcatalog.templeton.Server.mapReduceStreaming(Server.java:596)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at 
> com.sun.jersey.

[jira] [Commented] (HIVE-5260) Introduce HivePassThroughOutputFormat that allows Hive to use general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers

[Viraj Bhat (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766243#comment-13766243
 ] 

Viraj Bhat commented on HIVE-5260:
--

Sushanth I think the patch to apply to Hive 0.12 as well. I would be good to 
have this in Hive 0.12

> Introduce HivePassThroughOutputFormat that allows Hive to use general purpose 
> OutputFormats instead of HiveOutputFormats in StorageHandlers
> ---
>
> Key: HIVE-5260
> URL: https://issues.apache.org/jira/browse/HIVE-5260
> Project: Hive
>  Issue Type: Sub-task
>  Components: HBase Handler, HCatalog
>Affects Versions: 0.12.0
>Reporter: Sushanth Sowmyan
>Assignee: Viraj Bhat
> Fix For: 0.13.0
>
> Attachments: HIVE-5260.2.patch, HIVE-5260.patch
>
>
> This is a task being created to address the hive-side of HIVE-4331.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4388) HBase tests fail against Hadoop 2

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765900#comment-13765900
 ] 

Sushanth Sowmyan commented on HIVE-4388:


Brock,

HBASE-9477 you link to seems to have been closed. Are you working on respinning 
this? I would like for this patch to be part of 0.12 so that we have a "more 
seamless" hbase-hadoop2 integration in 0.12. I'm willing to help with the 
rebase if you need.

> HBase tests fail against Hadoop 2
> -
>
> Key: HIVE-4388
> URL: https://issues.apache.org/jira/browse/HIVE-4388
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Gunther Hagleitner
>Assignee: Brock Noland
> Attachments: HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, 
> HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, 
> HIVE-4388.patch, HIVE-4388-wip.txt
>
>
> Currently we're building by default against 0.92. When you run against hadoop 
> 2 (-Dhadoop.mr.rev=23) builds fail because of: HBASE-5963.
> HIVE-3861 upgrades the version of hbase used. This will get you past the 
> problem in HBASE-5963 (which was fixed in 0.94.1) but fails with: HBASE-6396.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5253) Create component to compile and jar dynamic code

[Edward Capriolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Capriolo updated HIVE-5253:
--

Attachment: HIVE-5253.patch.txt

> Create component to compile and jar dynamic code
> 
>
> Key: HIVE-5253
> URL: https://issues.apache.org/jira/browse/HIVE-5253
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Edward Capriolo
> Attachments: HIVE-5253.patch.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5253) Create component to compile and jar dynamic code

[Edward Capriolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Capriolo updated HIVE-5253:
--

Attachment: (was: cli_driver.diff)

> Create component to compile and jar dynamic code
> 
>
> Key: HIVE-5253
> URL: https://issues.apache.org/jira/browse/HIVE-5253
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Edward Capriolo
> Attachments: HIVE-5253.patch.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5253) Create component to compile and jar dynamic code

[Edward Capriolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Capriolo updated HIVE-5253:
--

Attachment: cli_driver.diff

> Create component to compile and jar dynamic code
> 
>
> Key: HIVE-5253
> URL: https://issues.apache.org/jira/browse/HIVE-5253
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Edward Capriolo
> Attachments: HIVE-5253.patch.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5253) Create component to compile and jar dynamic code

[Edward Capriolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Capriolo updated HIVE-5253:
--

Assignee: Edward Capriolo
  Status: Patch Available  (was: Open)

> Create component to compile and jar dynamic code
> 
>
> Key: HIVE-5253
> URL: https://issues.apache.org/jira/browse/HIVE-5253
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Edward Capriolo
>Assignee: Edward Capriolo
> Attachments: HIVE-5253.patch.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5112) Upgrade protobuf to 2.5 from 2.4

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766251#comment-13766251
 ] 

Sushanth Sowmyan commented on HIVE-5112:



Compiled code for protobuf is supposed to be backward-compatible across 
protobuf releases(old classes with new protobuf jar) . The only case where 
protobuf backward compatibility is supposed to bite you is when you have 
generated code with an old protobuf jar that you're trying to compile with a 
new protobuf jar. It's supposed to cause compile-time hassles, not runtime 
hassles.

So, I wanted to see if it was a difficult task upgrading hive-protobuf to a 
newer protobuf as well and so, tried to check out hive-protobuf to test to see 
if it was a simple update to 2.5 for that. I see only one .proto file, and it's 
used by the tests. I was able to generate new code for it that compiles, but I 
get test failures all over from the version of hive_test that I compiled 
(return code 9 and all that)

That got me thinking - if the problem comes from a "protobuf 2.5 is not 
production-certified" stance, then shouldn't hive-0.12 also not match 
production-certification?



> Upgrade protobuf to 2.5 from 2.4
> 
>
> Key: HIVE-5112
> URL: https://issues.apache.org/jira/browse/HIVE-5112
> Project: Hive
>  Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Owen O'Malley
> Fix For: 0.13.0
>
> Attachments: HIVE-5112.2.patch, HIVE-5112.D12429.1.patch
>
>
> Hadoop and Hbase have both upgraded protobuf. We should as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-5290) Some HCatalog tests have been behaving flaky

[Brock Noland (JIRA)]

Brock Noland created HIVE-5290:
--

 Summary: Some HCatalog tests have been behaving flaky
 Key: HIVE-5290
 URL: https://issues.apache.org/jira/browse/HIVE-5290
 Project: Hive
  Issue Type: Test
Affects Versions: 0.13.0
Reporter: Brock Noland
Assignee: Brock Noland




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5066) [WebHCat] Other code fixes for Windows

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-5066:
-

Status: Patch Available  (was: Open)

> [WebHCat] Other code fixes for Windows
> --
>
> Key: HIVE-5066
> URL: https://issues.apache.org/jira/browse/HIVE-5066
> Project: Hive
>  Issue Type: Sub-task
>  Components: HCatalog
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-5034-1.patch, HIVE-5066-2.patch, HIVE-5066-3.patch
>
>
> This is equivalent to HCATALOG-526, but updated to sync with latest trunk.
> NO PRECOMMIT TESTS

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5066) [WebHCat] Other code fixes for Windows

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-5066:
-

Description: 
This is equivalent to HCATALOG-526, but updated to sync with latest trunk.
NO PRECOMMIT TESTS

  was:This is equivalent to HCATALOG-526, but updated to sync with latest trunk.


> [WebHCat] Other code fixes for Windows
> --
>
> Key: HIVE-5066
> URL: https://issues.apache.org/jira/browse/HIVE-5066
> Project: Hive
>  Issue Type: Sub-task
>  Components: HCatalog
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-5034-1.patch, HIVE-5066-2.patch, HIVE-5066-3.patch
>
>
> This is equivalent to HCATALOG-526, but updated to sync with latest trunk.
> NO PRECOMMIT TESTS

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4844) Add varchar data type

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4844:
--

Attachment: HIVE-4844.D12891.1.patch

jdere requested code review of "HIVE-4844 [jira] Add varchar data type".

Reviewers: JIRA

Changes to HIVE-4844 per review from ashutosh

Add new varchar data types which have support for more SQL-compliant behavior, 
such as SQL string comparison semantics, max length, etc.
Char type will be added as another task.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D12891

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/common/type/HiveBaseChar.java
  common/src/java/org/apache/hadoop/hive/common/type/HiveVarchar.java
  common/src/test/org/apache/hadoop/hive/common/type/TestHiveVarchar.java
  data/files/datatypes.txt
  data/files/vc1.txt
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/GenericUDFEncode.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseCompare.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcatWS.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToDate.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToVarchar.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtils.java
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFunctionRegistry.java
  ql/src/test/queries/clientnegative/invalid_varchar_length_1.q
  ql/src/test/queries/clientnegative/invalid_varchar_length_2.q
  ql/src/test/queries/clientnegative/invalid_varchar_length_3.q
  ql/src/test/queries/clientpositive/alter_varchar1.q
  ql/src/test/queries/clientpositive/ctas_varchar.q
  ql/src/test/queries/clientpositive/partition_varchar1.q
  ql/src/test/queries/clientpositive/varchar_1.q
  ql/src/test/queries/clientpositive/varchar_2.q
  ql/src/test/queries/clientpositive/varchar_cast.q
  ql/src/test/queries/clientpositive/varchar_comparison.q
  ql/src/test/queries/clientpositive/varchar_join1.q
  ql/src/test/queries/clientpositive/varchar_nested_types.q
  ql/src/test/queries/clientpositive/varchar_udf1.q
  ql/src/test/queries/clientpositive/varchar_union1.q
  ql/src/test/results/clientnegative/invalid_varchar_length_1.q.out
  ql/src/test/results/clientnegative/invalid_varchar_length_2.q.out
  ql/src/test/results/clientnegative/invalid_varchar_length_3.q.out
  ql/src/test/results/clientpositive/alter_varchar1.q.out
  ql/src/test/results/clientpositive/ctas_varchar.q.out
  ql/src/test/results/clientpositive/partition_varchar1.q.out
  ql/src/test/results/clientpositive/varchar_1.q.out
  ql/src/test/results/clientpositive/varchar_2.q.out
  ql/src/test/results/clientpositive/varchar_cast.q.out
  ql/src/test/results/clientpositive/varchar_comparison.q.out
  ql/src/test/results/clientpositive/varchar_join1.q.out
  ql/src/test/results/clientpositive/varchar_nested_types.q.out
  ql/src/test/results/clientpositive/varchar_udf1.q.out
  ql/src/test/results/clientpositive/varchar_union1.q.out
  serde/if/serde.thrift
  serde/src/gen/thrift/gen-cpp/serde_constants.cpp
  serde/src/gen/thrift/gen-cpp/serde_constants.h
  
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java
  
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java
  
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java
  
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java
  serde/src/gen/thrift/gen-php/org/apache/hadoop/hive/serde/Types.php
  serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py
  serde/src/gen/thrift/gen-rb/serde_constants.rb
  serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java
  
serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java
  
serde/src/jav

[jira] [Updated] (HIVE-5290) Some HCatalog tests have been behaving flaky

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-5290:
---

Attachment: HIVE-5290.patch

Attached patch makes the TestHCatLoader* and TestSequenceFileReadWrite 
significantly more reliable for me.

> Some HCatalog tests have been behaving flaky
> 
>
> Key: HIVE-5290
> URL: https://issues.apache.org/jira/browse/HIVE-5290
> Project: Hive
>  Issue Type: Test
>Affects Versions: 0.13.0
>Reporter: Brock Noland
>Assignee: Brock Noland
> Attachments: HIVE-5290.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5290) Some HCatalog tests have been behaving flaky

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-5290:
---

Status: Patch Available  (was: Open)

> Some HCatalog tests have been behaving flaky
> 
>
> Key: HIVE-5290
> URL: https://issues.apache.org/jira/browse/HIVE-5290
> Project: Hive
>  Issue Type: Test
>Affects Versions: 0.13.0
>Reporter: Brock Noland
>Assignee: Brock Noland
> Attachments: HIVE-5290.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request 14113: support standard UDFs and generic UDFs in vectorized mode with an adaptor

[Brock Noland]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14113/#review26071
---


Hey, 

First off let me say you've don't a massive job in the vectorization branch! 
Nice work! I obviously don't have a enough context to do a real review. 
However, I just noticed you guys wanted to merge this branch soon so I took a 
peak at this issue. There is a large amount of trailing whitespace in this 
patch. Can you remove that? It shows up in red on RB.


ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java


Does something require ArrayList to be on the LHS?


- Brock Noland


On Sept. 12, 2013, 11:57 p.m., Eric Hanson wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14113/
> ---
> 
> (Updated Sept. 12, 2013, 11:57 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-4961
> https://issues.apache.org/jira/browse/HIVE-4961
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Added support standard UDFs and generic UDFs in vectorized mode with an 
> adaptor. Includes unit tests.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
> 3d8ade6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorUDFAdaptor.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorUDFArgDesc.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 5b467bb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 048824a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 
> 52fe47b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/GenericUDFIsNull.java
>  PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/LongUDF.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestUDF.java 
> PRE-CREATION 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorUDFAdaptor.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/14113/diff/
> 
> 
> Testing
> ---
> 
> Also did extensive ad-hoc end-to end tests with a variety of UDFs.
> 
> 
> Thanks,
> 
> Eric Hanson
> 
>

[jira] [Commented] (HIVE-5251) Change the CLI's poor escapting for semi colon.

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766264#comment-13766264
 ] 

Edward Capriolo commented on HIVE-5251:
---

Cli escapes are really bad. I know I am half responsible for the current logic.

{quote}
hive> set x=' \' ';
hive> set x;
x=' ' '
hive> compile 
> ` public class x { }` as groovy named x.groovy;
Added /tmp/0.jar to class path
Added resource: /tmp/0.jar
{quote}

This patch should gives us better escaping.

> Change the CLI's poor escapting for semi colon.
> ---
>
> Key: HIVE-5251
> URL: https://issues.apache.org/jira/browse/HIVE-5251
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Edward Capriolo
>Assignee: Edward Capriolo
>
> Currently the CLI has rather flaky ; handling. When specifying inline code 
> this character is important and we should fix the cli so ; escaping is better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5251) Change the CLI's poor escapting for semi colon.

[Edward Capriolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Capriolo updated HIVE-5251:
--

Status: Patch Available  (was: Open)

> Change the CLI's poor escapting for semi colon.
> ---
>
> Key: HIVE-5251
> URL: https://issues.apache.org/jira/browse/HIVE-5251
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Edward Capriolo
>Assignee: Edward Capriolo
> Attachments: HIVE-5251.patch.txt
>
>
> Currently the CLI has rather flaky ; handling. When specifying inline code 
> this character is important and we should fix the cli so ; escaping is better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5251) Change the CLI's poor escapting for semi colon.

[Edward Capriolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Capriolo updated HIVE-5251:
--

Attachment: HIVE-5251.patch.txt

> Change the CLI's poor escapting for semi colon.
> ---
>
> Key: HIVE-5251
> URL: https://issues.apache.org/jira/browse/HIVE-5251
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Edward Capriolo
>Assignee: Edward Capriolo
> Attachments: HIVE-5251.patch.txt
>
>
> Currently the CLI has rather flaky ; handling. When specifying inline code 
> this character is important and we should fix the cli so ; escaping is better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5251) Change the CLI's poor escapting for semi colon.

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766266#comment-13766266
 ] 

Edward Capriolo commented on HIVE-5251:
---

Let's see what the patch breaks. Even if it does break some test cases I think 
we should move  forward.

> Change the CLI's poor escapting for semi colon.
> ---
>
> Key: HIVE-5251
> URL: https://issues.apache.org/jira/browse/HIVE-5251
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Edward Capriolo
>Assignee: Edward Capriolo
> Attachments: HIVE-5251.patch.txt
>
>
> Currently the CLI has rather flaky ; handling. When specifying inline code 
> this character is important and we should fix the cli so ; escaping is better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4844) Add varchar data type

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766271#comment-13766271
 ] 

Phabricator commented on HIVE-4844:
---

ashutoshc has accepted the revision "HIVE-4844 [jira] Add varchar data type".

  +1 Thanks, Jason for making changes.

REVISION DETAIL
  https://reviews.facebook.net/D12891

BRANCH
  HIVE-4844.2

ARCANIST PROJECT
  hive

To: JIRA, ashutoshc, jdere


> Add varchar data type
> -
>
> Key: HIVE-4844
> URL: https://issues.apache.org/jira/browse/HIVE-4844
> Project: Hive
>  Issue Type: New Feature
>  Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-4844.10.patch, HIVE-4844.11.patch, 
> HIVE-4844.12.patch, HIVE-4844.13.patch, HIVE-4844.14.patch, 
> HIVE-4844.15.patch, HIVE-4844.16.patch, HIVE-4844.1.patch.hack, 
> HIVE-4844.2.patch, HIVE-4844.3.patch, HIVE-4844.4.patch, HIVE-4844.5.patch, 
> HIVE-4844.6.patch, HIVE-4844.7.patch, HIVE-4844.8.patch, HIVE-4844.9.patch, 
> HIVE-4844.D12699.1.patch, HIVE-4844.D12891.1.patch, screenshot.png
>
>
> Add new varchar data types which have support for more SQL-compliant 
> behavior, such as SQL string comparison semantics, max length, etc.
> Char type will be added as another task.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5127) Upgrade xerces and xalan for WebHCat

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766288#comment-13766288
 ] 

Thejas M Nair commented on HIVE-5127:
-

Got it now. I failed to notice the comment being removed!
+1 Will be useful to have this in 0.12 as well.


> Upgrade xerces and xalan for WebHCat
> 
>
> Key: HIVE-5127
> URL: https://issues.apache.org/jira/browse/HIVE-5127
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.12.0
>
> Attachments: HIVE-5127.patch
>
>
> Currently webhcat log files are full of exceptions like this, which obscures 
> the real output and may cause perf issues.
> Upgrading to more recent versions of xerces/xalan fixes this.
> Add the following to hive/hcatalog/webhcat/svr/pom.xml
> 
> xerces
> xercesImpl
> 2.9.1
> 
> 
> xalan
> xalan
> 2.7.1
> 
> 13/08/20 16:54:04 ERROR conf.Configuration: Failed to set 
> setXIncludeAware(true) for parser 
> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl@48dbb335:java.lang.UnsupportedOperationException:
>  This parser does not support specification "null" version "null"
> java.lang.UnsupportedOperationException: This parser does not support 
> specification "null" version "null"
> at 
> javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
> at 
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1892)
> at 
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1861)
> at 
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1778)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:870)
> at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:171)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:305)
> at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:288)
> at 
> org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:383)
> at 
> org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:281)
> at 
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:422)
> at 
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:168)
> at 
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:151)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at 
> org.apache.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:99)
> at 
> org.apache.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:95)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
> at 
> org.apache.hcatalog.templeton.LauncherDelegator.queueAsUser(LauncherDelegator.java:95)
> at 
> org.apache.hcatalog.templeton.LauncherDelegator.enqueueController(LauncherDelegator.java:77)
> at 
> org.apache.hcatalog.templeton.JarDelegator.run(JarDelegator.java:52)
> at 
> org.apache.hcatalog.templeton.StreamingDelegator.run(StreamingDelegator.java:53)
> at 
> org.apache.hcatalog.templeton.Server.mapReduceStreaming(Server.java:596)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at 
> com.sun.jersey.server.impl.uri.rules.R

[jira] [Updated] (HIVE-5161) Additional SerDe support for varchar type

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5161:
--

Attachment: D12897.1.patch

jdere requested code review of "HIVE-5161 [jira] Additional SerDe support for 
varchar type".

Reviewers: JIRA

HIVE-5161: Additional SerDe support for varchar type

Breaking out support for varchar for the various SerDes as an additional task.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D12897

AFFECTED FILES
  ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ColumnStatisticsImpl.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcStruct.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java
  ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto
  ql/src/test/queries/clientpositive/varchar_serde.q
  ql/src/test/results/clientpositive/varchar_serde.q.out
  serde/src/java/org/apache/hadoop/hive/serde2/RegexSerDe.java
  
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java
  serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/30867/

To: JIRA, jdere


> Additional SerDe support for varchar type
> -
>
> Key: HIVE-5161
> URL: https://issues.apache.org/jira/browse/HIVE-5161
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers, Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: D12897.1.patch, HIVE-5161.1.patch, HIVE-5161.2.patch, 
> HIVE-5161.3.patch
>
>
> Breaking out support for varchar for the various SerDes as an additional task.
> NO PRECOMMIT TESTS - can't run tests until HIVE-4844 is committed

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request 10180: HIVE-4171

[Prasad Mujumdar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10180/#review26073
---


LGTM

- Prasad Mujumdar


On Sept. 12, 2013, 3:08 a.m., Thejas Nair wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10180/
> ---
> 
> (Updated Sept. 12, 2013, 3:08 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Navis Ryu, and Prasad Mujumdar.
> 
> 
> Bugs: HIVE-4171
> https://issues.apache.org/jira/browse/HIVE-4171
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-4171
> 
> 
> Diffs
> -
> 
>   cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java aeced48 
>   cli/src/java/org/apache/hadoop/hive/cli/CliSessionState.java d11b873 
>   cli/src/test/org/apache/hadoop/hive/cli/TestCliSessionState.java a95e206 
>   
> hcatalog/core/src/main/java/org/apache/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzer.java
>  0e011e4 
>   
> hcatalog/core/src/main/java/org/apache/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzerBase.java
>  3740795 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzer.java
>  75837ef 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzerBase.java
>  c144135 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 0bc4425 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 9de7dcf 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 4dcb260 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 344c9b1 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 932ec66 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
> 3413f51 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 6d50109 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6888504 
>   ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java d318f71 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 9588061 
>   ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java 8b8c276 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 4ee5267 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 
> 81a3396 
>   ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/10180/diff/
> 
> 
> Testing
> ---
> 
> Manually tested, unit test to be added.
> 
> 
> Thanks,
> 
> Thejas Nair
> 
>

[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766303#comment-13766303
 ] 

Carl Steinbach commented on HIVE-5283:
--

[~jnp] Can you please post a review request on phabricator or reviewboard?

Also, is the plan to commit this patch or to merge the vectorization branch 
into trunk?


> Merge vectorization branch to trunk
> ---
>
> Key: HIVE-5283
> URL: https://issues.apache.org/jira/browse/HIVE-5283
> Project: Hive
>  Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch
>
>
> The purpose of this jira is to upload vectorization patch, run tests etc. The 
> actual work will continue under HIVE-4160 umbrella jira.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3764) Support metastore version consistency check

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-3764:
--

Description: 
Today there's no version/compatibility information stored in hive metastore. 
Also the datanucleus configuration property to automatically create missing 
tables is enabled by default. If you happen to start an older or newer hive or 
don't run the correct upgrade scripts during migration, the metastore would end 
up corrupted. The autoCreate schema is not always sufficient to upgrade 
metastore when migrating to newer release. It's not supported with all 
databases. Besides the migration often involves altering existing table, 
changing or moving data etc.

Hence it's very useful to have some consistency check to make sure that hive is 
using correct metastore and for production systems the schema is not 
automatically by running hive.

Besides it would be helpful to add a tool that can leverage this version 
information to figure out the required set of upgrade scripts, and execute 
those against the configured metastore. Now that Hive includes Beeline client, 
it can be used to execute the scripts.


  was:
Today there's no version/compatibility information stored in hive metastore. 
Also the datanucleus configuration property to automatically create missing 
tables is enabled by default. If you happen to start an older or newer hive or 
don't run the correct upgrade scripts during migration, the metastore would end 
up corrupted. The autoCreate schema is not always sufficient to upgrade 
metastore when migrating to newer release. It's not supported with all 
databases. Besides the migration often involves altering existing table, 
changing or moving data etc.

Hence it's very useful to have some consistency check to make sure that hive is 
using correct metastore and for production systems the schema is not 
automatically by running hive.



> Support metastore version consistency check
> ---
>
> Key: HIVE-3764
> URL: https://issues.apache.org/jira/browse/HIVE-3764
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
>
> Today there's no version/compatibility information stored in hive metastore. 
> Also the datanucleus configuration property to automatically create missing 
> tables is enabled by default. If you happen to start an older or newer hive 
> or don't run the correct upgrade scripts during migration, the metastore 
> would end up corrupted. The autoCreate schema is not always sufficient to 
> upgrade metastore when migrating to newer release. It's not supported with 
> all databases. Besides the migration often involves altering existing table, 
> changing or moving data etc.
> Hence it's very useful to have some consistency check to make sure that hive 
> is using correct metastore and for production systems the schema is not 
> automatically by running hive.
> Besides it would be helpful to add a tool that can leverage this version 
> information to figure out the required set of upgrade scripts, and execute 
> those against the configured metastore. Now that Hive includes Beeline 
> client, it can be used to execute the scripts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Review Request 14120: HIVE-3764: Support metastore version consistency check

[Prasad Mujumdar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14120/
---

Review request for hive.


Bugs: HIVE-3764
https://issues.apache.org/jira/browse/HIVE-3764


Repository: hive-git


Description
---

- Added a new table in the metastore schema to store the Hive version in the 
metastore.
- Metastore handler compare the version stored in the schema with its own 
version. If there's a mismatch, then it can either record the correct version 
or raise error. The behavior is configurable via a new Hive config. This config 
when set, also restrict dataNucleus to auto upgrade the schema.
- The new schema creation and upgrade scripts record the new version in the 
metastore version table.
- Added 0.12 upgrade scripts for all supported DBs to creates the new table 
version tables in 0.12 metastore schema
- Added a new schemaTool that can perform new schema initialization or upgrade 
based on the schema version and product version.


Diffs
-

  beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java PRE-CREATION 
  beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java PRE-CREATION 
  beeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java 
PRE-CREATION 
  bin/ext/schemaTool.sh PRE-CREATION 
  bin/schematool PRE-CREATION 
  build-common.xml ad5ac23 
  build.xml 3e87163 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 22149e4 
  conf/hive-default.xml.template 9a3fc1d 
  metastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.12.0.derby.sql cce544f 
  metastore/scripts/upgrade/derby/upgrade-0.10.0-to-0.11.0.derby.sql cae7936 
  metastore/scripts/upgrade/derby/upgrade-0.11.0-to-0.12.0.derby.sql 492cc93 
  metastore/scripts/upgrade/derby/upgrade.order.derby PRE-CREATION 
  metastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.12.0.mysql.sql 22a77fe 
  metastore/scripts/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql 375a05f 
  metastore/scripts/upgrade/mysql/upgrade.order.mysql PRE-CREATION 
  metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql PRE-CREATION 
  metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql 85a0178 
  metastore/scripts/upgrade/oracle/upgrade-0.10.0-to-0.11.0.mysql.sql 
PRE-CREATION 
  metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql a2d0901 
  metastore/scripts/upgrade/oracle/upgrade.order.oracle PRE-CREATION 
  metastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql PRE-CREATION 
  metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql 7b319ba 
  metastore/scripts/upgrade/postgres/upgrade-0.11.0-to-0.12.0.postgres.sql 
9da0a1b 
  metastore/scripts/upgrade/postgres/upgrade.order.postgres PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
39dda92 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java a08c728 
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e410c3a 
  metastore/src/model/org/apache/hadoop/hive/metastore/model/MVersionTable.java 
PRE-CREATION 
  metastore/src/model/package.jdo c42b5b0 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 8066784 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 0f9b16c 
  metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/14120/diff/


Testing
---

Added new tests for schema verification and schemaTool.


Thanks,

Prasad Mujumdar

[jira] [Updated] (HIVE-3764) Support metastore version consistency check

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-3764:
--

Affects Version/s: 0.8.0
   0.9.0
   0.10.0
   0.11.0
Fix Version/s: 0.12.0

> Support metastore version consistency check
> ---
>
> Key: HIVE-3764
> URL: https://issues.apache.org/jira/browse/HIVE-3764
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.8.0, 0.9.0, 0.10.0, 0.11.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Fix For: 0.12.0
>
>
> Today there's no version/compatibility information stored in hive metastore. 
> Also the datanucleus configuration property to automatically create missing 
> tables is enabled by default. If you happen to start an older or newer hive 
> or don't run the correct upgrade scripts during migration, the metastore 
> would end up corrupted. The autoCreate schema is not always sufficient to 
> upgrade metastore when migrating to newer release. It's not supported with 
> all databases. Besides the migration often involves altering existing table, 
> changing or moving data etc.
> Hence it's very useful to have some consistency check to make sure that hive 
> is using correct metastore and for production systems the schema is not 
> automatically by running hive.
> Besides it would be helpful to add a tool that can leverage this version 
> information to figure out the required set of upgrade scripts, and execute 
> those against the configured metastore. Now that Hive includes Beeline 
> client, it can be used to execute the scripts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3764) Support metastore version consistency check

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-3764:
--

Status: Open  (was: Patch Available)

> Support metastore version consistency check
> ---
>
> Key: HIVE-3764
> URL: https://issues.apache.org/jira/browse/HIVE-3764
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
>
> Today there's no version/compatibility information stored in hive metastore. 
> Also the datanucleus configuration property to automatically create missing 
> tables is enabled by default. If you happen to start an older or newer hive 
> or don't run the correct upgrade scripts during migration, the metastore 
> would end up corrupted. The autoCreate schema is not always sufficient to 
> upgrade metastore when migrating to newer release. It's not supported with 
> all databases. Besides the migration often involves altering existing table, 
> changing or moving data etc.
> Hence it's very useful to have some consistency check to make sure that hive 
> is using correct metastore and for production systems the schema is not 
> automatically by running hive.
> Besides it would be helpful to add a tool that can leverage this version 
> information to figure out the required set of upgrade scripts, and execute 
> those against the configured metastore. Now that Hive includes Beeline 
> client, it can be used to execute the scripts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3764) Support metastore version consistency check

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-3764:
--

Attachment: HIVE-3764-0.13-addional-file.patch

Additional files for trunk (0.13)

> Support metastore version consistency check
> ---
>
> Key: HIVE-3764
> URL: https://issues.apache.org/jira/browse/HIVE-3764
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.8.0, 0.9.0, 0.10.0, 0.11.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Fix For: 0.12.0
>
> Attachments: HIVE-3764-0.13-addional-file.patch, HIVE-3764.4.patch
>
>
> Today there's no version/compatibility information stored in hive metastore. 
> Also the datanucleus configuration property to automatically create missing 
> tables is enabled by default. If you happen to start an older or newer hive 
> or don't run the correct upgrade scripts during migration, the metastore 
> would end up corrupted. The autoCreate schema is not always sufficient to 
> upgrade metastore when migrating to newer release. It's not supported with 
> all databases. Besides the migration often involves altering existing table, 
> changing or moving data etc.
> Hence it's very useful to have some consistency check to make sure that hive 
> is using correct metastore and for production systems the schema is not 
> automatically by running hive.
> Besides it would be helpful to add a tool that can leverage this version 
> information to figure out the required set of upgrade scripts, and execute 
> those against the configured metastore. Now that Hive includes Beeline 
> client, it can be used to execute the scripts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5261) Make the Hive HBase storage handler work from HCatalog, and use HiveStorageHandlers instead of HCatStorageHandlers

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766334#comment-13766334
 ] 

Hudson commented on HIVE-5261:
--

ABORTED: Integrated in Hive-trunk-hadoop2 #425 (See 
[https://builds.apache.org/job/Hive-trunk-hadoop2/425/])
HIVE-5261 : Make the Hive HBase storage handler work from HCatalog, and use 
HiveStorageHandlers instead of HCatStorageHandlers (Viraj Bhat via Sushanth 
Sowmyan) (hashutosh: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522583)
* /hive/trunk/hcatalog/build-support/ant/checkstyle.xml
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/mapreduce/HCatStorageHandler.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/CreateTableHook.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/DefaultRecordWriterContainer.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputFormatContainer.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileRecordWriterContainer.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatBaseInputFormat.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatBaseOutputFormat.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatRecordReader.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatStorageHandler.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/InitializeInput.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/PartInfo.java
* 
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/security/StorageDelegationAuthorizationProvider.java
* 
/hive/trunk/hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/DummyStorageHandler.java
* /hive/trunk/hcatalog/storage-handlers/hbase/pom.xml
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseHCatStorageHandler.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/RevisionManager.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/SkeletonHBaseTest.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseInputFormat.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHCatHBaseInputFormat.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHiveHBaseStorageHandler.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHiveHBaseTableOutputFormat.java
* 
/hive/trunk/hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestPigHBaseStorageHandler.java


> Make the Hive HBase storage handler work from HCatalog, and use 
> HiveStorageHandlers instead of HCatStorageHandlers
> --
>
> Key: HIVE-5261
> URL: https://issues.apache.org/jira/browse/HIVE-5261
> Project: Hive
>  Issue Type: Sub-task
>  Components: HBase Handler, HCatalog
>Affects Versions: 0.12.0
>Reporter: Sushanth Sowmyan
>Assignee: Viraj Bhat
> Fix For: 0.13.0
>
> Attachments: HIVE-5261.patch
>
>
> This is a task being created for the HCat side of HIVE-4331

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5260) Introduce HivePassThroughOutputFormat that allows Hive to use general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766335#comment-13766335
 ] 

Hudson commented on HIVE-5260:
--

ABORTED: Integrated in Hive-trunk-hadoop2 #425 (See 
[https://builds.apache.org/job/Hive-trunk-hadoop2/425/])
HIVE-5260 : Introduce HivePassThroughOutputFormat that allows Hive to use 
general purpose OutputFormats instead of HiveOutputFormats in StorageHandlers 
(Viraj Bhat via Ashutosh Chauhan) (hashutosh: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522574)
* 
/hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
* 
/hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
* /hive/trunk/hbase-handler/src/test/results/positive/external_table_ppd.q.out
* 
/hive/trunk/hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java


> Introduce HivePassThroughOutputFormat that allows Hive to use general purpose 
> OutputFormats instead of HiveOutputFormats in StorageHandlers
> ---
>
> Key: HIVE-5260
> URL: https://issues.apache.org/jira/browse/HIVE-5260
> Project: Hive
>  Issue Type: Sub-task
>  Components: HBase Handler, HCatalog
>Affects Versions: 0.12.0
>Reporter: Sushanth Sowmyan
>Assignee: Viraj Bhat
> Fix For: 0.13.0
>
> Attachments: HIVE-5260.2.patch, HIVE-5260.patch
>
>
> This is a task being created to address the hive-side of HIVE-4331.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5278) Move some string UDFs to GenericUDFs, for better varchar support

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5278:
--

Attachment: D12909.1.patch

jdere requested code review of "HIVE-5278 [jira] Move some string UDFs to 
GenericUDFs, for better varchar support".

Reviewers: JIRA

HIVE-5278: Move some string UDFs to GenericUDFs, for better varchar support

To better support varchar/char types in string UDFs, select UDFs should be 
converted to GenericUDFs. This allows the UDF to return the resulting 
char/varchar length in the type metadata.

This work is being split off as a separate task from HIVE-4844. The initial 
UDFs as part of this work are concat/lower/upper.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D12909

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLower.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUpper.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java
  ql/src/test/results/compiler/plan/groupby2.q.xml
  ql/src/test/results/compiler/plan/udf6.q.xml

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/30879/

To: JIRA, jdere


> Move some string UDFs to GenericUDFs, for better varchar support
> 
>
> Key: HIVE-5278
> URL: https://issues.apache.org/jira/browse/HIVE-5278
> Project: Hive
>  Issue Type: Improvement
>  Components: Types, UDF
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: D12909.1.patch, HIVE-5278.1.patch
>
>
> To better support varchar/char types in string UDFs, select UDFs should be 
> converted to GenericUDFs. This allows the UDF to return the resulting 
> char/varchar length in the type metadata.
> This work is being split off as a separate task from HIVE-4844. The initial 
> UDFs as part of this work are concat/lower/upper.
> NO PRECOMMIT TESTS - dependent on HIVE-4844

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Review Request 14121: Base64 decoding for binary type

[Mohammad Islam]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14121/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-5221
https://issues.apache.org/jira/browse/HIVE-5221


Repository: hive-git


Description
---

WIP Patch


Copied from the JIRA:

Hi,
I am using Hive 10. When I create an external table with column type as Binary, 
the query result on the table is showing some junk values for the column with 
binary datatype.
Please find below the query I have used to create the table:
CREATE EXTERNAL TABLE BOOL1(NB BOOLEAN,email STRING, bitfld BINARY)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '^'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION '/user/hivetables/testbinary';
The query I have used is : select * from bool1
The sample data in the hdfs file is:
0^a...@abc.com^001
1^a...@abc.com^010
^a...@abc.com^011
^a...@abc.com^100
t^a...@abc.com^101
f^a...@abc.com^110
true^a...@abc.com^111
false^a...@abc.com^001
123^ ^01100010
12344^ ^0111
Please share your inputs if it is possible.
Thanks,
Arun


Diffs
-

  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 4900a41 
  
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java
 28f8d6a 
  serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 
11f5f07 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java ae12f20 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java a08b4a8 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 
606208c 
  
serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java
 dbd60f7 
  
serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java
 afbf454 

Diff: https://reviews.apache.org/r/14121/diff/


Testing
---

WIP:
working on writing new test cases.


Thanks,

Mohammad Islam

[jira] [Updated] (HIVE-5221) Issue in colun type with data type as BINARY

[Mohammad Kamrul Islam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mohammad Kamrul Islam updated HIVE-5221:


Status: Patch Available  (was: Open)

> Issue in colun type with data type as BINARY
> 
>
> Key: HIVE-5221
> URL: https://issues.apache.org/jira/browse/HIVE-5221
> Project: Hive
>  Issue Type: Bug
>Reporter: Arun Vasu
>Assignee: Mohammad Kamrul Islam
>Priority: Critical
> Attachments: HIVE-5221.1.patch
>
>
> Hi,
> I am using Hive 10. When I create an external table with column type as 
> Binary, the query result on the table is showing some junk values for the 
> column with binary datatype.
> Please find below the query I have used to create the table:
> CREATE EXTERNAL TABLE BOOL1(NB BOOLEAN,email STRING, bitfld BINARY)
>  ROW FORMAT DELIMITED
>FIELDS TERMINATED BY '^'
>LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/user/hivetables/testbinary';
> The query I have used is : select * from bool1
> The sample data in the hdfs file is:
> 0^a...@abc.com^001
> 1^a...@abc.com^010
>  ^a...@abc.com^011
>  ^a...@abc.com^100
> t^a...@abc.com^101
> f^a...@abc.com^110
> true^a...@abc.com^111
> false^a...@abc.com^001
> 123^^01100010
> 12344^^0111
> Please share your inputs if it is possible.
> Thanks,
> Arun

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5221) Issue in colun type with data type as BINARY

[Mohammad Kamrul Islam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mohammad Kamrul Islam updated HIVE-5221:


Attachment: HIVE-5221.1.patch

WIP Patch.
Can be reviewed at : https://reviews.apache.org/r/14121/


> Issue in colun type with data type as BINARY
> 
>
> Key: HIVE-5221
> URL: https://issues.apache.org/jira/browse/HIVE-5221
> Project: Hive
>  Issue Type: Bug
>Reporter: Arun Vasu
>Assignee: Mohammad Kamrul Islam
>Priority: Critical
> Attachments: HIVE-5221.1.patch
>
>
> Hi,
> I am using Hive 10. When I create an external table with column type as 
> Binary, the query result on the table is showing some junk values for the 
> column with binary datatype.
> Please find below the query I have used to create the table:
> CREATE EXTERNAL TABLE BOOL1(NB BOOLEAN,email STRING, bitfld BINARY)
>  ROW FORMAT DELIMITED
>FIELDS TERMINATED BY '^'
>LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/user/hivetables/testbinary';
> The query I have used is : select * from bool1
> The sample data in the hdfs file is:
> 0^a...@abc.com^001
> 1^a...@abc.com^010
>  ^a...@abc.com^011
>  ^a...@abc.com^100
> t^a...@abc.com^101
> f^a...@abc.com^110
> true^a...@abc.com^111
> false^a...@abc.com^001
> 123^^01100010
> 12344^^0111
> Please share your inputs if it is possible.
> Thanks,
> Arun

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5122) Add partition for multiple partition ignores locations for non-first partitions

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766352#comment-13766352
 ] 

Thejas M Nair commented on HIVE-5122:
-

[~navis] Your changes in this patch make the logic in both anltr grammar and 
DDLSemanticAnalyzer code much cleaner, it looks good. But I didn't understand 
what was causing this bug (and how the change fixes it). Was it some anltr 
weird behavior with old grammar that caused the issue?


> Add partition for multiple partition ignores locations for non-first 
> partitions
> ---
>
> Key: HIVE-5122
> URL: https://issues.apache.org/jira/browse/HIVE-5122
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-5122.D12411.1.patch, HIVE-5122.D12411.2.patch
>
>
> http://www.mail-archive.com/user@hive.apache.org/msg09151.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4998) support jdbc documented table types in default configuration

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-4998:


Attachment: HIVE-4998.1.patch

> support jdbc documented table types in default configuration
> 
>
> Key: HIVE-4998
> URL: https://issues.apache.org/jira/browse/HIVE-4998
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.11.0
>Reporter: Thejas M Nair
> Attachments: HIVE-4998.1.patch
>
>
> The jdbc table types supported by hive server2 are not the documented typical 
> types [1] in jdbc, they are hive specific types (MANAGED_TABLE, 
> EXTERNAL_TABLE, VIRTUAL_VIEW). 
> HIVE-4573 added support for the jdbc documented typical types, but the HS2 
> default configuration is to return the hive types 
> The default configuration should result in the expected jdbc typical behavior.
> [1] 
> http://docs.oracle.com/javase/6/docs/api/java/sql/DatabaseMetaData.html?is-external=true#getTables(java.lang.String,
>  java.lang.String, java.lang.String, java.lang.String[])

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HIVE-5237) Incorrect group-by aggregation in 0.11.0

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Børge Svingen resolved HIVE-5237.
-

Resolution: Duplicate

> Incorrect group-by aggregation in 0.11.0
> 
>
> Key: HIVE-5237
> URL: https://issues.apache.org/jira/browse/HIVE-5237
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Børge Svingen
>Priority: Critical
>
> group by with sub queries does not correctly aggregate results in Hive 0.11.0.
> To reproduce:
> Put the file
> {code}
> 1,b
> 2,c
> 2,b
> 3,a
> 3,c
> 4,a
> {code}
> in HDFS, and run
> {code}
> create external table abc (x int, y string) row format delimited fields 
> terminated by ',' location '/data/';
> {code}
> The query
> {code}
> select
> x,
> count(*)
> from
> (select
> x,
> y
> from
> abc
> group by
>   x,
>   y
> ) a
> group by
> x;
> {code}
> will then give the result
> {code}
> 2 1
> 3 1
> 2 1
> 4 1
> 3 1
> 1 1
> {code}
> instead of the correct
> {code}
> 1 1
> 2 2
> 3 2
> 4 1
> {code}
> In 0.9.0 and 0.10.0 this is all working correctly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Review Request 14122: support jdbc documented table types in default configuration

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14122/
---

Review request for hive.


Bugs: HIVE-4998
https://issues.apache.org/jira/browse/HIVE-4998


Repository: hive-git


Description
---

The jdbc table types supported by hive server2 are not the documented typical 
types [1] in jdbc, they are hive specific types (MANAGED_TABLE, EXTERNAL_TABLE, 
VIRTUAL_VIEW).
HIVE-4573 added support for the jdbc documented typical types, but the HS2 
default configuration is to return the hive types
The default configuration should result in the expected jdbc typical behavior.
[1] 
http://docs.oracle.com/javase/6/docs/api/java/sql/DatabaseMetaData.html?is-external=true#getTables(java.lang.String,
 java.lang.String, java.lang.String, java.lang.String[])


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c5a8ff3 
  conf/hive-default.xml.template 181011b 
  jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java c658dbd 

Diff: https://reviews.apache.org/r/14122/diff/


Testing
---


Thanks,

Thejas Nair

[jira] [Commented] (HIVE-5112) Upgrade protobuf to 2.5 from 2.4

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766292#comment-13766292
 ] 

Thejas M Nair commented on HIVE-5112:
-

There are discussions in the hadoop mailing lists about having the hadoop 2 GA 
in few weeks. So I think it is worthwhile that we aim to have 0.12 to be 
compatible with the hadoop 2 line as well.

[~appodictic] I don't think your -1 is reasonable for the following reasons- 
- I don't consider it feasible to stop upgrading jars in hive because the old 
version of jar was used by some udf/serde/inputformat. (But I do think that in 
long term we need to explore ways of isolating the hive jars/classes vs user 
jars/classes) 
- In the example you cited, as Sushanth mentioned, you don't need to make any 
code changes to use protobuf 2.5. I suspect most users who are using protobuf 
with dynamic schema would not choose to work with generated code. So the number 
of users this is likely to affect is small.
- The upgrade is inevitable. With it being in trunk, it is in its way into the 
0.13 release.

I request you to please reconsider your -1. 


> Upgrade protobuf to 2.5 from 2.4
> 
>
> Key: HIVE-5112
> URL: https://issues.apache.org/jira/browse/HIVE-5112
> Project: Hive
>  Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Owen O'Malley
> Fix For: 0.13.0
>
> Attachments: HIVE-5112.2.patch, HIVE-5112.D12429.1.patch
>
>
> Hadoop and Hbase have both upgraded protobuf. We should as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5237) Incorrect group-by aggregation in 0.11.0

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766354#comment-13766354
 ] 

Børge Svingen commented on HIVE-5237:
-

Thanks, resolving this one, then.

> Incorrect group-by aggregation in 0.11.0
> 
>
> Key: HIVE-5237
> URL: https://issues.apache.org/jira/browse/HIVE-5237
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Børge Svingen
>Priority: Critical
>
> group by with sub queries does not correctly aggregate results in Hive 0.11.0.
> To reproduce:
> Put the file
> {code}
> 1,b
> 2,c
> 2,b
> 3,a
> 3,c
> 4,a
> {code}
> in HDFS, and run
> {code}
> create external table abc (x int, y string) row format delimited fields 
> terminated by ',' location '/data/';
> {code}
> The query
> {code}
> select
> x,
> count(*)
> from
> (select
> x,
> y
> from
> abc
> group by
>   x,
>   y
> ) a
> group by
> x;
> {code}
> will then give the result
> {code}
> 2 1
> 3 1
> 2 1
> 4 1
> 3 1
> 1 1
> {code}
> instead of the correct
> {code}
> 1 1
> 2 2
> 3 2
> 4 1
> {code}
> In 0.9.0 and 0.10.0 this is all working correctly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3764) Support metastore version consistency check

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-3764:
--

Status: Patch Available  (was: Open)

Review request on https://reviews.apache.org/r/14120/

> Support metastore version consistency check
> ---
>
> Key: HIVE-3764
> URL: https://issues.apache.org/jira/browse/HIVE-3764
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.11.0, 0.10.0, 0.9.0, 0.8.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Fix For: 0.12.0
>
> Attachments: HIVE-3764.4.patch
>
>
> Today there's no version/compatibility information stored in hive metastore. 
> Also the datanucleus configuration property to automatically create missing 
> tables is enabled by default. If you happen to start an older or newer hive 
> or don't run the correct upgrade scripts during migration, the metastore 
> would end up corrupted. The autoCreate schema is not always sufficient to 
> upgrade metastore when migrating to newer release. It's not supported with 
> all databases. Besides the migration often involves altering existing table, 
> changing or moving data etc.
> Hence it's very useful to have some consistency check to make sure that hive 
> is using correct metastore and for production systems the schema is not 
> automatically by running hive.
> Besides it would be helpful to add a tool that can leverage this version 
> information to figure out the required set of upgrade scripts, and execute 
> those against the configured metastore. Now that Hive includes Beeline 
> client, it can be used to execute the scripts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3764) Support metastore version consistency check

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-3764:
--

Attachment: (was: HIVE-3764-1.patch)

> Support metastore version consistency check
> ---
>
> Key: HIVE-3764
> URL: https://issues.apache.org/jira/browse/HIVE-3764
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
>
> Today there's no version/compatibility information stored in hive metastore. 
> Also the datanucleus configuration property to automatically create missing 
> tables is enabled by default. If you happen to start an older or newer hive 
> or don't run the correct upgrade scripts during migration, the metastore 
> would end up corrupted. The autoCreate schema is not always sufficient to 
> upgrade metastore when migrating to newer release. It's not supported with 
> all databases. Besides the migration often involves altering existing table, 
> changing or moving data etc.
> Hence it's very useful to have some consistency check to make sure that hive 
> is using correct metastore and for production systems the schema is not 
> automatically by running hive.
> Besides it would be helpful to add a tool that can leverage this version 
> information to figure out the required set of upgrade scripts, and execute 
> those against the configured metastore. Now that Hive includes Beeline 
> client, it can be used to execute the scripts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4998) support jdbc documented table types in default configuration

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-4998:


Assignee: Thejas M Nair
  Status: Patch Available  (was: Open)

> support jdbc documented table types in default configuration
> 
>
> Key: HIVE-4998
> URL: https://issues.apache.org/jira/browse/HIVE-4998
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.11.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-4998.1.patch
>
>
> The jdbc table types supported by hive server2 are not the documented typical 
> types [1] in jdbc, they are hive specific types (MANAGED_TABLE, 
> EXTERNAL_TABLE, VIRTUAL_VIEW). 
> HIVE-4573 added support for the jdbc documented typical types, but the HS2 
> default configuration is to return the hive types 
> The default configuration should result in the expected jdbc typical behavior.
> [1] 
> http://docs.oracle.com/javase/6/docs/api/java/sql/DatabaseMetaData.html?is-external=true#getTables(java.lang.String,
>  java.lang.String, java.lang.String, java.lang.String[])

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3764) Support metastore version consistency check

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-3764:
--

Attachment: HIVE-3764.4.patch

Patch for 0.12 branch

> Support metastore version consistency check
> ---
>
> Key: HIVE-3764
> URL: https://issues.apache.org/jira/browse/HIVE-3764
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.8.0, 0.9.0, 0.10.0, 0.11.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Fix For: 0.12.0
>
> Attachments: HIVE-3764.4.patch
>
>
> Today there's no version/compatibility information stored in hive metastore. 
> Also the datanucleus configuration property to automatically create missing 
> tables is enabled by default. If you happen to start an older or newer hive 
> or don't run the correct upgrade scripts during migration, the metastore 
> would end up corrupted. The autoCreate schema is not always sufficient to 
> upgrade metastore when migrating to newer release. It's not supported with 
> all databases. Besides the migration often involves altering existing table, 
> changing or moving data etc.
> Hence it's very useful to have some consistency check to make sure that hive 
> is using correct metastore and for production systems the schema is not 
> automatically by running hive.
> Besides it would be helpful to add a tool that can leverage this version 
> information to figure out the required set of upgrade scripts, and execute 
> those against the configured metastore. Now that Hive includes Beeline 
> client, it can be used to execute the scripts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5290) Some HCatalog tests have been behaving flaky

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766571#comment-13766571
 ] 

Ashutosh Chauhan commented on HIVE-5290:


Glad finally there is a hope for these hcat tests : ) Thanks, [~brocknoland] 
for looking into this. Tried applying your patch, got compile error:
{code}
hcatalog/hcatalog-pig-adapter/src/test/java/org/apache/hive/hcatalog/pig/TestE2EScenarios.java:204:
 cannot find symbol
[javac] symbol  : constructor 
TaskID(org.apache.hadoop.mapreduce.JobID,org.apache.hadoop.mapreduce.TaskType,int)
[javac] location: class org.apache.hadoop.mapreduce.TaskID
[javac] TaskAttemptID taskId = new TaskAttemptID(new TaskID(new 
JobID("200908190029", 1), TaskType.REDUCE, 1), 1);
[javac]  ^

{code}

> Some HCatalog tests have been behaving flaky
> 
>
> Key: HIVE-5290
> URL: https://issues.apache.org/jira/browse/HIVE-5290
> Project: Hive
>  Issue Type: Test
>Affects Versions: 0.13.0
>Reporter: Brock Noland
>Assignee: Brock Noland
> Attachments: HIVE-5290.patch, HIVE-5290.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5291) join32_lessSize.q has ordering problem under hadoop-2

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766602#comment-13766602
 ] 

Ashutosh Chauhan commented on HIVE-5291:


Actually, I cannot repro the fail you are mentioning on latest trunk. I did
{noformat}
ant test -Dtestcase=TestCliDriver -Dmodule=ql -Dqfile=join32_lessSize.q 
-Dhadoop.mr.rev=23
{noformat}
and I got:
{noformat}
[junit] Done query: join32_lessSize.q elapsedTime=433s
[junit] Cleaning up TestCliDriver
[junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 436.32 sec
{noformat}

So, I am not sure what is there to fix.

> join32_lessSize.q has ordering problem under hadoop-2
> -
>
> Key: HIVE-5291
> URL: https://issues.apache.org/jira/browse/HIVE-5291
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.13.0
>
> Attachments: HIVE-5291.patch
>
>
> Test just needs more ORDER BY and output updated.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5289) PTest2 should disable checking of libraries during batch exec

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766607#comment-13766607
 ] 

Thejas M Nair commented on HIVE-5289:
-

+! LGTM


> PTest2 should disable checking of libraries during batch exec
> -
>
> Key: HIVE-5289
> URL: https://issues.apache.org/jira/browse/HIVE-5289
> Project: Hive
>  Issue Type: Test
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.13.0
>
> Attachments: HIVE-5289.patch
>
>
> PTest2 has two phases:
> 1) Build from source
> 2) Exec in parallel
> during phase two we don't want ivy making HTTP requests.
> NO PRECOMMIT TESTS

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4690) stats_partscan_1.q makes different result with different hadhoop.mr.rev

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766520#comment-13766520
 ] 

Hive QA commented on HIVE-4690:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12602857/HIVE-4690.2.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 3109 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.listener.TestNotificationListener.testAMQListener
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/716/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/716/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

> stats_partscan_1.q makes different result with different hadhoop.mr.rev 
> 
>
> Key: HIVE-4690
> URL: https://issues.apache.org/jira/browse/HIVE-4690
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 0.11.0
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-4690.2.patch, HIVE-4690.D11163.1.patch
>
>
> stats_partscan_1.q uses mapred.min/max.split.size and logs number of files, 
> which can be different with different hadoop.mr.rev.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-5291) join32_lessSize.q has ordering problem under hadoop-2

[Brock Noland (JIRA)]

Brock Noland created HIVE-5291:
--

 Summary: join32_lessSize.q has ordering problem under hadoop-2
 Key: HIVE-5291
 URL: https://issues.apache.org/jira/browse/HIVE-5291
 Project: Hive
  Issue Type: Sub-task
  Components: Tests
Reporter: Brock Noland
Assignee: Brock Noland
 Fix For: 0.13.0


Test just needs more ORDER BY and output updated.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5289) PTest2 should disable checking of libraries during batch exec

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766610#comment-13766610
 ] 

Thejas M Nair commented on HIVE-5289:
-

+1 (accidentally used ! instead of 1 previous time!)


> PTest2 should disable checking of libraries during batch exec
> -
>
> Key: HIVE-5289
> URL: https://issues.apache.org/jira/browse/HIVE-5289
> Project: Hive
>  Issue Type: Test
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.13.0
>
> Attachments: HIVE-5289.patch
>
>
> PTest2 has two phases:
> 1) Build from source
> 2) Exec in parallel
> during phase two we don't want ivy making HTTP requests.
> NO PRECOMMIT TESTS

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4732) Reduce or eliminate the expensive Schema equals() check for AvroSerde

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766543#comment-13766543
 ] 

Ashutosh Chauhan commented on HIVE-4732:


Thanks, [~kamrul] for addressing concern. Overall looks good. One quick 
question: I see in AvroGenericRecordWritable::write(Dataoutput out) this line 
out.writeUTF(recordReaderID.toString()); Doesn't this mean id is now persisted 
on-disk? I thought id is generated by reader at read time and than added to 
record, but I don't get why while writing record we need to write it. Seems 
like I am missing something obvious.

> Reduce or eliminate the expensive Schema equals() check for AvroSerde
> -
>
> Key: HIVE-4732
> URL: https://issues.apache.org/jira/browse/HIVE-4732
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Reporter: Mark Wagner
>Assignee: Mohammad Kamrul Islam
> Attachments: HIVE-4732.1.patch, HIVE-4732.v1.patch
>
>
> The AvroSerde spends a significant amount of time checking schema equality. 
> Changing to compare hashcodes (which can be computed once then reused) will 
> improve performance.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5291) join32_lessSize.q has ordering problem under hadoop-2

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766582#comment-13766582
 ] 

Ashutosh Chauhan commented on HIVE-5291:


Patch is not applying cleanly on trunk.

> join32_lessSize.q has ordering problem under hadoop-2
> -
>
> Key: HIVE-5291
> URL: https://issues.apache.org/jira/browse/HIVE-5291
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.13.0
>
> Attachments: HIVE-5291.patch
>
>
> Test just needs more ORDER BY and output updated.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5291) join32_lessSize.q has ordering problem under hadoop-2

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766623#comment-13766623
 ] 

Brock Noland commented on HIVE-5291:


Then it's a java 7 issue.

> join32_lessSize.q has ordering problem under hadoop-2
> -
>
> Key: HIVE-5291
> URL: https://issues.apache.org/jira/browse/HIVE-5291
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.13.0
>
> Attachments: HIVE-5291.patch
>
>
> Test just needs more ORDER BY and output updated.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3745) Hive does improper "=" based string comparisons for strings with trailing whitespaces

[Dave Winters (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766629#comment-13766629
 ] 

Dave Winters commented on HIVE-3745:


This issue is causing real-world problems. We hit this with our product in 
building a SQL-H connector.
Facebook has patched this internally.
Why is it not out in the community or a Hive conf parm that is settable?

> Hive does improper "=" based string comparisons for strings with trailing 
> whitespaces
> -
>
> Key: HIVE-3745
> URL: https://issues.apache.org/jira/browse/HIVE-3745
> Project: Hive
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 0.9.0
>Reporter: Harsh J
>Assignee: Gang Tim Liu
>
> Compared to other systems such as DB2, MySQL, etc., which disregard trailing 
> whitespaces in a string used when comparing two strings with the "{{=}}" 
> relational operator, Hive does not do this.
> For example, note the following line from the MySQL manual: 
> http://dev.mysql.com/doc/refman/5.1/en/char.html
> {quote}
> All MySQL collations are of type PADSPACE. This means that all CHAR and 
> VARCHAR values in MySQL are compared without regard to any trailing spaces. 
> {quote}
> Hive still is whitespace sensitive and regards trailing spaces of a string as 
> worthy elements when comparing. Ideally {{LIKE}} should consider this 
> strongly, but {{=}} should not.
> Is there a specific reason behind this difference of implementation in Hive's 
> SQL?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5291) join32_lessSize.q has ordering problem under hadoop-2

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766594#comment-13766594
 ] 

Brock Noland commented on HIVE-5291:


Shoot, I will fix this and HIVE-5290 after lunch.

> join32_lessSize.q has ordering problem under hadoop-2
> -
>
> Key: HIVE-5291
> URL: https://issues.apache.org/jira/browse/HIVE-5291
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.13.0
>
> Attachments: HIVE-5291.patch
>
>
> Test just needs more ORDER BY and output updated.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HIVE-3745) Hive does improper "=" based string comparisons for strings with trailing whitespaces

[Gang Tim Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gang Tim Liu reassigned HIVE-3745:
--

Assignee: Kevin Wilfong  (was: Gang Tim Liu)

> Hive does improper "=" based string comparisons for strings with trailing 
> whitespaces
> -
>
> Key: HIVE-3745
> URL: https://issues.apache.org/jira/browse/HIVE-3745
> Project: Hive
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 0.9.0
>Reporter: Harsh J
>Assignee: Kevin Wilfong
>
> Compared to other systems such as DB2, MySQL, etc., which disregard trailing 
> whitespaces in a string used when comparing two strings with the "{{=}}" 
> relational operator, Hive does not do this.
> For example, note the following line from the MySQL manual: 
> http://dev.mysql.com/doc/refman/5.1/en/char.html
> {quote}
> All MySQL collations are of type PADSPACE. This means that all CHAR and 
> VARCHAR values in MySQL are compared without regard to any trailing spaces. 
> {quote}
> Hive still is whitespace sensitive and regards trailing spaces of a string as 
> worthy elements when comparing. Ideally {{LIKE}} should consider this 
> strongly, but {{=}} should not.
> Is there a specific reason behind this difference of implementation in Hive's 
> SQL?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4443) [HCatalog] Have an option for GET queue to return all job information in single call

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766676#comment-13766676
 ] 

Daniel Dai commented on HIVE-4443:
--

Yes, document update is in HIVE-4446

> [HCatalog] Have an option for GET queue to return all job information in 
> single call 
> -
>
> Key: HIVE-4443
> URL: https://issues.apache.org/jira/browse/HIVE-4443
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.12.0
>
> Attachments: HIVE-4443-1.patch, HIVE-4443-2.patch, HIVE-4443-3.patch
>
>
> Currently do display a summary of all jobs, one has to call GET queue to 
> retrieve all the jobids and then call GET queue/:jobid for each job. It would 
> be nice to do this in a single call.
> I would suggest:
> * GET queue - mark deprecate
> * GET queue/ - mark deprecate
> * DELETE queue/ - mark deprecate
> * GET jobs - return the list of JSON objects jobid but no detailed info
> * GET jobs/fields=* - return the list of JSON objects containing detailed Job 
> info
> * GET jobs/ - return the single JSON object containing the detailed 
> Job info for the job with the given ID (equivalent to GET queue/)
> * DELETE jobs/ - equivalent to DELETE queue/
> NO PRECOMMIT TESTS 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5282) Some tests don't use ${system:test.dfs.mkdir} for mkdir

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766417#comment-13766417
 ] 

Hudson commented on HIVE-5282:
--

SUCCESS: Integrated in Hive-trunk-h0.21 #2328 (See 
[https://builds.apache.org/job/Hive-trunk-h0.21/2328/])
HIVE-5282 : Some tests don't use  for mkdir (Brock Noland via Ashutosh Chauhan) 
(hashutosh: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522741)
* 
/hive/trunk/ql/src/test/queries/clientpositive/load_hdfs_file_with_space_in_the_name.q
* /hive/trunk/ql/src/test/queries/clientpositive/schemeAuthority2.q


> Some tests don't use ${system:test.dfs.mkdir} for mkdir
> ---
>
> Key: HIVE-5282
> URL: https://issues.apache.org/jira/browse/HIVE-5282
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-5282.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5206) Support parameterized primitive types

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766418#comment-13766418
 ] 

Hudson commented on HIVE-5206:
--

SUCCESS: Integrated in Hive-trunk-h0.21 #2328 (See 
[https://builds.apache.org/job/Hive-trunk-h0.21/2328/])
HIVE-5206 : Support parameterized primitive types (Jason Dere via Ashutosh 
Chauhan) (hashutosh: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522715)
* 
/hive/trunk/contrib/src/java/org/apache/hadoop/hive/contrib/util/typedbytes/TypedBytesRecordReader.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/VirtualColumn.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/SettableUDF.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/AbstractPrimitiveObjectInspector.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/BaseTypeParams.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/ParameterizedPrimitiveTypeUtils.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/PrimitiveTypeInfo.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/PrimitiveTypeSpec.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java
* 
/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoUtils.java


> Support parameterized primitive types
> -
>
> Key: HIVE-5206
> URL: https://issues.apache.org/jira/browse/HIVE-5206
> Project: Hive
>  Issue Type: Improvement
>  Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 0.13.0
>
> Attachments: HIVE-5206.1.patch, HIVE-5206.2.patch, HIVE-5206.3.patch, 
> HIVE-5206.4.patch, HIVE-5206.D12693.1.patch
>
>
> Support for parameterized types is needed for char/varchar/decimal support. 
> This adds a type parameters value to the 
> PrimitiveTypeEntry/PrimitiveTypeInfo/PrimitiveObjectInspector objects. 
> NO PRECOMMIT TESTS - dependent on HIVE-5203/HIVE-5204

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5112) Upgrade protobuf to 2.5 from 2.4

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766693#comment-13766693
 ] 

Edward Capriolo commented on HIVE-5112:
---

{quote}
The upgrade is inevitable. With it being in trunk, it is in its way into the 
0.13 release.
{quote}
Yes but this will give users time to prepare for it. Which is what rushing the 
upgrade will take away.

{quote}
I don't consider it feasible to stop upgrading jars in hive because the old 
version of jar was used by some udf/serde/inputformat. 
{quote}
You want this upgrade because it effects the OCR input formats you care about. 
It effects new beta version that you want to support. 

Q. Why don't we remove the dependency from hive all together? A. We can't 
because it's baked via an input format.

{quote}
But I do think that in long term we need to explore ways of isolating the hive 
jars/classes vs user jars/classes.
{quote}
But why do that? We can just bake in all the things we care about! Then 
whenever it screws someone up we can just say "long term we might research a 
way not to inconvenience people, but for now your inconvenienced"

I don't care really take away my -1. I would not want the to inconvenience the 
0 people running future-ware versions in production.












> Upgrade protobuf to 2.5 from 2.4
> 
>
> Key: HIVE-5112
> URL: https://issues.apache.org/jira/browse/HIVE-5112
> Project: Hive
>  Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Owen O'Malley
> Fix For: 0.13.0
>
> Attachments: HIVE-5112.2.patch, HIVE-5112.D12429.1.patch
>
>
> Hadoop and Hbase have both upgraded protobuf. We should as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4531) [WebHCat] Collecting task logs to hdfs

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-4531:
-

Release Note: 
In POST pig/hive/jar/steaming request, if statusdir is set and enablelog=true, 
webhcat will create a $statusdir/logs directory after task finish. The attempts 
here include completed attempts and failed attempts. The layout for logs 
directory is:

logs/$job_id (directory for $job_id)
logs/$job_id/job.xml.html
logs/$job_id/$attempt_id (directory for $attempt_id)
logs/$job_id/$attempt_id/stderr
logs/$job_id/$attempt_id/stdout
logs/$job_id/$attempt_id/syslog

NO PRECOMMIT TESTS

  was:
In POST pig/hive/jar/steaming request, if statusdir is set and enablelog=true, 
webhcat will create a $statusdir/logs directory after task finish. The attempts 
here include completed attempts and failed attempts. The layout for logs 
directory is:

logs/$job_id (directory for $job_id)
logs/$job_id/job.xml.html
logs/$job_id/$attempt_id (directory for $attempt_id)
logs/$job_id/$attempt_id/stderr
logs/$job_id/$attempt_id/stdout
logs/$job_id/$attempt_id/syslog


> [WebHCat] Collecting task logs to hdfs
> --
>
> Key: HIVE-4531
> URL: https://issues.apache.org/jira/browse/HIVE-4531
> Project: Hive
>  Issue Type: New Feature
>  Components: HCatalog
>Reporter: Daniel Dai
> Attachments: HIVE-4531-1.patch, HIVE-4531-2.patch, HIVE-4531-3.patch, 
> HIVE-4531-4.patch, HIVE-4531-5.patch, HIVE-4531-6.patch, HIVE-4531-7.patch, 
> HIVE-4531-8.patch, samplestatusdirwithlist.tar.gz
>
>
> It would be nice we collect task logs after job finish. This is similar to 
> what Amazon EMR does.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira