Re: Question regarding lock manager

You do not need ZooKeeper to use ACID in Hive. The first thing I would check is that you have configured your system as described on this page: https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions. Also, make sure you have not set hive.lock.manager to zookeeper. There are other fea

[ANNOUNCE] Apache Hive 2.3.7 Released

The Apache Hive team is proud to announce the release of Apache Hive version 2.3.7. The Apache Hive (TM) data warehouse software facilitates querying and managing large datasets residing in distributed storage. Built on top of Apache Hadoop (TM), it provides, among others: * Tools to enable easy

Re: metastore without hadoop

It needs Hadoop libraries; it still uses some HDFS libraries for file reading and password management. It does not require a running, installed Hadoop. It does not ship with the required Hadoop libraries to avoid version clashes when installed on a running Hadoop system. Alan. On Thu, Apr 9, 20

Re: If Hive Metastore is compatibility with MariaDB version 10.x.?

Hive is tested against MariaDB 5.5, so I can't say whether it will work against version 10. You would need to do some testing with it to see. Alan. On Fri, Jan 17, 2020 at 4:29 AM Oleksiy S wrote: > Hi all. > > Could you please help? Customer asked if Hive Metastore is compatible with > MariaD

Re: Locks with ACID: need some clarifications

say "you just can't have two simultaneous deletes in the same > partition", simultaneous means for the same transaction ? > If a create 2 "transactions" for 2 deletes on the same table/partition it > works. Am I right ? > > > Le lun. 9 sept. 2019 à 19:04, Al

Re: Locks with ACID: need some clarifications

In Hive 2 update and delete take what are called semi-shared locks (meaning they allow shared locks through, while not allowing other semi-shared locks), and insert and select take shared locks. So you can insert or select while deleting, you just can't have two simultaneous deletes in the same pa

[ANNOUNCE] Apache Hive 3.1.2 released

The Apache Hive team is proud to announce the release of Apache Hive version 3.1.2. The Apache Hive (TM) data warehouse software facilitates querying and managing large datasets residing in distributed storage. Built on top of Apache Hadoop (TM), it provides, among others: * Tools to enable easy

[ANNOUNCE] Apache Hive 2.3.6 Released

The Apache Hive team is proud to announce the release of Apache Hive version 2.3.6. The Apache Hive (TM) data warehouse software facilitates querying and managing large datasets residing in distributed storage. Built on top of Apache Hadoop (TM), it provides, among others: * Tools to enable easy d

Re: Question on Hive metastore thrift uri

It depends on how you configure the system. If you are using HS2 you can configure it to talk directly to the metastoredb (by providing it with the JDBC connection information and setting the metastore thrift url to localhost) or to talk through a metastore server instance (by not providing the JD

Re: Restrict users from creating tables in default warehouse

The easiest way to do this is through grant and revoke statements and/or file permissions. Hive has several authorization schemes (storage based auth, sql standard auth, integration with Ranger and Sentry) added over several releases. Which version of Hive are you using and which, if any, of thes

Re: Hive Insert and Select only specific columns ( not all columns ) - Partitioned table

You need to provide a value for the deptno partition key. You can't insert into a partitioned table without providing a value for the partition column. You can either give it a static value: insert into table emp_parquet partition (deptno = 'x') select empno, ename from emp or you can set it dyn

Re: hcatalog and hiveserver2

HCatalog was built as an interface to allow tools such as Pig and MapReduce to access Hive tabular data, for both read and write. In more recent versions of Hive, HCatalog has not been updated to support the newest features, such as reading or writing transactional data or, in Hive 3.x, accessing

Re: Consuming delta from Hive tables

derstand if I can use write_id to consume only > updated rows. > Store the maximum write_id(X) seen in the result and next time query for > all rows with row_id greater than X. > > Thanks, > Bhargav > > On Fri, May 17, 2019 at 10:37 PM Alan Gates wrote: > >> Sorry, loo

Re: Consuming delta from Hive tables

Sorry, looks like you sent this earlier and I missed it. A couple of things. One, write_id is per transaction per table. So for table T, all rows written in w1 will have the same write_id, though they will each have their own monotonically increasing row_ids. Row_ids are scoped by a write_id, s

Re: Any HIVE DDL statement takes minutes to execute

> there was no issue. > > > > HTH > > > > > > On Thu, 16 May 2019 at 08:16, Iulian Mongescu > wrote: > > Hi Alan, > > > > I’m using MySQL (Mariadb) for the metastore and I was thinking on this > possibility too but from all my tests on metast

Re: Any HIVE DDL statement takes minutes to execute

What are you using as the RDBMS for your metastore? A first place I'd look is if the communications with the RDBMS are slow for some reason. Alan. On Wed, May 15, 2019 at 10:34 AM Iulian Mongescu wrote: > Hello, > > > > I'm working on a HDP-2.6.5.0 cluster with kerberos enabled and I have a >

[ANNOUNCE] Apache Hive 2.3.5 Released

The Apache Hive team is proud to announce the release of Apache Hive version 2.3.5. The Apache Hive (TM) data warehouse software facilitates querying and managing large datasets residing in distributed storage. Built on top of Apache Hadoop (TM), it provides, among others: * Tools to enable easy

Re: Consuming delta from Hive tables

The other issue is an external system has no ability to control when the compactor is run (it rewrites deltas into the base files and thus erases intermediate states that would interest you). The mapping of writeids (table specific) to transaction ids (system wide) is also cleaned intermittently,

Re: HS2: Permission denied for my own table?

See https://cwiki.apache.org/confluence/display/Hive/Setting+up+HiveServer2#SettingUpHiveServer2-Impersonation Alan. On Tue, Apr 16, 2019 at 10:03 PM Kaidi Zhao wrote: > Hello! > > Did I miss anything here or it is an known issue? Hive 1.2.1, hadoop > 2.7.x, kerberos, impersonation. > > Using h

Re: How to update Hive ACID tables in Flink

here: https://orc.apache.org/docs/acid.html > The INSERT/UPDATE/DELETE seems to be implemented: > OPERATIONSERIALIZATION > INSERT 0 > UPDATE 1 > DELETE 2 > Do you think this approach is suitable ? > > > > Le mar. 12 mars 2019 à 19:30, Alan Gates a écrit : > &g

Re: Read Hive ACID tables in Spark or Pig

ade such improvements it has lost its old friends in the >> process. Hope one day all the friends speak together again: pig, spark, >> presto read/write ACID together. >> >> On Sat, Mar 09, 2019 at 02:23:48PM -0800, Alan Gates wrote: >> > There's only been one si

Re: How to update Hive ACID tables in Flink

Have you looked at Hive's streaming ingest? https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest It is designed for this case, though it only handles insert (not update), so if you need updates you'd have to do the merge as you are currently doing. Alan. On Mon, Mar 11, 2019 at

Re: Read Hive ACID tables in Spark or Pig

ose specific connectors ? > > Thanks > > > On Wed, Mar 06, 2019 at 09:51:51AM -0800, Alan Gates wrote: > > Pig is in the same place as Spark, that the tables need to be compacted > first. > > The issue is that outside readers don't understand which records in the >

Re: Read Hive ACID tables in Spark or Pig

Pig is in the same place as Spark, that the tables need to be compacted first. The issue is that outside readers don't understand which records in the delta files are valid and which are not. Theoretically all this is possible, as outside clients could get the valid transaction list from the meta

Re: Standalone Metastore Question

The standalone metastore released in 3.0 is the exact same metastore released with Hive 3.0. The only differences are in the install tool 'schematool' and the start and stop script. Hive 3 is being used in production a number of places. I don't know if anyone is running the metastore alone in pr

Re: Difference in performance of temp table vs subqueries

That's a broad question and it depends on what you're doing. Since temp tables will materialize the intermediate result while subqueries will not I'd guess in most cases subqueries are faster. But again, it depends on what you're doing, and you'd need to benchmark your particular queries both way

Re: Hive Metastore Hook to to fire only on success

Which version of Hive are you on and which hook are you seeing fire? Based on looking at the master code you should only see the commitCreateTable hook call if the creation succeeds. Alan. On Thu, Oct 4, 2018 at 12:36 AM Daniel Haviv wrote: > Hi, > I'm writing a HMS hook and I noticed that the

Re: How to Grant All Privileges for All Databases except one in Hive SQL

re looking for here. > > Yes i am looking something like this only and since it is not available, > does that mean i have to go for each table ? > > I am asking because we have many DBs and a lot of tables within each DB so > is there any other way ? > > Regards, > Anup Tiw

Re: Question about OVER clause

This article might be helpful. It's for SQL Server, but the semantics should be similar. https://www.sqlpassion.at/archive/2015/01/22/sql-server-windowing-functions-rows-vs-range/ Alan. On Wed, Sep 19, 2018 at 6:47 AM 孙志禹 wrote: > Dears, >What is the difference between *ROW BETWEEN* and *

Re: How to Grant All Privileges for All Databases except one in Hive SQL

-++-+-+-++---++--+ > | anup ||| | readonly| > ROLE| SELECT | false | 1537187896000 | hadoop | > > +---+++-+-+-+

Re: How to Grant All Privileges for All Databases except one in Hive SQL

You can see a full list of what grant supports at https://cwiki.apache.org/confluence/display/Hive/SQL+Standard+Based+Hive+Authorization#SQLStandardBasedHiveAuthorization-Grant There is no "grant x to user on all databases" or regex expressions for database names. So you'll have to do the databas

Re: Hive Metada as a microservice

In 3.0, you can download the metastore as a separate artifact, either source or binary (e.g. http://ftp.wayne.edu/apache/hive/hive-standalone-metastore-3.0.0/). It does not require any other parts of Hive beyond what's released in that artifact. I'm not sure if this meets your definition of a loo

Re: drop partitions

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-DropPartitions Alan. On Sat, Jun 16, 2018 at 8:03 PM Mahender Sarangam < mahender.bigd...@outlook.com> wrote: > Hi All, > > What is right syntax for dropping the partitions. Alter table drop if > exists partiti

Re: Oracle 11g Hive 2.1 metastore backend

We currently run our Oracle tests against 11g, but that is only for the 3.0 and beyond releases. Given the error I am guessing this is a result of the Oracle version plus the datanucleus version, which we changed between 2.1 and 2.3. Alan. On Wed, Jun 6, 2018 at 12:12 PM Mich Talebzadeh wrote:

Re: Is 'application' a reserved word?

It is. You can see the definitive list of keywords at https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g (Note this is for the master branch, you can switch the branch around to find the list for a particular release.) It would be good to file a JIR

Re: Combining hive tables as one query

ionality as in > postgres and views are also not helping here. > > On Tue, May 15, 2018 at 11:19 AM, Alan Gates wrote: > >> In general this is done using joins, as in all SQL engines. A google >> search on "intro to SQL joins" will suggest a number of resources,

Re: Combining hive tables as one query

In general this is done using joins, as in all SQL engines. A google search on "intro to SQL joins" will suggest a number of resources, for example https://www.essentialsql.com/get-ready-to-learn-sql-12-introduction-to-database-joins/ Alan. On Tue, May 15, 2018 at 7:37 AM, Sowjanya Kakarala wro

Re: Dynamic vs Static partition

When doing dynamic partitioning, Hive needs to look at each record and determine which partition to place it in. In the case where all records go to the same partition, it is more efficient to tell Hive up front, that is, to use static partitioning. So you can use dynamic partition for large file

Re: Creating Surrogate Keys in Hive

It isn't possible to guarantee sequential keys because tasks run in parallel. You can write a UDF to assign a unique id or sequential ids within a task. Alan. On Tue, Nov 21, 2017 at 3:53 AM, kishore kumar wrote: > Hi, > > Could some one suggest how to create surrogate keys sequentially in hiv

Re: Options for connecting to Apache Hive

There are ODBC drivers available for Hive, though they aren’t part of the open source package and are not free. Google can help you find them. As Elliot says, you can use the Thrift protocol, which is what the JDBC driver uses. You can find the thrift definition in the code at https://github.com

Re: HCatClient vs HiveMetaStoreClient (or IMetaStoreClient)

HCatClient is useful if you are already using HCat. If not, use HiveMetaStoreClient. It’s been kept much more up to date. Alan. On Fri, Nov 10, 2017 at 9:23 AM, Patel,Stephen wrote: > From a cursory inspection, it seems that HCatClient provides a subset of > the functionality that the HiveMet

Re: on master branch, hive code has some itests maven build error

I would suggest removing -Pdist in the initial mvn command. That should only be used to build tarballs for distribution. So your initial mvn command should just be: mvn clean install -DskipTests Alan. On Sat, Sep 23, 2017 at 3:55 AM, eric wong wrote: > I try to add some q file test to hive, s

Re: Aug. 2017 Hive User Group Meeting

The address is at the top of the text description, even though it isn’t in the location field: 5470 Great America Parkway, Santa Clara, CA Alan. On Mon, Aug 21, 2017 at 5:50 PM, dan young wrote: > For us out of town folks, where is the location of this meetup? Says > Hortonworks but do you hav

Re: please help unsubscribing from mailing lists

I think http://untroubled.org/ezmlm/manual/Unsubscribing.html#Unsubscribing has what you need. Alan. On Tue, Aug 15, 2017 at 1:09 PM, Chris Drome wrote: > I am currently subscribed to all three Hive mailing lists (user, dev, > commits) using cdr...@yahoo-inc.com. > > I'm trying to unsubscribe f

Re: FYI: Backports of Hive UDFs

sure Data, Inc. > http://myui.github.io/ > > 2017-06-02 2:24 GMT+09:00 Alan Gates : > > I'm curious why these can't be backported inside Hive. If someone is > > willing to do the work to do the backport we can check them into the > Hive 1 > > branch. >

Re: FYI: Backports of Hive UDFs

I'm curious why these can't be backported inside Hive. If someone is willing to do the work to do the backport we can check them into the Hive 1 branch. On Thu, Jun 1, 2017 at 1:44 AM, Makoto Yui wrote: > Hi, > > I created a repository for backporting recent Hive UDFs (as of v2.2.0) > to legacy

Re: Compaction - get compacted files

Answers inline. Alan. > On Mar 29, 2017, at 03:08, Riccardo Iacomini > wrote: > > Hello, > I have some questions about the compaction process. I need to manually > trigger compaction operations on a standard partitioned orc table (not ACID), > and be able to get back the list of compacted fi

Re: Adding a Hive Statement of SQL Conformance to the docs

+1. I think this will be great for existing and potential Hive users. Alan. > On Jan 13, 2017, at 9:09 AM, Carter Shanklin wrote: > > I get asked from time to time what Hive's level of SQL conformance is, and > it's difficult to provide a clean answer. Most SQL systems have some detailed > s

Re: HMS connections to meta db

Do you mean the connection between the Hive client and the Hive metastore (if you are using the command line?) or the connection between the metastore server code and the RDBMS. The connection to the RDBMS uses JDBC connection pooling to avoid making and tearing down many connections. The conn

Re: [ANNOUNCE] Apache Hive 2.1.1 Released

Apache keeps just the latest version of each release on the mirrors. You can find all Hive releases at https://archive.apache.org/dist/hive/ if you need 2.1.0. Alan. > On Dec 8, 2016, at 14:40, Stephen Sprague wrote: > > out of curiosity any reason why release 2.1.0 disappeared from > apach

Re: I delete my table in hive,but the file in HDFS not be deleted

Is the table external or managed? External tables do not remove their data when dropped, managed tables do. Alan. > On Dec 6, 2016, at 18:08, 446463...@qq.com wrote: > > I meet a problem in hive. > > I drop a table in hive and the table name ' user_info_20161206' > ---

Re: Compaction in hive

What exactly do you mean by compaction? Hive has a compactor that runs over ACID tables to handle the delta files[1], but I’m guessing you don’t mean that. Are you wanting to concatenate files in existing tables? The usual way to do that is alter table concatenate[2]. Or do you mean somethin

Re: Difference between MANAGED_TABLE and EXTERNAL_TABLE in org.apache.hadoop.hive.metastore.TableType

Hive does not assume that it owns the data for an external table. Thus when an external table is dropped, the data is not deleted. People often use this as a way to load data into a directory in HDFS and then “cast” a table structure over it by creating an external table with that directory as

Re: Problems with Hive Streaming. Compactions not working. Out of memory errors.

I’m guessing that this is an issue in the metastore database where it is unable to read from the transaction tables due to the ingestion rate. What version of Hive are you using? What database are you storing the metadata in? Alan. > On Nov 29, 2016, at 00:05, Diego Fustes Villadóniga wrote:

Re: Adding a New Primitive Type in Hive

A few questions: 1) What operators do you envision UUID supporting? Are there UDFs specific to it? Are there constraints on assuring its uniqueness? 2) A more general form of question 1, what about UUID is different from a string or decimal(20, 0) (either of which should be able to store a UU

Re: kylin log file is to large

This question would be better asked on the kylin lists, since Hive doesn’t start either kylin.sh or diag.sh. Alan. > On Nov 14, 2016, at 06:29, 446463...@qq.com wrote: > > Hi: > I run kylin instance in mine test environment.I find that the kylin log > increases so fast. and my hard disk is f

Re: Hive metadata on Hbase

Some thoughts on this: First, there’s no plan to remove the option to use an RDBMS such as Oracle as your backend. Hive’s RawStore interface is built such that various implementations of the metadata storage can easily coexist. Obviously different users will make different choices about what

Re: Hive in-memory offerings in forthcoming releases

Hive doesn’t usually publish long term roadmaps. I am not familiar with either SAP ASE or Oracle 12c so I can’t say whether Hive is headed in that direction or not. We see LLAP as very important for speeding up Hive processing, especially in the cloud where fetches from blob storage are very

Re: Hive orc use case

; relying on this email's technical content is explicitly disclaimed. The > author will in no case be liable for any monetary damages arising from such > loss, damage or destruction. > > > On 26 September 2016 at 18:54, Alan Gates wrote: > alter table compact

Re: Hive orc use case

netary damages arising from such > loss, damage or destruction. > > > On 26 September 2016 at 17:41, Alan Gates wrote: > ORC does not store data row by row. It decomposes the rows into columns, and > then stores pointer to those columns, as well as a number of indices and >

Re: Hive orc use case

ORC does not store data row by row. It decomposes the rows into columns, and then stores pointer to those columns, as well as a number of indices and statistics, in a footer of the file. Due to the footer, in the simple case you cannot read the file before you close it or append to it. We did

Re: How can I force Hive to start compaction on a table immediately

There’s no way to force immediate compaction. If there are compaction workers in the metastore that aren’t busy they should pick that up immediately. But there isn’t an ability to create a worker thread and start compacting. Alan. > On Aug 1, 2016, at 14:50, Mich Talebzadeh wrote: > > > Ra

Re: Hive compaction didn't launch

gt; deltas, compaction won't happen. So open transaction in partition A blocks > compaction in partition B. That's seems wrong to me. > > On Thu, Jul 28, 2016 at 7:06 PM, Alan Gates wrote: > Hive is doing the right thing there, as it cannot compact the deltas into a >

Re: Hive compaction didn't launch

27, 2016 at 8:46 PM, Igor Kuzmenko wrote: > Thanks for reply, Alan. My guess with Storm was wrong. Today I get same > behavior with running Storm topology. > Anyway, I'd like to know, how can I check that transaction batch was closed > correctly? > > On Wed, Jul 27, 2016

Re: Hive compaction didn't launch

I don’t know the details of how the storm application that streams into Hive works, but this sounds like the transaction batches weren’t getting closed. Compaction can’t happen until those batches are closed. Do you know how you had storm configured? Also, you might ask separately on the stor

Re: Want to be one contributor

] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the >

Re: Want to be one contributor

https://cwiki.apache.org/confluence/display/Hive/Home#Home-ResourcesforContributors is a good place to start. Welcome to Hive. Alan. > On Jul 14, 2016, at 16:01, Alpesh Patel wrote: > > Hi Guys, > > I am part of this group since 1 year. Just an audience and now want to be > contributor in

Re: Using Spark on Hive with Hive also using Spark as its execution engine

> On Jul 11, 2016, at 16:22, Mich Talebzadeh wrote: > > > • If I add LLAP, will that be more efficient in terms of memory usage > compared to Hive or not? Will it keep the data in memory for reuse or not. > Yes, this is exactly what LLAP does. It keeps a cache of hot data (hot co

Re: Delete hive partition while executing query.

at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1036) > ... 15 more > > So second thread definitely waits until first thread completes and than make > a

Re: Spark support for update/delete operations on Hive ORC transactional tables

This JIRA https://issues.apache.org/jira/browse/HIVE-12366 moved the heartbeat logic from the engine to the client. AFAIK this was the only issue preventing working with Spark as an engine. That JIRA was released in 2.0. I want to stress that to my knowledge no one has tested this combination

Re: Delete hive partition while executing query.

Do you have the system configured to use the DbTxnManager? See https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration for details on how to set this up. The transaction manager is what manages locking and makes sure that your queries don’t stomp each

Re: Hive and using Pooled Connections

It’s depends on how it’s configured. In $HIVE_HOME/conf/hive-site.xml you can set the datanucleus.connectionPoolingType variable to BONECP or DBCP. By default it should be using BONECP I believe. (I think NONE is also a valid value, but that doesn’t yet work with ACID turned on.) Alan. > On

Re: Hello, I have an issue about hcatalog

This looks to me like a Hadoop issue rather than Hive. It appears that you cannot connect to HDFS. Have you tried connecting to HDFS outside of Hive/HCatalog? Alan. > On May 18, 2016, at 04:24, Mark Memory wrote: > > hello guys, sorry to bother you. > > I'm using hcatalog to write hive t

Re: Hive SQL based authorization don't have support for group

By group here I assume you mean posix style file group (from HDFS)? No, there isn’t any connection right now. We’d like to be able to pick up groups from HDFS and define those as roles in Hive, but we haven’t added that feature. You’ll need to define a role that includes the members of that gr

Re: Standard Deviation in Hive 2 is still incorrect

Have you filed a JIRA ticket for this? If not, please do so we can track it and fix it. Patches are welcomed as well. :) Alan. > On Apr 4, 2016, at 15:27, Mich Talebzadeh wrote: > > > Hi, > > I reported back in April 2015 that what Hive calls Standard Deviation > Function STDDEV is a poi

Re: Hive footprint

> On Apr 18, 2016, at 15:34, Mich Talebzadeh wrote: > > Hi, > > > If Hive had the ability (organic) to have local variable and stored procedure > support then it would be top notch Data Warehouse. Given its metastore, I > don't see any technical reason why it cannot support these constructs.

Re: [VOTE] Bylaws change to allow some commits without review

+1. Alan. > On Apr 18, 2016, at 06:53, Lars Francke wrote: > > Thanks for the votes so far. Can I get some more people interested please? > > On Fri, Apr 15, 2016 at 7:35 PM, Jason Dere wrote: > ​+1 > From: Lefty Leverenz > Sent: Friday, April 15, 2016 1:36 AM > To: user@hive.apache.org > Su

Re: Automatic Update statistics on ORC tables in Hive

I resolved that as Won’t Fix. See the last comment on the JIRA for my rationale. Alan. > On Mar 28, 2016, at 03:53, Mich Talebzadeh wrote: > > Thanks. This does not seem to be implemented although the Jira says resolved. > It also mentions the timestamp of the last update stats. I do not see

Re: Spark SQL is not returning records for HIVE transactional tables on HDP

> On Mar 14, 2016, at 10:31, Mich Talebzadeh wrote: > > That is an interesting point Alan. > > Does this imply that Hive on Spark (Hive 2 encourages Spark or TEZ) is going > to have an issue with transactional tables? I was partially wrong. In HIVE-12366 Wei refactored this so that the hear

Re: Spark SQL is not returning records for HIVE transactional tables on HDP

I don’t know why you’re seeing Hive on Spark sometimes work with transactional tables and sometimes not. But note that in general it doesn’t work. The Spark runtime in Hive does not send heartbeats to the transaction/lock manager so it will timeout any job that takes longer than the heartbeat

Re: Hive StreamingAPI leaves table in not consistent state

I believe this is an issue in the Storm Hive bolt. I don’t have an Apache JIRA on it, but if you ask on the Hortonworks lists we can connect you with the fix for the storm bolt. Alan. > On Mar 10, 2016, at 04:02, Igor Kuzmenko wrote: > > Hello, I'm using Hortonworks Data Platform 2.3.4 which

Re: Hive Context: Hive Metastore Client

One way people have gotten around the lack of LDAP connectivity in HS2 has been to use Apache Knox. That project’s goal is to provide a single login capability for Hadoop related projects so that users can tie their LDAP or Active Directory servers into Hadoop. Alan. > On Mar 8, 2016, at 16:0

Re: How does Hive do authentication on UDF

There are several Hive authorization schemes, but at the moment none of them restrict function use. At some point we’d like to add that feature to SQL standard authorization (see https://cwiki.apache.org/confluence/display/Hive/SQL+Standard+Based+Hive+Authorization ) but no one has done it yet

Re: Hive-2.0.1 Release date

HPLSQL released with Hive in 2.0, but it hasn’t been integrated as in there isn’t a single parser that handles HPLSQL and Hive SQL. The work to integrate it is significant, and a new feature, so it definitely won’t be done in a bug fix release but in a feature bearing release (that is 2.x, not

Re: Hive 2 performance

HPLSQL is part of Hive, but it is not fully integrated into Hive itself yet. It is still an external module that handles the control flow while passing Hive SQL into Hive via JDBC. We’d like to integrate it fully with Hive’s parser but we’re not there yet. Alan. > On Feb 25, 2016, at 14:26,

Re: Stroing boolean value in Hive table

How the data is stored is up to the storage format (text, rcfile, orc, etc.). Do you mean in your text file you’d like booleans stored as 0 or 1? You could use the case statement to convert them to integers like: select case _boolvar_ when true then 1 when false then 0 end from … Alan. > On

Re: ORC format

ORC does not currently expose a primary key to the user, though we have talked of having it do that. As Mich says the indexing on ORC is oriented towards statistics that help the optimizer plan the query. This can be very important in split generation (determining which parts of the input wil

Re: Indexes in Hive

The issue with this is that HDFS lacks the ability to co-locate blocks. So if you break your columns into one file per column (the more traditional column route) you end up in a situation where 2/3 of the time only one of your columns is being locally read, which results in a significant perfo

Re: Immutable data in Hive

Traditionally data in Hive was write once (insert) read many. You could append to tables and partitions, add new partitions, etc. You could remove data by dropping tables or partitions. But there was no updates of data or deletes of particular rows. This was what was meant by immutable. Hi

Re: Loop if table is not empty

Have you looked at the new procedural HPL/SQL available in recent Hive? If you are using an older version of Hive you can check out hplsql.org, which allows you to install it separately. Alan. Thomas Achache December 28, 2015 at 2:30 Hi everyone, I am running

Re: Attempt to do update or delete using transaction manager that does not support these operations. (state=42000,code=10294)

to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. *From:*Alan Gates [mailto:alanfga...@gmail.com] *Sent:* 22 December 2015 20:39 *To:* user@hive.apache.org *Subject:* Re: Attempt to do update or delete us

Re: Attempt to do update or delete using transaction manager that does not support these operations. (state=42000,code=10294)

Also note that transactions only work with MR or Tez as the backend. The required work to have them work with Spark hasn't been done. Alan. Mich Talebzadeh December 22, 2015 at 9:43 Dropped and created table tt as follows: drop table if exists tt; create table

Re: Difference between ORC and RC files

ORC offers a number of features not available in RC files: * Better encoding of data. Integer values are run length encoded. Strings and dates are stored in a dictionary (and the resulting pointers then run length encoded). * Internal indexes and statistics on the data. This allows for more

Re: Hive partition load

Yes, you can load different partitions simultaneously. Alan. Suyog Parlikar December 17, 2015 at 5:02 Hello everyone, Can we load different partitions of a hive table simultaneously. Is there any locking issues in that if yes what are they? Please find below

Re: How to register permanent function during hive thrift server is running

No restart of the thrift service should be required. Alan. Todd December 3, 2015 at 3:12 Hi, I am using Hive 0.14.0, and have hive thrift server running.During its running, I would use “create function” to add a permanent function, Does hive support this **without rest

Re: [VOTE] Hive 2.0 release plan

Hive 2.0 will not be 100% backwards compatible with 1.x. The following JIRA link shows JIRAs already committed to 2.0 that break compatibility: https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20fixVersion%20%3D%202.0.0%20AND%20%22Hadoop%20Flags%22%20%3D%20%22Incompatible%20

Re: Query performance correlated to increase in delta files?

Are you running the compactor as part of your metastore? It's occasionally compacts the delta files in order to reduce read time. See https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions for details. Alan. Sai Gopalakrishnan November

Re: ORC tables loading

The reads and writes both happen in parallel, so as more nodes are available for read and write, at least in this case, the time stays roughly the same. Alan. James Pirz November 16, 2015 at 21:23 Hi, I am using Hive 1.2 with ORC tables on Hadoop 2.6 on a cluste

Re: hive locking doubt

You are correct that DbTxnManager does not support the explicit locking of tables. Instead it obtains locks based on SQL statements that are being executed. If you use the DummyTxnManager (the default) and set concurrency to true and the lock manager to ZooKeeperHiveLockManager then your loc

Re: clarification please

Ashok Kumar October 28, 2015 at 22:43 hi gurus, kindly clarify the following please * Hive currently does not support indexes or indexes are not used in the query Mostly true. There is a create index, but Hive does not use the resulting index by default.

  1   2   >