Re: Issues with hive storage based authorization

version is 1.2.1 and Hadoop 2.7.3. I tried to build sentry 1.5.1 from github for the configuration mentioned however not able to build it successfully as its giving error for hdfs-name-node plugin. Will look at it in more details. Thanks for all your suggestions and help. Regards, Vijay On Wed

Re: Issues with hive storage based authorization

privilege on the warehouse dir was only able to create database. However that works without even enabling the hive storage based authorization. So not sure how hive storage base authorization will provided additional security. Definitely I am missing something. Please suggest. Thanks, Vijay On Thu

Re: Issues with hive storage based authorization

No its not. Thanks, Vijay On Thu, Nov 9, 2017 at 1:09 PM, Jörn Franke wrote: > Is your Hadoop cluster kerberized? > > On 9. Nov 2017, at 06:57, Vijay Toshniwal > wrote: > > Hi Team, > > > > I am facing issues while configuring hive storage based authorization. I

Issues with hive storage based authorization

://stackoverflow.com/ questions/43734947/does-the-storage-based-authorization- or-sql-standards-based-hive-authorization-w?rq=1 where the default authorization is not working as expected. Request you to provide your inputs on the same. Thanks, Vijay

Bucketed table info

Hi. If I have an orc table bucketed and sorted on a column, where does hive keep the mapping from column value to bucket? Specifically, if I know the column value, and need to find the specific hdfs file, is there an api to do this? Related, is there any documentation on how the read path works f

RE: on duplicate update equivalent?

Dimensions change, and I'd rather do update than recreate a snapshot. On 23-Sep-2016 17:23, "Markovitz, Dudu" wrote: > If these are dimension tables, what do you need to update there? > > > > Dudu > > > > *From:* Vijay Ramachandran [mailto:vi...@li

Re: on duplicate update equivalent?

On Fri, Sep 23, 2016 at 3:47 PM, Mich Talebzadeh wrote: > What is the use case for UPSERT in Hive. The functionality does not exist > but there are other solutions. > > Are we talking about a set of dimension tables with primary keys hat need > to be updated (existing rows) or inserted (new rows)

on duplicate update equivalent?

Hello. Is there a way to write a query with a behaviour equivalent to mysql's "on duplicate update"? i.e., try to insert, and if key exists, update the row instead? thanks,

Re: [ANNOUNCE] New Hive Committer - Wei Zheng

Congrats Wei Zheng!! On Mar 10, 2016 6:57 AM, "Vikram Dixit K" wrote: > The Apache Hive PMC has voted to make Wei Zheng a committer on the Apache > Hive Project. Please join me in congratulating Wei. > > Thanks > Vikram. >

Create ORC Table on Tez Failed

cution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask Can anyone please let me know what could be the likely cause of the issue and possible way to get rid this error message. We are kind of stuck. Please let us know your thoughts. Thanks & Regards Vijay -- The con

Create ORC Table on Tez Failed

cution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask Can anyone please let me know what could be the likely cause of the issue and possible way to get rid this error message. We are kind of stuck. Please let us know your thoughts. Thanks & Regards Vijay -- The con

Unsubscribe

Unsubscribe

Re: Hive 0.11 & Hive 0.12 - Hive MySQL Metastore is not used for INSERT

I have modified the title of the JIRA to properly reflect the issue faced - Hive 0.11 & Hive 0.12 - derby.log and TempStatsStore are created when using MySQL Metastore Regards Naga On Sun, Feb 2, 2014 at 2:15 PM, Naga Vijay wrote: > Hi, > > I have checked my hive-site.xml ; it i

Re: Hive 0.11 & Hive 0.12 - Hive MySQL Metastore is not used for INSERT

tabase connexion( connexion > string) because in this file, that happend connexion to the metastore when > you start Hive. > > Hope this helps you. > regards. > Matouk > > > 2014-02-02 Naga Vijay : > > Hi, >> >> I am facing this situation for the below m

Hive 0.11 & Hive 0.12 - Hive MySQL Metastore is not used for INSERT

Hi, I am facing this situation for the below mentioned hql file - > Hive MySQL Metastore is used for table t1 > Hive MySQL Metastore is not used for table t2 (derby.log is created in the directory) -- -- database pp_test_hive_metastore drop database if exists pp_test_hive_metastore c

Re: How to load a web log file (text format) to Hive with compression

Hive has built-in support for gzipped text files so you don't need to do anything special with them (meaning you just load them like regular files). However, there is one important limitation with this approach these files are not splittable. Depending on the size of your log files and how many of

Re: Use RANK OVER PARTITION function in Hive 0.11

As the error message states: "One ore more arguments are expected," you have to pass a column to the rank function. On Wed, Jul 17, 2013 at 1:12 AM, Jérôme Verdier wrote: > Hi Richa, > > I have tried a simple query without joins, etc > > SELECT RANK() OVER (PARTITION BY mag.co_societe ORDER

Re: Enhancing Query Join to speed up Query

> igor > > decide.com > > > > > > On Thu, Jun 13, 2013 at 11:31 AM, Naga Vijay wrote: > >> > >> Sure, Will do > >> > >> > >> On Thu, Jun 13, 2013 at 10:42 AM, Stephen Sprague > >> wrote: > >>> > >>

Re: Enhancing Query Join to speed up Query

dt >> FROM >> ( SELECT >> item_id, create_dt >> FROM >> A >> WHERE >>item_id = 'I001' >>AND >> category_name = 'C001' >>

Enhancing Query Join to speed up Query

Hi, Which of the two query options is better? SELECT a.item_id, a.create_dt FROM a JOIN b ON (a.item_id = b.item_id) WHERE a.item_id = 'I001' ANDa.category_name = 'C001'; - or - SELECT a.item_id, a.create_dt FROM a JOIN b ON (a.item_id = b.item_id AND a.item_id = 'I001') WHERE

Which approach for handling heavily nested JSONs ?

Hi, I have been dealing with some heavily nested and complex JSON data. It has all sorts of combinations like: Struct I wanted to know which approach you find better: using the SerDe or using the UDFs. In my opinion the two approaches can be compared in the following way. P

Re: Rolling MAU computation

For both simplicity and efficiency, I'd recommend making the mausummary table partitioned on date and generate the MAU data each day. There is no reason to generate MAU data for a given day more than once (unless you find some problems with the source data or something). On Fri, Oct 12, 2012 at 1:

Re: Custom UserDefinedFunction in Hive

1:26 AM, Raihan Jamal wrote: > Hi Vijay, > > Thanks for the suggestion, If upgrading to Hive was under my control then I > would have done for sure, but I am working in a company and they are running > Hive 0.6 on all the cluster, And I told them to upgrade the Hive version but >

Re: Custom UserDefinedFunction in Hive

Given the implementation of the UDF, I don't think hive would be able to use partition pruning. Especially the version you're using. I'd really recommend upgrading to a later version that has the hiveconf support. That can save a lot of trouble rather than trying to get things working on 0.6 On Tu

Re: Passing date as command line arguments

There can't be a space in the variable name. Try this: ${hiveconf:start_date} On Fri, Aug 3, 2012 at 11:06 PM, Techy Teck wrote: > Can anyone help me out on this? > > I am stuck on this and I have no clue what wrong I am doing? > > > > > On Fri, Aug 3, 2012 at 3:48 PM, Techy Teck wrote: >> >> I

Re: HBASE and HIVE Integration

Bejoy Ks wrote: > Hi Vijay > > Your current error looks like some issue with the Select query. Is the > select query working as desired? > hive> SELECT * FROM pokes where foo=98; > > Regards > Bejoy KS > > -- > *From:* vijay shi

Re: HBASE and HIVE Integration

Operator.java:546) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) ... 8 more Caused by: or FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask hive> On Thu, Jul 26, 2012 at 9:55 AM, Bejoy Ks wrote: > Hi Vijay > > Is your hbase working in

Re: HBASE and HIVE Integration

Sent from my iPhone On Jul 25, 2012, at 8:58 PM, vijay shinde wrote: > Hi Bejoy, > > I made some changes as per your suggetion. > > Here is the error from the > http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004 Job: > > Error: java.lan

Re: HBASE and HIVE Integration

8_0004 with errors FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask hive> I am lost need help badly! Vijay On Wed, Jul 25, 2012 at 9:47 AM, Bejoy Ks wrote: > Hi Vijay > > You have provided the hbase master directly. (It is fine for si

Re: HBASE and HIVE Integration

apache.hadoop.hive.ql.exec.MapRedTask * Let me know if you need any additonal information. Thanks, Vijay On Wed, Jul 25, 2012 at 5:30 AM, Bejoy KS wrote: > ** > Hi Vijay > > Can you share more details like > > The CDH Version/Hive version you are using > > Steps you fol

HBASE and HIVE Integration

I am facing issue while executing HIVE queries on HBASE-HIVE integration. I followed the wiki hbase-hive integration https://cwiki.apache.org/Hive/hbaseintegration.html I have already passed all the required jars for auxpath in hive-site.xml file. I am using Cloudera CDH demo VM.. Any help would b

Re: Invalid Table Alias

There seems to be something wrong with the query (the red highlighted portion). What is mi in "abs(datediff(mi"? On Sun, Jul 22, 2012 at 11:28 AM, Techy Teck wrote: > Can anyone give some sort of pointers or suggestions why the below query is > giving me exception and the same query works fine in

Re: Exploding Array of Struct in Hive

prod_and_ts) prod_and_ts ) tt2 LATERAL VIEW explode(split(timestamps, "#")) exploded_table2 as prod_and_ts_split; Thanks, Vijay On Wed, Jul 18, 2012 at 2:24 PM, Raihan Jamal wrote: > CREATE TABLE IF NOT EXISTS TestingTable2 > > ( > > USER_ID BIGINT, > > PURCHAS

Re: Get only the date from DateType data

string to_date(string timestamp) does the trick. The official hive UDF page has all the useful functions that are available: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF -Vijay On Wed, Jul 18, 2012 at 2:55 PM, Raihan Jamal wrote: > And CREATED_TIME is string data t

Re: Anything wrong with this query?

When you say LATERAL VIEW explode(purchased_item) exploded_table as prod_and_ts The last alias is an alias to a column; not a table. explode will only produce one column per row; it cannot produce multiple columns. I hope that helps. On Tue, Jul 10, 2012 at 8:31 PM, Raihan Jamal wrote: > I have t

Re: Invalid Function rank in HiveQL

2012-07-09 07:28:47 > > > 1015826235 2600035533812012-07-09 07:09:56 > > > 1015826235 2600035533832012-07-09 06:58:47 > > > 1015826235 2600035533892012-07-09 06:54:37 > > > > > > Raihan Jamal > > > > On

Re: Invalid Function rank in HiveQL

so. But when I am >>>>> again >>>>> running the below query- >>>>> >>>>> SELECT buyer_id, item_id, created_time >>>>> FROM ( >>>>> SELECT buyer_id, item_id, Rank(buyer_id) as rk, created_time >

Re: What's wrong with this query?

t; > > > Raihan Jamal > > > > On Tue, Jul 10, 2012 at 1:28 PM, Vijay wrote: >> >> You need to use from_unixtime() >> >> On Tue, Jul 10, 2012 at 1:01 PM, Raihan Jamal >> wrote: >> > Here timestamps is a string and I always get NULL

Re: What's wrong with this query?

You need to use from_unixtime() On Tue, Jul 10, 2012 at 1:01 PM, Raihan Jamal wrote: > Here timestamps is a string and I always get NULL in the second column when > I try to get date out of the timestamp. Anything wrong I am doing? > > > select A.timestamps, to_date(A.timestamps) from > (select u

Re: Invalid Function rank in HiveQL

hive has no built-in rank function. you'd need to use a user-defined function (UDF) to simulate it. there are a few custom implementations on the net that you can leverage. On Mon, Jul 9, 2012 at 10:40 PM, Raihan Jamal wrote: > What's wrong with the below query. > > > SELECT buyer_id, item_id, cr

Re: Always mysterious "Could not obtain block" error for large jobs [Hive 0.8.1/ Hadoop 1.0.0 on Mac Mini Cluster]

Another critical variable to check is dfs.datanode.max.xcievers. The default value is 256. You should bump that up to 4096 or higher. -Vijay On Thu, Mar 1, 2012 at 7:07 PM, Abhishek Parolkar wrote: > Hi There! >   I have been doing an interesting experiment of building mac mini cluster &

Re: Regex and serde with hive

If the format is simply delimited like this, you don't need to use the RegexSerde. Hive's default format with the right "FIELDS TERMINATED BY" setting will work great. -Vijay On Thu, Dec 22, 2011 at 8:49 PM, Raghunath, Ranjith wrote: > I have been struggling with this

Cannot use explode with sub queries?

nd any examples with explode and sub queries. Thanks, Vijay

Re: Having trouble using regex serde

The log lines are in some kind of JSON format though. The regex needs to literally match the log lines. On Fri, Sep 30, 2011 at 8:52 PM, Jonathan wrote: > Hi, > I am  trying to parse an apache2 log using > the 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'. I am able to load > the tables usin

Re: SerDe question

option is to probably parse the text file and generate a different file with each word on a separate line and then load it into hive. Hope that helps, Vijay On Tue, Sep 27, 2011 at 6:45 PM, Mark Kerzner wrote: > Hi, Hive experts, > > Would you see what I am doing wrong? For a simple test

unsubscribe

Questions about Hive Database/Schema support

y way to move existing tables into a new database from CLI but it should be possible to do this directly by modifying the metastore right? Is there anything to watch out for? Thanks, Vijay

Re: Hive & MS SQL Server

If this is important enough you should probably patch the source code to get this working and then submit a patch so those changes can be rolled into a future version of hive. My 2 cents. -Vijay On Mon, Mar 28, 2011 at 10:30 AM, Viral Bajaria wrote: > Appan, > Not sure if you meant to em

Re: Hive too slow?

gging through the web UI can tell you how much time is spent on map vs reduce. It can also provide some insight into the I/O operations. Good luck! -Vijay On Mon, Mar 7, 2011 at 9:24 PM, abhishek pathak wrote: > I suspected as such.My system is a Core2Duo,1.86 Ghz.I understand that > map-re

Question about Transform and M/R scripts

erid, time, url FROM weblogs ) weblogs reduce weblogs.userid, weblogs.time, weblogs.url using 'counter.pl' as user, count; Thanks, Vijay

Re: Hive queries consuming 100% cpu

It spawned only one reduce task. The reducer ran for 117 seconds, which seems long for this query. On Thu, Feb 3, 2011 at 2:31 PM, Viral Bajaria wrote: > Hey Vijay, > You can go to the mapred ui, normally it runs on port 50030 of the namenode > and see how many map jobs got created for y

Hive queries consuming 100% cpu

ls or easy options to profile hive query execution? Thanks in advance, Vijay

Lots of Failed/Killed Task Attempts while running successful hive queries

lways seem proportional to the total number of tasks (and to the length of the entire job to an extent). Does anybody have any insight into this? Thanks, Vijay

Re: Help with last 30 day unique user query

nd_date; > > (repeated N times to cover each day of your time window). > > alex > > On Thu, Oct 14, 2010 at 11:36 PM, Vijay wrote: > >> Hi, I need help with this scenario. We have a table of events which has >> columns date, event (not important for this discuss

Re: Help with last 30 day unique user query

t a string to epoch time and back to a > string. e.g., > > select from_unixtime(unix_timestamp('2010-10-10', '-MM-dd') + > 60*60*24*30, '-MM-dd') from src limit 1; > > will given you the date which is 30 days later than 2010-10-10. > &g

Help with last 30 day unique user query

also be as optimized as possible as this table could be huge. Thanks, Vijay