Re: Odp: Re: hive 4.0.0-alpha2 external table locks

DbTxnManager doesn't have support for explicit locking. You could use HMSClient to issue an explicit lock request or create a lock record manually in the backend DB `HIVE_LOCKS` table. Maybe you could temporarily restrict access to those tables via Ranger or drop them from the HMS while performi

Re: Odp: Re: hive 4.0.0-alpha2 external table locks

Any other options with DbTxnManager to prevent the access from hive queries to files on hdfs while performing maintenance of external tables ?

Re: Odp: Re: hive 4.0.0-alpha2 external table locks

Unfortunately no. DummyTxnManager doesn't support DB locks, only Zookeeper and in-memory. So you won't be able to transparently lock the external table from other non-DummyTxtManager's.

Re: Odp: Re: hive 4.0.0-alpha2 external table locks

Would it force non-DummyTxtManager instances to not access external tables during exclusive locks? Dnia 1 lutego 2023 15:55 Denys Kuzmenko napisał(a): DbTxnManager comes with DbLockManager implementation that doesn't support explicit lock/unlock functionality. That's

Re: Odp: Re: hive 4.0.0-alpha2 external table locks

DbTxnManager comes with DbLockManager implementation that doesn't support explicit lock/unlock functionality. It was designed to be used only for ACID tables, however, due to code refactor challenges is reused for external tables as well. This comes with few drawbacks like suboptimal behavior (

Re: Odp: Re: hive 4.0.0-alpha2 external table locks

We are using external tables on early stages of ETL. Next we switch to managed tables as we don't need to worry about partition management. With DummyTxnManager we had problems dealing with managed tables. Regards, Wojtek Dnia 1 lutego 2023 10:31 Denys Kuzmenko nap

Re: Odp: Re: hive 4.0.0-alpha2 external table locks

If you are using external tables only, you could try switching to DummyTxnManager. It supports explicit lock requests.

Odp: Re: hive 4.0.0-alpha2 external table locks

QL is not posible: org.apache.hadoop.hive.ql.metadata.HiveException: Current transaction manager does not support explicit lock requests.  Transaction manager:   org.apache.hadoop.hive.ql.lockmgr.DbTxnManager How to lock external table partition in hive4 to perform any maintenance on it?  Regards, Wojtek

Re: hive 4.0.0-alpha2 external table locks

lock requests. Transaction manager: > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager > > How to lock external table partition in hive4 to perform any maintenance > on it? > > Regards, > Wojtek >

hive 4.0.0-alpha2 external table locks

pport explicit lock requests.  Transaction manager:   org.apache.hadoop.hive.ql.lockmgr.DbTxnManager How to lock external table partition in hive4 to perform any maintenance on it?  Regards, Wojtek

external table using delta format

when execute below command in beeline or pyspark, the table metadata is stored successfully in hive metastore with below warning CREATE EXTERNAL TABLE testtable USING DELTA LOCATION 's3a://path/to/delta/delta-folder/' WARN HiveExternalCatalog: Couldn't find corresponding Hive

Re: External table replication in Hive

ta to create a dev metastore). >> >> I'm using Hive version 3.1.2 and it only supports managed table >> replication and lacks external table replication. >> >> I want to try and get external tables also replicate on Hive 3. A couple >> of questions here: &

Re: External table replication in Hive

separate instance (for > example, replicate the prod metadata to create a dev metastore). > > I'm using Hive version 3.1.2 and it only supports managed table > replication and lacks external table replication. > > I want to try and get external tables also replicate on Hive 3. A c

External table replication in Hive

Hi, I want to replicate the Hive metastore to create a separate instance (for example, replicate the prod metadata to create a dev metastore). I'm using Hive version 3.1.2 and it only supports managed table replication and lacks external table replication. I want to try and get external t

Re: Too many S3 API calls for simple queries like select and create external table

parquet >> format on S3. >> >> 2. We copied one parquet file object(data) to a separate S3 >> bucket(target) so now our target bucket contains one parquet file data in >> following hierarchy on S3 :- >> s3:///Test/00_0 (Size of object : 1218 Bytes) >>

Re: Too many S3 API calls for simple queries like select and create external table

; hierarchy on S3 :- > s3:///Test/00_0 (Size of object : 1218 Bytes) > > 3. After that, we have executed following 3 command in Apache Hive 2.1.1 > managed by us on EC2 cluster :- > > (i) Create External table on top of above S3 location :- > > CREATE EXTERNAL TABLE

slow create external table on s3

Hello, I was running some create partitioned external table queries looking like: # 30 partitions inside CREATE EXTERNAL TABLE table1 (value string) PARTITIONED BY (shard string) LOCATION 's3a://path/date=2021-02-01/'; INFO : Completed compiling command(queryId=); Time taken: 7.753 se

[Kafka Handler] Unable to query external table created for Kafka

; CREATE EXTERNAL TABLE brand_affinity_atc_struct ( `vertices` STRUCT >>>) PARTITIONED BY (itemId STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler' TBLPROPERTIES ( "kafka.topic" = &q

Re: rename output error during hive query on AWSs3-external table

Not a solution, but looking at the source code of S3AFileSystem.java (Hadoop 2.8.5), I think the Exception raised inside S3AFileSystem.rename() is swallowed and only a new HiveException is reported. So, in order to find out the root cause, I guess you might need to set Log level to DEBUG and see wh

RE: rename output error during hive query on AWSs3-external table

Check this thread: https://forums.aws.amazon.com/thread.jspa?messageID=922594 From: Souvikk Roy Sent: Tuesday, February 4, 2020 3:06 AM To: user@hive.apache.org Subject: rename output error during hive query on AWSs3-external table Hello, We are using some external tables backed by aws S3

rename output error during hive query on AWSs3-external table

Hello, We are using some external tables backed by aws S3. And we are intermittently getting this error, most likely at the last stage of the reduce, I see some similar posts in net but could not find any solution, Is there any way yo solve it: org.apache.hadoop.hive.ql.metadata.HiveException: Un

drop partition from external table

Hi all, I have a problem dropping a partition in an external table in Hive 1.2.1. The table is an external table with location pointing to an HDFS snapshot. Show create table mydb.my_table: CREATE EXTERNAL TABLE `sax777h.te_ax_hd_inp_rap_por_on`( ... ) PARTITIONED BY ( mydate date ) ROW FORMAT

Trouble debugging Apache Hive DECIMAL type in external table - Decimal precision out of allowed range [1,38]

I'm hoping one or more of the Apache Hive experts monitoring this list can help me debug an issue with DECIMAL type in my external table that has me stumped. Here's the context: --- The Context --- Version: Hive 2.1.1 (from CDH6) I'm mapping a Hive external table to a data stor

Write permission check while adding partition on an external table

Hello, Currently, when you are trying to add a partition to an external table it checks for write permission on partition location. Why is that required? Can we add the partition and check for write permission on the write path and fail the query? Is it something that already exists or an

Re: Hive external table not working in sparkSQL when subdirectories are present

ich.wordpress.com >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explic

Re: Hive external table not working in sparkSQL when subdirectories are present

case be liable for any monetary damages arising from such >> loss, damage or destruction. >> >> >> >>> On Tue, 6 Aug 2019 at 07:58, Rishikesh Gawade >>> wrote: >>> Hi. >>> I have built a Hive external table on top of a directory

Re: Hive external table not working in sparkSQL when subdirectories are present

no case be liable for any monetary damages arising from such loss, damage or destruction. On Tue, 6 Aug 2019 at 07:58, Rishikesh Gawade wrote: > Hi. > I have built a Hive external table on top of a directory 'A' which has > data stored in ORC format. This directory has seve

Re: How to implement partitioned external table.

@Zoltan Appreciate to your replay, i will open a new topic at developer list. Best regards Penghui Zoltan Haindrich 于2019年4月11日周四 下午4:21写道： > > > On 4/4/19 10:22 AM, PengHui Li wrote: > > Hi guys, > > > > I am integrating hive and pulsar(http://pulsar.apache.org < > http://pulsar.apache.org/>)

Re: How to implement partitioned external table.

On 4/4/19 10:22 AM, PengHui Li wrote: Hi guys, I am integrating hive and pulsar(http://pulsar.apache.org ) by HiveStorageHandler and HiveMetaHook, I want to add a feature can divide the data into several parts(pulsar topics) when use hive `PARTITIONED BY`. But  don

How to implement partitioned external table.

Hi guys, I am integrating hive and pulsar(http://pulsar.apache.org) by HiveStorageHandler and HiveMetaHook, I want to add a feature can divide the data into several parts(pulsar topics) when use hive `PARTITIONED BY`. But don't know how to implement it based on HiveStorageHandler and HiveMetaHook.

How to implement partitioned external table with StorageHandler.

Hi guys,

RE: External table data and Ranger Security (doAs=false)

You'll need to create Ranger HDFS Policies to allow the specific user access to the external table directory. Thanks Shawn Weeks -Original Message- From: Kristopher Kane Sent: Friday, March 29, 2019 9:11 AM To: user@hive.apache.org Subject: External table data and Ranger Sec

External table data and Ranger Security (doAs=false)

If using a default external table location, in a cluster with Ranger Authorization, the table location and data are owned by the `hive` user. Since the table is external, there doesn't seem to be a way to delete this data other than impersonating or becoming the `hive` or `hdfs` principal

Re: adding parquet partitions to external table (msck repair table not working)

Hello, Sorry for the late response. Just got back from vacation. Regarding Hive version: 2.3.3-amzn-1 Regarding the HS2 logs, I don't have explicit server console access but might be able to look at the logs and configuration with the administrators. I've just implemented the manual alter table

Re: adding parquet partitions to external table (msck repair table not working)

What version of Hive are you using? You should look at the HS2 logs to see if there were any errors from msck command which ignored such partitions. Also, would be worth to take a look at hive.msck.path.validation configuration in case it is set to "ignore" which silently ignores invalidate partiti

adding parquet partitions to external table (msck repair table not working)

Hello, I have a parquet datasource that I'm writing via spark. For some reason this particular source will not pick up added partitions with msck repair table. However if I alter table tablename / add partition (key=value) then it works. However this is more cumbersome than msck repair table. Is

Re: Create external table with s3 location error

SocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) > > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > at java.net.Socket.connect(Socket.java:58

Re: Create external table with s3 location error

Yes, it works. Thank you very much, Garry From: Suresh Kumar Sethuramaswamy Reply-To: "user@hive.apache.org" Date: Wednesday, November 7, 2018 at 3:10 PM To: "user@hive.apache.org" Subject: Re: Create external table with s3 location error Thanks for the logs. Couple of th

Re: Create external table with s3 location error

.(SessionHiveMetaStoreClient.java:74) From: Suresh Kumar Sethuramaswamy Reply-To: "user@hive.apache.org" Date: Wednesday, November 7, 2018 at 2:50 PM To: "user@hive.apache.org" Subject: Re: Create external table with s3 location error Are you using EMR or Apache hadoop open source? Can you share

Re: Create external table with s3 location error

Are you using EMR or Apache hadoop open source? Can you share your hive megastore logs? On Wed, Nov 7, 2018, 2:19 PM Garry Chen hi All, > > I am try to create a external table using s3 as location > but failed. I add my access key and security key in hive-site.xml and

Create external table with s3 location error

hi All, I am try to create a external table using s3 as location but failed. I add my access key and security key in hive-site.xml and reboot the server. Any suggestion? hive> create external table kv (key int, values string) location 's3://cu-iclick/test&

Re: External Table Creation is slow/hangs

ter). > > Our (very roundabout) solution so far is to copy all the files in that > master folder to another directory, delete the files, create the external > table when the directory is empty, and to transfer the files back. We need > to keep the original directory name as other

External Table Creation is slow/hangs

(very roundabout) solution so far is to copy all the files in that master folder to another directory, delete the files, create the external table when the directory is empty, and to transfer the files back. We need to keep the original directory name as other processes depend on it and can&

Re: Hive External Table on particular set of files.

they are maintianing two folders DATA and METADATA. Where DATA > holds actual data and METADATA holds RowCount of that day's data.We need to > create external table on top of copied data, where it only consider > *.json.gz extension files only, excluding other file extensions. We do

Hive External Table on particular set of files.

ed to create external table on top of copied data, where it only consider *.json.gz extension files only, excluding other file extensions. We dont want to copy files into another location since they are large in size. We also tried INPUT_ FILENAME virtual column, it didn't work. Any sugge

Re: Hive External Table with Zero Bytes files

rongly Folder Name, it 's same DAY=20180325 (Folder) and same has Filename. actually in our upstream, our source table is partitioned by Date. Whenever a table is partitioned, we see Zero Byte. Now when we create external table with partitioned by columns and fire select query no data is retur

Re: Hive External Table with Zero Bytes files

ur upstream, our source table > is partitioned by Date. Whenever a table is partitioned, we see Zero Byte. > Now when we create external table with partitioned by columns and fire > select query no data is returned. . If I delete manually those files (Zero > Bytes), we were able to read.

Re: Hive External Table with Zero Bytes files

en we create external table with partitioned by columns and fire select query no data is returned. . If I delete manually those files (Zero Bytes), we were able to read. /Mahender On 4/28/2018 6:36 AM, Thai Bui wrote: Your external table is referencing the .../day=201803250 location which is

Re: Hive External Table with Zero Bytes files

Thanks Thai. I have mentioned wrongly Folder Name, it 's same DAY=20180325 (Folder) and same has Filename. actually in our upstream, our source table is partitioned by Date. Whenever a table is partitioned, we see Zero Byte. Now when we create external table with partitioned by columns and

Re: Hive External Table with Zero Bytes files

> We are copying data from upstream system into our storage S3. As part of > copy, directories along with Zero bytes files are been copied. Is this exactly the same issue as the previous thread or a different one?

Re: Hive External Table with Zero Bytes files

Your external table is referencing the .../day=201803250 location which is empty. Point your table to the capital .../DAY=201803250 and you should be able to read the data there. Also, it looks like you want external partitioned table. You’ll need to create an external table with a partition

Re: Hive External Table with Zero Bytes files

Gentle Ping. Please help me on below issue. Has any one faced same issue On 4/27/2018 1:28 AM, Mahender Sarangam wrote: Hi, Can any one faced issue while fetching data from external table. We are copying data from upstream system into our storage S3. As part of copy, directories along with

Hive External Table with Zero Bytes files

Hi, Can any one faced issue while fetching data from external table. We are copying data from upstream system into our storage S3. As part of copy, directories along with Zero bytes files are been copied. Source File Format is in JSON format. Below is Folder Hierarchy Structure DATE

Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

> - Original Message - > From: "Anup Tiwari" > To: user@hive.apache.org > Sent: Thursday, April 12, 2018 10:50:23 AM > Subject: Re: Unable to read hive external table data which is linked to s3 > after upgradation from 2.1.1 to 2.3.3 > > > > > &g

Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

: Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3 Hi Richard , After looking at hive 2.3.3 logs i found that when we are loading all configuration parameters then below message is getting printed :- Values omitted for securi

Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

-- Original Message - > From: "Anup Tiwari" > To: user@hive.apache.org > Sent: Thursday, April 12, 2018 10:06:33 AM > Subject: Re: Unable to read hive external table data which is linked to s3 > after upgradation from 2.1.1 to 2.3.3 > > > We are not using EM

Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

6:33 AM Subject: Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3 We are not using EMR. Also we have set below params for accessing s3 bucket in hive-site.xml which are same as what we have set in hive 2.1.1. * fs.s3a.acces

Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

se whatever AWS login you > use would have to have the correct permissions in the IAM. > > - Original Message - > From: "Anup Tiwari" > To: user@hive.apache.org > Sent: Thursday, April 12, 2018 9:11:37 AM > Subject: Unable to read hive external table data which

Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

have the correct permissions in the IAM. - Original Message - From: "Anup Tiwari" To: user@hive.apache.org Sent: Thursday, April 12, 2018 9:11:37 AM Subject: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3 Hi All, When i am

Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

Hi All, When i am trying to read s3 linked external table in 2.3.3 ; i am getting errors.It was working properly in 2.1.1. please find below details and let me know if i am missing something :- *Hadoop Version :- *2.8.0 *Query :-* select log_date,count(1) as cnt from test.tt1 group by

Loading data from one swift container into external table pointing to another container fails

Hello, Loading data from one swift container into external table pointing to another container is failing for me. hive> load data inpath 'swift://mycontainer.default/source_test' into table my_test; Loading data to table default.my_test Failed with exception copyFiles: error while

Re: Cannot create external table on S3; class S3AFileSystem not found

gt; > > > Elliot. > > > > > > > > > On Sat, 9 Dec 2017 at 00:08, Scott Halgrim > > > > > wrote: > > > > > > Hi, > > > > > > > > > > > > I’ve been struggling with this for a few hours, hopefu

Re: Cannot create external table on S3; class S3AFileSystem not found

grim > wrote: > >> Hi, >> >> I’ve been struggling with this for a few hours, hopefully somebody here >> can help me out. >> >> We have a lot of data in parquet format on S3 and we want to use Hive to >> query it. I’m running on ubuntu and we ha

Re: Cannot create external table on S3; class S3AFileSystem not found

ave a lot of data in parquet format on S3 and we want to use Hive to > > > query it. I’m running on ubuntu and we have a MySQL metadata store on AWS > > > RDS. > > > > > > The command in the hive client I’m trying to run is: > > > > CREATE EXT

Re: Cannot create external table on S3; class S3AFileSystem not found

his for a few hours, hopefully somebody here > can help me out. > > We have a lot of data in parquet format on S3 and we want to use Hive to > query it. I’m running on ubuntu and we have a MySQL metadata store on AWS > RDS. > > The command in the hive client I’m trying to run

Cannot create external table on S3; class S3AFileSystem not found

is: > CREATE EXTERNAL TABLE > my_schema.my_table > (account_id INT, > action VARCHAR(282), > another_id INT > yaid INT, > `date` TIMESTAMP, > deleted_at TIMESTAMP, > id INT, > lastchanged TIMESTAMP, > thing_index DOUBLE, > old_id INT, > parent_id INT, > ru

Fwd: Pro and Cons of using HBase table as an external table in HIVE

Sorry. Need to send via right email address. Begin forwarded message: From: Michael Segel mailto:mse...@segel.com>> Subject: Re: Pro and Cons of using HBase table as an external table in HIVE Date: June 9, 2017 at 7:37:22 AM CDT To: user@hive.apache.org<mailto:user@hive.apache.org>

Re: Pro and Cons of using HBase table as an external table in HIVE

if you just wrote your code in Java. > On Jun 7, 2017, at 5:13 AM, Ramasubramanian Narayanan > mailto:ramasubramanian.naraya...@gmail.com>> > wrote: > > Hi, > > Can you please let us know Pro and Cons of using HBase table as an external > table in HIVE. > &g

Re: Pro and Cons of using HBase table as an external table in HIVE

shdown, then you will be faster, >> assuming that the query triggers an implied range scan. >> This would be a simple thing. However keep in mind that you’re going to >> generate a map/reduce job (unless using a query engine like Tez) where you >> wouldn’t if you just wrote your code in

Re: Pro and Cons of using HBase table as an external table in HIVE

erate a map/reduce job (unless using a query engine like Tez) where you > wouldn’t if you just wrote your code in Java. > > > > > > On Jun 7, 2017, at 5:13 AM, Ramasubramanian Narayanan < > ramasubramanian.naraya...@gmail.com> wrote: > > > > Hi, > > > >

Re: Pro and Cons of using HBase table as an external table in HIVE

using a query engine like Tez) where you wouldn’t if you just wrote your code in Java. > On Jun 7, 2017, at 5:13 AM, Ramasubramanian Narayanan > wrote: > > Hi, > > Can you please let us know Pro and Cons of using HBase table as an external > table in HIVE. > > Wi

Re: Pro and Cons of using HBase table as an external table in HIVE

eate and update them physically. You can read more about it here https://sonra.io/2017/05/15/dimensional-modeling-and-kimball-data-marts-in-the-age-of-big-data-and-hadoop/ On 07/06/2017 11:13, Ramasubramanian Narayanan wrote: Hi, Can you please let us know Pro and Cons of using HBase table as a

Re: Pro and Cons of using HBase table as an external table in HIVE

As I know using Hive on Hbase can only be done through Hive Example hive> create external table MARKETDATAHBASE (key STRING, TICKER STRING, TIMECREATED STRING, PRICE STRING) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES ("hbase.column

Pro and Cons of using HBase table as an external table in HIVE

Hi, Can you please let us know Pro and Cons of using HBase table as an external table in HIVE. Will there be any performance degrade when using Hive over HBase instead of using direct HIVE table. The table that I am planning to use in HBase will be master table like account, customer. Wanting

Re: Hive Tez on External Table running on Single Mapper

> > 'skip.header.line.count'='1', Trying removing that config option. I've definitely seen footer markers disabling file splitting, possibly header also does. Cheers, Gopal

Re: Hive Tez on External Table running on Single Mapper

7;totalSize'='20971513935', > 'transient_lastDdlTime'='1485091440'); > > > On 1/24/2017 10:27 AM, Mahender Sarangam wrote: >> When i see properties of external table, STORED AS INPUTFORMAT >> 'org.apache.hadoop.mapred.TextInputFormat

Re: Hive Tez on External Table running on Single Mapper

Here are the table properties TBLPROPERTIES ( 'numFiles'='1', 'serialization.null.format'='', 'skip.header.line.count'='1', 'totalSize'='20971513935', 'transient_lastDdlTime'='1485091440

Re: Hive Tez on External Table running on Single Mapper

Here is the screenshot [cid:part1.CDBA25BE.CDE6CB4A@outlook.com] On 1/24/2017 10:27 AM, Mahender Sarangam wrote: When i see properties of external table, STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' On 1/23/2017 2:25 PM, Gopal Vijayaraghavan wrote: We have 20 G

Re: Hive Tez on External Table running on Single Mapper

When i see properties of external table, STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' On 1/23/2017 2:25 PM, Gopal Vijayaraghavan wrote: >> We have 20 GB txt File, When we have created external table on top of 20 >> Gb file, we see Tez is creating onl

Re: Hive Tez on External Table running on Single Mapper

> We have 20 GB txt File, When we have created external table on top of 20 > Gb file, we see Tez is creating only one mapper. For an uncompressed file, that is very strange. Is this created as "STORED AS TEXTFILE" or some other strange format? Cheers, Gopal

Hive Tez on External Table running on Single Mapper

Hi, We have 20 GB txt File, When we have created external table on top of 20 Gb file, we see Tez is creating only one mapper. We have applied setting like reducing distribution block size less 128 MB. Manually Set Mappers and Reducer but of no use. We are using Tez 0.7 version, is there any

create external table on s3 results in 403 failure if the folder does not exist

Hi, I am currently using hive 2.1 on emr5.0 and every time I tried to create an external table against a folder that does not exist on s3, hive returns s3 403 exception even though I can read/write to that bucket using awscli on the same host. I remembered this working on an earlier version of

Re: hive external table on gzip

> Hi I have huge gzip on hdfs and |I’d like to create an external table on top > of them > Any code example? Cheers > Ps > I cannot use snappy or lzo for some constraints > > -- > Kind regards > Mario Amatucci > CG TB PS GDC PRAGUE THINK BIG >

Re: hive external table on gzip

pretty simple --1 Move gz file or files into HDFS: Multiple files can be in that staging directory with hdfs dfs -copyFromLocal /*.gz hdfs://rhes564:9000/data/stg/ --2 Create an external table. Just one will do CREATE EXTERNAL TABLE stg_t2 ... STORED AS TEXTFILE LOCATION '/data/stg/

RE: hive external table on gzip

Hi I have huge gzip on hdfs and |I'd like to create an external table on top of them Any code example? Cheers Ps I cannot use snappy or lzo for some constraints -- Kind regards Mario Amatucci CG TB PS GDC PRAGUE THINK BIG

Re: alter partitions on hive external table

04/zzz/data.txt > > > > hdfs dfs -put t /tmp > > > > > ---- > > -- hive > > > > > > > n We’re creating the ext

RE: alter partitions on hive external table

a/yada/yada/cust3/yyy/20150204/zzz/data.txt hdfs dfs -put t /tmp -- hive ---- n We’re creating the external table with the requested partition columns create external

RE: alter partitions on hive external table

… are just logical connections between certain values and specific directories … From: Markovitz, Dudu [mailto:dmarkov...@paypal.com] Sent: Monday, June 06, 2016 6:07 PM To: user@hive.apache.org Subject: RE: alter partitions on hive external table Hi Raj 1. I don’t understand the reason

RE: alter partitions on hive external table

Hi Raj 1. I don’t understand the reason for this change, can you please elaborate? 2. External table is just an interface. Instructions for how to read existing data. Partitions of external table are just a logical connections between certain values and a specific directories

Re: alter partitions on hive external table

?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 6 June 2016 at 08:18, raj hive wrote: > Hi Mich, > > table type is external table. Yes, I am doing this for certain queries > where userid as the most significant column. > > On Mon, Jun 6, 2016 at 12:35 PM, Mich Tal

Re: alter partitions on hive external table

Hi Mich, table type is external table. Yes, I am doing this for certain queries where userid as the most significant column. On Mon, Jun 6, 2016 at 12:35 PM, Mich Talebzadeh wrote: > That order datetime/userid/customerId looks more natural to me. > > Two questions: > > What

Re: alter partitions on hive external table

That order datetime/userid/customerId looks more natural to me. Two questions: What is the type of table in Hive? Are you doing this for certain queries where you think userid as the most significant column is going to help queries better? HTH Dr Mich Talebzadeh LinkedIn * https://www.li

Re: alter partitions on hive external table

Hi The first idea pops up is: 1. HDFS commands to copy your existing structure and data to support a new partitions structure. 2. Create a new on temporary hive external table 3. (optional) if you created temporary table then drop old one and insert ... select from temporary table

alter partitions on hive external table

Hi friends, I have created partitions on hive external tables. partitions on datetime/userid/customerId. now i have to change the order of the partitions for the existing data for all the dates. order of the partition is custerid/userid/datetime. Anyone can help me, how to alter the partitions

Is it possible to use external table on top of Elasticsearch index for arbitrary FTS

I know that external table can be defined like this CREATE EXTERNAL TABLE artists ( id BIGINT, nameSTRING, links STRUCT) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource' = 'radio/artists', 'es.query&#x

Re: Hive2 create external table in kerberos enabled cluster

are enabled on HDFS and Hive plugins. In case I create external table and Location in HDFS is empty or there are small amount of files then create table is ok via beeline. But in case there are loads of files in example 10 000 or more then I can not create external table via beeline. But I can

Hive2 create external table in kerberos enabled cluster

Hi I have kerberos and ranger enabled cluster. Ranger plugins are enabled on HDFS and Hive plugins. In case I create external table and Location in HDFS is empty or there are small amount of files then create table is ok via beeline. But in case there are loads of files in example 10 000 or

Re: Create external table

I have kerberos enabled in my cluster. In case I create external table using beeline I see from hdfs namenode log that it does Kerberos auth for every single file I guess. It may be the reason why creating external hive table fails in case I have loads of directories and files under them

Re: Create external table

/files_10k/f1966.txt ... Connected to: Apache Hive (version 1.2.1.2.3.4.0-3485) Driver: Hive JDBC (version 1.2.1.2.3.4.0-3485) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://bigdata29.webmedia.int:1/> create external table files_10k (i int) row format delimited fie

Re: Create external table

://hadoopnn1.estpak.ee:1/def> create external table files_10k (i int) row format delimited fields terminated by '\t' location '/user/margusja/files_10k'; No rows affected (0.197 seconds) 2: jdbc:hive2://hadoopnn1.estpak.ee:1/def> drop table files_10k; No rows

  1   2   3   4   >