hive.compactor.max.num.delta
This lets control how many deltas are opened at once. By default it’s 500
which may be too much.
So the compactor will use this do exactly what Owen is suggesting.
The current impl will do everything sequentially but better than OOM.
Eugene
From: Owen O'Malley
Ok, if you are against the wall, I'd suggest looking at the CompactorMR
class, which is the class that the Metastore uses to launch the compactor
jobs. You'll need to write code to call it with Table, StorageDescriptor,
and ValidTxnList to do the minor compaction on a set of transactions. For
I've already tried giving the compactor 256+ gigabytes of memory. All that
changes is how long for it run out of memory.
Thanks
Shawn Weeks
From: Owen O'Malley
Sent: Monday, September 17, 2018 3:37:09 PM
To: user@hive.apache.org
Subject: Re: Hive Compaction
How many files is it trying to merge at once? By far the easiest thing to
do will be to give the compactor job more heap to work with. In theory you
could do multiple rounds of minor compaction to get around the problem.
Unfortunately, the tool isn't designed to do that and I'm worried that
Tried the Binary thing but since Hive Streaming in HDP 2.6 doesn't support
Binary column types that's not going to work. See HIVE-18613.
Thanks
Shawn Weeks
From: Shawn Weeks
Sent: Monday, September 17, 2018 12:28:25 PM
To: user@hive.apache.org
Subject: Re:
Hive doesn't have a "grant select on db.*" option, which is what I think
you're looking for here.
Yes i am looking something like this only and since it is not available,
does that mean i have to go for each table ?
I am asking because we have many DBs and a lot of tables within each DB so
is
2018-09-17 11:20:26,404 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error
running child : java.lang.OutOfMemoryError: Java heap space
at
com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:864)
at
Shawn,
Can you provide the stack trace that you get with the OOM?
Thanks,
Owen
On Mon, Sep 17, 2018 at 9:27 AM Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:
> Hi Shawn
>
> You might be running into issues related to huge protobuf objects from
> huge string columns. Without
Hi Shawn
You might be running into issues related to huge protobuf objects from huge
string columns. Without
https://issues.apache.org/jira/plugins/servlet/mobile#issue/ORC-203 there isn’t
an option other than providing sufficiently large memory. If you can reload the
data with binary type
Let me start off by saying I've backed myself into a corner and would rather
not reprocess the data if possible. I have a Hive Transactional table in Hive
1.2.1 H that was loaded via NiFi Hive Streaming with a fairly large String
column containing XML Documents. Awful I know and I'm working on
What you are seeing is correct behavior. Select on the database means the
user can see objects in the database (ie, tables, views). To see contents
of those objects you have to grant access on those objects. Hive doesn't
have a "grant select on db.*" option, which is what I think you're looking
Hi Alan,
I have given select access of a database to a role which is attached to a
user but after this also that user is not able to execute select statements
on tables of that database. But if i provide access at table level then
that is working. Can you please help me here ?
Hive Version :
13 matches
Mail list logo
