Encryption (or the lack of it) doesn't explain missing HFiles.
Most likely if you are having a problem with encryption, this will manifest
as follows: HFiles will be present. However, you will find many
IOExceptions in the regionserver logs as they attempt to open the HFiles
but fail because the d
4 AM 02:21
To: user@hbase.apache.org
Subject: Re: HBase file encryption, inconsistencies observed and data loss
Encryption (or the lack of it) doesn't explain missing HFiles.
Most likely if you are having a problem with encryption, this will manifest as
follows: HFiles will be present. How
se of the information contained herein in any way (including, but
> not limited to, total or partial disclosure, reproduction, or dissemination)
> by persons other than the intended recipient(s) is prohibited. If you receive
> this e-mail in error, please notify the sender by phone or
l
>
>
>
> org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogWriter
>
>
>
>
>
> hbase.regionserver.wal.encryption
>
> false
>
>
>
>
>
> Regards
>
> -Shankar
>
>
> This e-mail and its attachments contain confidential information
io I tried (Anoop suggested), with the below
> > configuration (instead of deleting the below 3 config paramters
> >
> > Kepp all but make only 'hbase.regionserver.wal.encryption=false') the
> > encrypted wal file is getting processed
> >
> > Successfully, and the query
t; >
> org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogWriter
> > >
> > >
> > >
> > >
> > >
> > > hbase.regionserver.wal.encryption
> > >
> > > true
> > >
> > >
> > >
> > >
oldWALs folder.
>> > >
>> > > So when I revert back the below 3 configuration in Region Server side
>> and
>> > > restart, since the WAL is already moved to oldWALS/ folder,
>> > >
>> > > So it will not get processed.
>> > >
t;>> hbase.regionserver.hlog.writer.impl
>> org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogWriter
>>>
>>>
>>>
>>>
>>>
>>> hbase.regionserver.wal.encryption
>>>
>>> true
>>>
>>>
>> --
efore the
>>>> RegionServer machine went down) is not coming,
>>>>
>>>> One more thing what I observed is even when the WAL file not
successfully
>>>> processed then also it is moving to /oldWALs folder.
>>>>
>>>> So when I revert
RS_LOG_REPLAY_OPS-host1:60020-0]
>>>>> wal.HLogSplitter: Processed 0 edits across 0 regions; log
> file=hdfs://hacluster/hbase/WALs/host1,60020,1406383007151-splitting/host1%2C60020%2C1406383007151.1406383069334.meta
>>>>> is corrupted = false progress failed = false
&
29, 2014 07:36
> To: user@hbase.apache.org
> Subject: Re: HBase file encryption, inconsistencies observed and data
> loss
>
> Yes in btw the restart the config was changed. In steps the #4 was that.
> Wal encryption config is changed to false. Well that is ok but the
> reader
Hi,
After step 4 ( i.e disabling of WAL encryption, removing
SecureProtobufReader/Writer and restart), read of encrypted WAL fails mainly
due to EOF exception at Basedecoder. This is not considered as error and these
WAL are being moved to /oldWALs.
Following is observed in log files:
2014-07-
Looking at HLogSplitter#getNextLogLine() :
try {
return in.next();
} catch (EOFException eof) {
// truncated files are expected if a RS crashes (see HBASE-2643)
LOG.info("EOF from hlog " + path + ". continuing");
return null;
The EOFException is not treated a
In BaseDecoder#rethrowEofException() :
if (!isEof) throw ioEx;
LOG.error("Partial cell read caused by EOF: " + ioEx);
EOFException eofEx = new EOFException("Partial cell read");
eofEx.initCause(ioEx);
throw eofEx;
throwing EOFException would not propagate the "Partial cell
Let's take this to JIRA
On Wed, Jul 30, 2014 at 12:50 PM, Ted Yu wrote:
> In BaseDecoder#rethrowEofException() :
>
> if (!isEof) throw ioEx;
>
> LOG.error("Partial cell read caused by EOF: " + ioEx);
>
> EOFException eofEx = new EOFException("Partial cell read");
>
> eofEx.initC
I logged HBASE-11620 for this issue.
If my proposal is accepted, I can provide a patch.
Cheers
On Wed, Jul 30, 2014 at 12:56 PM, Andrew Purtell
wrote:
> Let's take this to JIRA
>
>
> On Wed, Jul 30, 2014 at 12:50 PM, Ted Yu wrote:
>
> > In BaseDecoder#rethrowEofException() :
> >
> > if (
16 matches
Mail list logo
