Re: Values go to a wrong table during recovery.
Hi Denis, This doesn't sound like a known bug to me. Your hypothesis is reasonable, since WALs use a surrogate ID, which maps to table ID/tablet information, when read back. It is possible that it incorrectly interprets this mapping and replays data into the wrong table. Given the amount of testing we do, my instinct is to think this is unlikely, but if we can confirm this bug, it would definitely be a very critical one. To rule out some scenarios, is it possible that your clients are writing to the wrong tables? Have you ever seen a failure affecting a table which does not exist (like what might happen if there's an off-by-one error in the WAL code)? Or affecting the metadata tables? Can you reproduce this error reliably, or can you share the relevant ingest code which can reproduce this failure? Also, what kind of tablet server failures are you experiencing when this happens? If you could file a bug report at https://issues.apache.org/browse/ACCUMULO with any details and/or attachments to help us address the issue, we would greatly appreciate it. This seems like something we'd want to fix pretty quickly. Thanks! -- Christopher L Tubbs II http://gravatar.com/ctubbsii On Wed, Feb 18, 2015 at 6:26 PM, Denis wrote: > Hello. > > Few times I noticed that some tables have values they cannot have, and > those entries have timestamp close to a tabletserver failure time. > (I mean wrong format, one table has msgpack values at least 10 bytes > long and another table has 1-byte values and after a failure I read > one or two 1-byte values in the table where I expect to read msgpack). > > I suspect that during recovery process, when WAL is being read, some > entries are inserted to a wrong table. > > May be it is a know bug as I am still using Accumulo 1.6.1 >
Re: Values go to a wrong table during recovery.
Sorry, that link should be: https://issues.apache.org/jira/browse/ACCUMULO -- Christopher L Tubbs II http://gravatar.com/ctubbsii On Wed, Feb 18, 2015 at 6:42 PM, Christopher wrote: > Hi Denis, > > This doesn't sound like a known bug to me. Your hypothesis is reasonable, > since WALs use a surrogate ID, which maps to table ID/tablet information, > when read back. It is possible that it incorrectly interprets this mapping > and replays data into the wrong table. Given the amount of testing we do, > my instinct is to think this is unlikely, but if we can confirm this bug, > it would definitely be a very critical one. > > To rule out some scenarios, is it possible that your clients are writing > to the wrong tables? Have you ever seen a failure affecting a table which > does not exist (like what might happen if there's an off-by-one error in > the WAL code)? Or affecting the metadata tables? > > Can you reproduce this error reliably, or can you share the relevant > ingest code which can reproduce this failure? Also, what kind of tablet > server failures are you experiencing when this happens? > > If you could file a bug report at > https://issues.apache.org/browse/ACCUMULO with any details and/or > attachments to help us address the issue, we would greatly appreciate it. > This seems like something we'd want to fix pretty quickly. > > Thanks! > > > -- > Christopher L Tubbs II > http://gravatar.com/ctubbsii > > On Wed, Feb 18, 2015 at 6:26 PM, Denis wrote: > >> Hello. >> >> Few times I noticed that some tables have values they cannot have, and >> those entries have timestamp close to a tabletserver failure time. >> (I mean wrong format, one table has msgpack values at least 10 bytes >> long and another table has 1-byte values and after a failure I read >> one or two 1-byte values in the table where I expect to read msgpack). >> >> I suspect that during recovery process, when WAL is being read, some >> entries are inserted to a wrong table. >> >> May be it is a know bug as I am still using Accumulo 1.6.1 >> > >
Re: Values go to a wrong table during recovery.
On 2/18/15, Christopher wrote: > To rule out some scenarios, is it possible that your clients are writing to > the wrong tables? That was the first idea, so I added assert()'s to the code of the writers few days ago. No assert was triggered, but some invalid values appear after new tserver failure. > Have you ever seen a failure affecting a table which does > not exist (like what might happen if there's an off-by-one error in the WAL > code)? Or affecting the metadata tables? No. Also, no tables were created or deleted during last two months. > Can you reproduce this error reliably, or can you share the relevant ingest > code which can reproduce this failure? I will think how to reproduce it. What could be special about the code: inserts are performed to few (5..8) tables at once (one data table + few index tables) but no MultiTableBatchWriter is used. Few BatchWriter`s (one per table) are created and flushed consequentially, in the same thread. For Accumulo 1.4 it was a performance optimization, if worked faster than MultiTableBatchWriter. Not sure if it is so for 1.6.1, this code was not changed after migration to 1.6.1. In all cases with invalid values the index tables were affected (one of the index table had values typical for another of the index tables). > Also, what kind of tablet server failures are you experiencing when this > happens? Spontaneous power-offs. There is something wrong with the power units so every 2-3 days one of the servers suddenly turns off and reboots.
Re: Values go to a wrong table during recovery.
https://issues.apache.org/jira/browse/ACCUMULO-3603 -Eric On Wed, Feb 18, 2015 at 7:12 PM, Denis wrote: > On 2/18/15, Christopher wrote: > > > To rule out some scenarios, is it possible that your clients are writing > to > > the wrong tables? > That was the first idea, so I added assert()'s to the code of the > writers few days ago. No assert was triggered, but some invalid values > appear after new tserver failure. > > > Have you ever seen a failure affecting a table which does > > not exist (like what might happen if there's an off-by-one error in the > WAL > > code)? Or affecting the metadata tables? > No. > Also, no tables were created or deleted during last two months. > > > Can you reproduce this error reliably, or can you share the relevant > ingest > > code which can reproduce this failure? > > I will think how to reproduce it. > What could be special about the code: inserts are performed to few > (5..8) tables at once (one data table + few index tables) but no > MultiTableBatchWriter is used. Few BatchWriter`s (one per table) are > created and flushed consequentially, in the same thread. For Accumulo > 1.4 it was a performance optimization, if worked faster than > MultiTableBatchWriter. Not sure if it is so for 1.6.1, this code was > not changed after migration to 1.6.1. > In all cases with invalid values the index tables were affected (one > of the index table had values typical for another of the index > tables). > > > Also, what kind of tablet server failures are you experiencing when this > happens? > Spontaneous power-offs. There is something wrong with the power units > so every 2-3 days one of the servers suddenly turns off and reboots. >
Re: Values go to a wrong table during recovery.
I forget to say that HDFS Datanodes and Accumulo Tablet Servers share the same machines. When a machine powers off, one Tablet Server and one Datanode became unavailable. On 2/19/15, Eric Newton wrote: > https://issues.apache.org/jira/browse/ACCUMULO-3603 > > -Eric > > > On Wed, Feb 18, 2015 at 7:12 PM, Denis wrote: > >> On 2/18/15, Christopher wrote: >> >> > To rule out some scenarios, is it possible that your clients are >> > writing >> to >> > the wrong tables? >> That was the first idea, so I added assert()'s to the code of the >> writers few days ago. No assert was triggered, but some invalid values >> appear after new tserver failure. >> >> > Have you ever seen a failure affecting a table which does >> > not exist (like what might happen if there's an off-by-one error in the >> WAL >> > code)? Or affecting the metadata tables? >> No. >> Also, no tables were created or deleted during last two months. >> >> > Can you reproduce this error reliably, or can you share the relevant >> ingest >> > code which can reproduce this failure? >> >> I will think how to reproduce it. >> What could be special about the code: inserts are performed to few >> (5..8) tables at once (one data table + few index tables) but no >> MultiTableBatchWriter is used. Few BatchWriter`s (one per table) are >> created and flushed consequentially, in the same thread. For Accumulo >> 1.4 it was a performance optimization, if worked faster than >> MultiTableBatchWriter. Not sure if it is so for 1.6.1, this code was >> not changed after migration to 1.6.1. >> In all cases with invalid values the index tables were affected (one >> of the index table had values typical for another of the index >> tables). >> >> > Also, what kind of tablet server failures are you experiencing when >> > this >> happens? >> Spontaneous power-offs. There is something wrong with the power units >> so every 2-3 days one of the servers suddenly turns off and reboots. >> >
Re: Values go to a wrong table during recovery.
I updated ACCUMULO-3603 w/ details about an experiment I ran. On Wed, Feb 18, 2015 at 9:44 PM, Eric Newton wrote: > https://issues.apache.org/jira/browse/ACCUMULO-3603 > > -Eric > > > On Wed, Feb 18, 2015 at 7:12 PM, Denis wrote: > >> On 2/18/15, Christopher wrote: >> >> > To rule out some scenarios, is it possible that your clients are >> writing to >> > the wrong tables? >> That was the first idea, so I added assert()'s to the code of the >> writers few days ago. No assert was triggered, but some invalid values >> appear after new tserver failure. >> >> > Have you ever seen a failure affecting a table which does >> > not exist (like what might happen if there's an off-by-one error in the >> WAL >> > code)? Or affecting the metadata tables? >> No. >> Also, no tables were created or deleted during last two months. >> >> > Can you reproduce this error reliably, or can you share the relevant >> ingest >> > code which can reproduce this failure? >> >> I will think how to reproduce it. >> What could be special about the code: inserts are performed to few >> (5..8) tables at once (one data table + few index tables) but no >> MultiTableBatchWriter is used. Few BatchWriter`s (one per table) are >> created and flushed consequentially, in the same thread. For Accumulo >> 1.4 it was a performance optimization, if worked faster than >> MultiTableBatchWriter. Not sure if it is so for 1.6.1, this code was >> not changed after migration to 1.6.1. >> In all cases with invalid values the index tables were affected (one >> of the index table had values typical for another of the index >> tables). >> >> > Also, what kind of tablet server failures are you experiencing when >> this happens? >> Spontaneous power-offs. There is something wrong with the power units >> so every 2-3 days one of the servers suddenly turns off and reboots. >> > >
Re: Values go to a wrong table during recovery.
Denis, After our tests, it's highly unlikely this problem is in the server. If you can provide a test client that has ever replicated the problem, please attach it to the ticket. Otherwise, we'll close the ticket unless someone else can reproduce the problem. -Eric On Fri, Feb 20, 2015 at 1:46 PM, Keith Turner wrote: > I updated ACCUMULO-3603 w/ details about an experiment I ran. > > On Wed, Feb 18, 2015 at 9:44 PM, Eric Newton > wrote: > >> https://issues.apache.org/jira/browse/ACCUMULO-3603 >> >> -Eric >> >> >> On Wed, Feb 18, 2015 at 7:12 PM, Denis wrote: >> >>> On 2/18/15, Christopher wrote: >>> >>> > To rule out some scenarios, is it possible that your clients are >>> writing to >>> > the wrong tables? >>> That was the first idea, so I added assert()'s to the code of the >>> writers few days ago. No assert was triggered, but some invalid values >>> appear after new tserver failure. >>> >>> > Have you ever seen a failure affecting a table which does >>> > not exist (like what might happen if there's an off-by-one error in >>> the WAL >>> > code)? Or affecting the metadata tables? >>> No. >>> Also, no tables were created or deleted during last two months. >>> >>> > Can you reproduce this error reliably, or can you share the relevant >>> ingest >>> > code which can reproduce this failure? >>> >>> I will think how to reproduce it. >>> What could be special about the code: inserts are performed to few >>> (5..8) tables at once (one data table + few index tables) but no >>> MultiTableBatchWriter is used. Few BatchWriter`s (one per table) are >>> created and flushed consequentially, in the same thread. For Accumulo >>> 1.4 it was a performance optimization, if worked faster than >>> MultiTableBatchWriter. Not sure if it is so for 1.6.1, this code was >>> not changed after migration to 1.6.1. >>> In all cases with invalid values the index tables were affected (one >>> of the index table had values typical for another of the index >>> tables). >>> >>> > Also, what kind of tablet server failures are you experiencing when >>> this happens? >>> Spontaneous power-offs. There is something wrong with the power units >>> so every 2-3 days one of the servers suddenly turns off and reboots. >>> >> >> >
Re: Values go to a wrong table during recovery.
You said that you were operating this on 1.4. Is this the exact same cluster or is it just the same code you were using? Did you have walogs laying around when you upgraded? Did you upgrade through 1.5 or straight from 1.4 to 1.6? On Fri, Feb 20, 2015 at 1:46 PM, Keith Turner wrote: > I updated ACCUMULO-3603 w/ details about an experiment I ran. > > On Wed, Feb 18, 2015 at 9:44 PM, Eric Newton > wrote: > >> https://issues.apache.org/jira/browse/ACCUMULO-3603 >> >> -Eric >> >> >> On Wed, Feb 18, 2015 at 7:12 PM, Denis wrote: >> >>> On 2/18/15, Christopher wrote: >>> >>> > To rule out some scenarios, is it possible that your clients are >>> writing to >>> > the wrong tables? >>> That was the first idea, so I added assert()'s to the code of the >>> writers few days ago. No assert was triggered, but some invalid values >>> appear after new tserver failure. >>> >>> > Have you ever seen a failure affecting a table which does >>> > not exist (like what might happen if there's an off-by-one error in >>> the WAL >>> > code)? Or affecting the metadata tables? >>> No. >>> Also, no tables were created or deleted during last two months. >>> >>> > Can you reproduce this error reliably, or can you share the relevant >>> ingest >>> > code which can reproduce this failure? >>> >>> I will think how to reproduce it. >>> What could be special about the code: inserts are performed to few >>> (5..8) tables at once (one data table + few index tables) but no >>> MultiTableBatchWriter is used. Few BatchWriter`s (one per table) are >>> created and flushed consequentially, in the same thread. For Accumulo >>> 1.4 it was a performance optimization, if worked faster than >>> MultiTableBatchWriter. Not sure if it is so for 1.6.1, this code was >>> not changed after migration to 1.6.1. >>> In all cases with invalid values the index tables were affected (one >>> of the index table had values typical for another of the index >>> tables). >>> >>> > Also, what kind of tablet server failures are you experiencing when >>> this happens? >>> Spontaneous power-offs. There is something wrong with the power units >>> so every 2-3 days one of the servers suddenly turns off and reboots. >>> >> >> >
Re: Values go to a wrong table during recovery.
> If you can provide a test client that has ever replicated the problem, > please attach it to the ticket. I have seen it 3 times within a month timeframe, so I do not know how to reproduce it reliable. Perhaps, I have to backup walogs next time and look into them. > Is this the exact same cluster or is it just the same code you were using? Same code, another cluster. > Did you have walogs laying around when you upgraded? In 1.4-cluster (and first time in 1.6-cluster) I had walogs enabled for data tables and disabled for index tables. There was a bug in 1.4, if a tablet had empty walog there were some startup issues (tablet remains offline or something like this), and it happened often with index tables (hmm, the same tables I have this problem). So, in 1.4-cluster I disabled walog and ran full reindex periodically. After running 1.6-cluster some time I enabled walogs for all tables as the new cluster have less reliable hardware, which reboots from time to time. > Did you upgrade through 1.5 or straight from 1.4 to 1.6? >From 1.4 to 1.6. But it was not upgrade, it was copy of .rf files to a new cluster and then importdirectory. On 2/20/15, John Vines wrote: > You said that you were operating this on 1.4. Is this the exact same > cluster or is it just the same code you were using? Did you have walogs > laying around when you upgraded? Did you upgrade through 1.5 or straight > from 1.4 to 1.6? > > On Fri, Feb 20, 2015 at 1:46 PM, Keith Turner wrote: > >> I updated ACCUMULO-3603 w/ details about an experiment I ran. >> >> On Wed, Feb 18, 2015 at 9:44 PM, Eric Newton >> wrote: >> >>> https://issues.apache.org/jira/browse/ACCUMULO-3603 >>> >>> -Eric >>> >>> >>> On Wed, Feb 18, 2015 at 7:12 PM, Denis wrote: >>> On 2/18/15, Christopher wrote: > To rule out some scenarios, is it possible that your clients are writing to > the wrong tables? That was the first idea, so I added assert()'s to the code of the writers few days ago. No assert was triggered, but some invalid values appear after new tserver failure. > Have you ever seen a failure affecting a table which does > not exist (like what might happen if there's an off-by-one error in the WAL > code)? Or affecting the metadata tables? No. Also, no tables were created or deleted during last two months. > Can you reproduce this error reliably, or can you share the relevant ingest > code which can reproduce this failure? I will think how to reproduce it. What could be special about the code: inserts are performed to few (5..8) tables at once (one data table + few index tables) but no MultiTableBatchWriter is used. Few BatchWriter`s (one per table) are created and flushed consequentially, in the same thread. For Accumulo 1.4 it was a performance optimization, if worked faster than MultiTableBatchWriter. Not sure if it is so for 1.6.1, this code was not changed after migration to 1.6.1. In all cases with invalid values the index tables were affected (one of the index table had values typical for another of the index tables). > Also, what kind of tablet server failures are you experiencing when this happens? Spontaneous power-offs. There is something wrong with the power units so every 2-3 days one of the servers suddenly turns off and reboots. >>> >>> >> >
Re: Values go to a wrong table during recovery.
FYI, this was fixed in 1.6.2 https://issues.apache.org/jira/browse/ACCUMULO-3182 Denis wrote: There was a bug in1.4, if a tablet had empty walog there were some startup issues (tablet remains offline or something like this), and it happened often with index tables (hmm, the same tables I have this problem).
Re: Values go to a wrong table during recovery.
Hm, I think, that bug is much older, which has been fixed in 1.5 or next minor 1.4.x. Unfortunately, I did not put the bug number in code comment (only conn.tableOperations.setProperty(tableName, "table.walog.enabled", "false") // avoid Accumulo 1.4 bug with recovery from empty logs (fixed in 1.5) ), so it was long time before ACCUMULO-3182's Sep-2014 and I have seen here in the mailing list or in the bugtracker that it was fixed in 1.5) On 2/20/15, Josh Elser wrote: > FYI, this was fixed in 1.6.2 > https://issues.apache.org/jira/browse/ACCUMULO-3182 > > Denis wrote: >> There was a bug in1.4, if a tablet had empty walog there were some >> startup issues (tablet remains offline or something like this), and it >> happened often with index tables (hmm, the same tables I have this >> problem). >
Re: Values go to a wrong table during recovery.
I think it was ACCUMULO-1364 On 2/20/15, Denis wrote: > Hm, I think, that bug is much older, which has been fixed in 1.5 or > next minor 1.4.x. Unfortunately, I did not put the bug number in code > comment (only > > conn.tableOperations.setProperty(tableName, "table.walog.enabled", > "false") // avoid Accumulo 1.4 bug with recovery from empty logs > (fixed in 1.5) > > ), so it was long time before ACCUMULO-3182's Sep-2014 and I have seen > here in the mailing list or in the bugtracker that it was fixed in > 1.5) > > On 2/20/15, Josh Elser wrote: >> FYI, this was fixed in 1.6.2 >> https://issues.apache.org/jira/browse/ACCUMULO-3182 >> >> Denis wrote: >>> There was a bug in1.4, if a tablet had empty walog there were some >>> startup issues (tablet remains offline or something like this), and it >>> happened often with index tables (hmm, the same tables I have this >>> problem). >> >
Re: Values go to a wrong table during recovery.
Well, regardless, there was an outstanding issue where an empty WAL could have been made and would have blocked recovery. It's possible what I fixed was a regression, but it still sounds like what you saw :) Denis wrote: Hm, I think, that bug is much older, which has been fixed in 1.5 or next minor 1.4.x. Unfortunately, I did not put the bug number in code comment (only conn.tableOperations.setProperty(tableName, "table.walog.enabled", "false") // avoid Accumulo 1.4 bug with recovery from empty logs (fixed in 1.5) ), so it was long time before ACCUMULO-3182's Sep-2014 and I have seen here in the mailing list or in the bugtracker that it was fixed in 1.5) On 2/20/15, Josh Elser wrote: FYI, this was fixed in 1.6.2 https://issues.apache.org/jira/browse/ACCUMULO-3182 Denis wrote: There was a bug in1.4, if a tablet had empty walog there were some startup issues (tablet remains offline or something like this), and it happened often with index tables (hmm, the same tables I have this problem).
Re: Values go to a wrong table during recovery.
On Fri, Feb 20, 2015 at 2:33 PM, Denis wrote: > > Did you have walogs laying around when you upgraded? > > In 1.4-cluster (and first time in 1.6-cluster) I had walogs enabled > for data tables and disabled for index tables. > > There was a bug in 1.4, if a tablet had empty walog there were some > startup issues (tablet remains offline or something like this), and it > happened often with index tables (hmm, the same tables I have this > problem). > > So, in 1.4-cluster I disabled walog and ran full reindex periodically. > After running 1.6-cluster some time I enabled walogs for all tables as > the new cluster have less reliable hardware, which reboots from time > to time. > > So just to confirm, you have seen this issue appear while walogs are enabled for all tables, yes? Any chance you can share your code that writes across several BatchWriters? -- Sean
Re: Values go to a wrong table during recovery.
> So just to confirm, you have seen this issue appear while walogs are enabled > for all tables, yes? Yes. > Any chance you can share your code that writes across several BatchWriters? The writer is: // way faster than Connector.createMultiTableBatchWriter class MyMultiTableBatchWriter(maxMemory: Long, maxLatency: Long, maxWriteThreads: Int)(implicit val conn: Connector) { private[this] val writers = new scala.collection.mutable.HashMap[String, BatchWriter] def getBatchWriter(tableName: String): BatchWriter = synchronized { writers.getOrElseUpdate(tableName, conn.createBatchWriter(tableName, maxMemory, maxLatency, maxWriteThreads)) } def flush() = writers.foreach(_._2.flush()) def close() = writers.foreach(_._2.close()) } But the full code which creates/uses the indexes is more than 1Mb of scala code... On 2/20/15, Sean Busbey wrote: > On Fri, Feb 20, 2015 at 2:33 PM, Denis wrote: > >> > Did you have walogs laying around when you upgraded? >> >> In 1.4-cluster (and first time in 1.6-cluster) I had walogs enabled >> for data tables and disabled for index tables. >> >> There was a bug in 1.4, if a tablet had empty walog there were some >> startup issues (tablet remains offline or something like this), and it >> happened often with index tables (hmm, the same tables I have this >> problem). >> >> So, in 1.4-cluster I disabled walog and ran full reindex periodically. >> After running 1.6-cluster some time I enabled walogs for all tables as >> the new cluster have less reliable hardware, which reboots from time >> to time. >> >> > So just to confirm, you have seen this issue appear while walogs are > enabled for all tables, yes? > > Any chance you can share your code that writes across several BatchWriters? > > -- > Sean >
Re: Values go to a wrong table during recovery.
It is always used from the single thread On 2/20/15, Denis wrote: >> So just to confirm, you have seen this issue appear while walogs are >> enabled for all tables, yes? > > Yes. > >> Any chance you can share your code that writes across several >> BatchWriters? > > The writer is: > > // way faster than Connector.createMultiTableBatchWriter > class MyMultiTableBatchWriter(maxMemory: Long, maxLatency: Long, > maxWriteThreads: Int)(implicit val conn: Connector) { > private[this] val writers = new > scala.collection.mutable.HashMap[String, BatchWriter] > def getBatchWriter(tableName: String): BatchWriter = synchronized { > writers.getOrElseUpdate(tableName, > conn.createBatchWriter(tableName, maxMemory, maxLatency, > maxWriteThreads)) > } > def flush() = writers.foreach(_._2.flush()) > def close() = writers.foreach(_._2.close()) > } > > But the full code which creates/uses the indexes is more than 1Mb of > scala code... > > On 2/20/15, Sean Busbey wrote: >> On Fri, Feb 20, 2015 at 2:33 PM, Denis wrote: >> >>> > Did you have walogs laying around when you upgraded? >>> >>> In 1.4-cluster (and first time in 1.6-cluster) I had walogs enabled >>> for data tables and disabled for index tables. >>> >>> There was a bug in 1.4, if a tablet had empty walog there were some >>> startup issues (tablet remains offline or something like this), and it >>> happened often with index tables (hmm, the same tables I have this >>> problem). >>> >>> So, in 1.4-cluster I disabled walog and ran full reindex periodically. >>> After running 1.6-cluster some time I enabled walogs for all tables as >>> the new cluster have less reliable hardware, which reboots from time >>> to time. >>> >>> >> So just to confirm, you have seen this issue appear while walogs are >> enabled for all tables, yes? >> >> Any chance you can share your code that writes across several >> BatchWriters? >> >> -- >> Sean >> >