[jira] [Comment Edited] (PHOENIX-3811) Do not disable index on write failure by default
[
https://issues.apache.org/jira/browse/PHOENIX-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005218#comment-16005218
]
James Taylor edited comment on PHOENIX-3811 at 5/10/17 7:00 PM:
High level summary of changes: turns off automatic index rebuilding by default,
leaves indexes active upon a write failure, and provides a means of users
replaying a commit after a write failure to ensure the index is consistent with
data table. Would you have any spare cycles to review, [~tdsilva]?
Here's some more detail on the changes:
* Turns off the background partial index rebuild/catchup task by default for a
table. The reason is that users will typically have some kind of retry strategy
themselves (for example, a message queue that retries). They need this as when
a commit exception occurs, some of the data rows may have been written while
others will not have been (regardless of what state the index is in wrt the
data table). What ever retry mechanism is in use, these retries will also get
the index back in sync (see below for a new mechanism for mutable tables).
* Provides a means for the client to retry a commit at the timestamp at which
it was originally submitted. This is important for mutable data as otherwise
the retried commits may overwrite successful commits that occurred later. This
is accomplished by a) including the server timestamp at which the data rows
were (or would have been) committed in CommitException and b) Adds a new
connection property, {{PhoenixRuntime.REPLAY_AT_ATTRIB}}, which specifies a
timestamp at which the commit will occur and tells the system to ignore later
data updates (to ensure your index remains in sync with your data table).
* Provides an option (the default) to keep an index active even after a write
failure occurs. Many use cases are essentially down without the secondary index
in place and would rather the index be behind by a few rows wrt the data table
while the retries are occurring. This option is configurable globally with the
{{QueryServices.INDEX_FAILURE_DISABLE_INDEX}} config property and on a table by
table basis through the
{{PhoenixIndexFailurePolicy.DISABLE_INDEX_ON_WRITE_FAILURE}} table descriptor
property.
* Provides an option to turn on the partial rebuild index task on a
table-by-table basis (false by default). This option is orthogonal now to
whether an index remains active or is disabled. Note that if the existing
global {{PhoenixIndexFailurePolicy.INDEX_FAILURE_HANDLING_REBUILD_ATTRIB}}
config property is false, then the background thread won't run so the table
property won't matter. By default, the global property is true while the
table-by-table property is false to allow the user to turn the automatic
rebuild on for a particular table.
* Lowers the default frequency at which we look for indexes which need to be
partially rebuilt from every 10 seconds to once per minute.
* Fixes MutableIndexFailureIT test failures and adds more for the above new
options.
FYI, [~lhofhansl], [~apurtell], [~mvanwely].
was (Author: jamestaylor):
High level summary of changes: turns off automatic index rebuilding by default,
leaves indexes active upon a write failure, and provides a means of users
replaying a commit after a write failure to ensure the index is consistent with
data table. Would you have any spare cycles to review, [~tdsilva]?
Here's some more detail on the changes:
- Turns off the background partial index rebuild/catchup task by default for a
table. The reason is that users will typically have some kind of retry strategy
themselves (for example, a message queue that retries). They need this as when
a commit exception occurs, some of the data rows may have been written while
others will not have been (regardless of what state the index is in wrt the
data table). What ever retry mechanism is in use, these retries will also get
the index back in sync (see below for a new mechanism for mutable tables).
- Provides a means for the client to retry a commit at the timestamp at which
it was originally submitted. This is important for mutable data as otherwise
the retried commits may overwrite successful commits that occurred later. This
is accomplished by a) including the server timestamp at which the data rows
were (or would have been) committed in CommitException and b) Adds a new
connection property, {{PhoenixRuntime.REPLAY_AT_ATTRIB}}, which specifies a
timestamp at which the commit will occur and tells the system to ignore later
data updates (to ensure your index remains in sync with your data table).
- Provides an option (the default) to keep an index active even after a write
failure occurs. Many use cases are essentially down without the secondary index
in place and would rather the index be behind by a few rows wrt the data table
while the retries are occurring. This option is configu
[jira] [Comment Edited] (PHOENIX-3811) Do not disable index on write failure by default
[
https://issues.apache.org/jira/browse/PHOENIX-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005218#comment-16005218
]
James Taylor edited comment on PHOENIX-3811 at 5/10/17 7:19 PM:
High level summary of changes: turns off automatic index rebuilding by default,
leaves indexes active upon a write failure, and provides a means of users
replaying a commit after a write failure to ensure the index is consistent with
data table. Would you have any spare cycles to review, [~tdsilva]?
Here's some more detail on the changes:
* Turns off the background partial index rebuild/catchup task by default for a
table. The reason is that users will typically have some kind of retry strategy
themselves (for example, a message queue that retries). They need this as when
a commit exception occurs, some of the data rows may have been written while
others will not have been (regardless of what state the index is in wrt the
data table). What ever retry mechanism is in use, these retries will also get
the index back in sync (see below for a new mechanism for mutable tables).
* Provides a means for the client to retry a commit at the timestamp at which
it was originally submitted. This is important for mutable data as otherwise
the retried commits may overwrite successful commits that occurred later. This
is accomplished by a) including the server timestamp at which the data rows
were (or would have been) committed in CommitException and b) Adds a new
connection property, {{PhoenixRuntime.REPLAY_AT_ATTRIB}}, which specifies a
timestamp at which the commit will occur and tells the system to ignore later
data updates (to ensure your index remains in sync with your data table).
* Provides an option (the default) to keep an index active even after a write
failure occurs. Many use cases are essentially down without the secondary index
in place and would rather the index be behind by a few rows wrt the data table
while the retries are occurring. This option is configurable globally with the
{{QueryServices.INDEX_FAILURE_DISABLE_INDEX}} config property and on a table by
table basis through the
{{PhoenixIndexFailurePolicy.DISABLE_INDEX_ON_WRITE_FAILURE}} table descriptor
property.
* Provides an option to turn on the partial rebuild index task on a
table-by-table basis (false by default). This option is orthogonal now to
whether an index remains active or is disabled (i.e. the index can remain
active *and* be partially rebuilt/caught up in the background). Note that if
the existing global
{{PhoenixIndexFailurePolicy.INDEX_FAILURE_HANDLING_REBUILD_ATTRIB}} config
property is false, then the background thread won't run so the table property
won't matter. By default, the global property is true while the table-by-table
property is false to allow the user to turn the automatic rebuild on for a
particular table.
* Lowers the default frequency at which we look for indexes which need to be
partially rebuilt from every 10 seconds to once per minute.
* Fixes MutableIndexFailureIT test failures and adds more for the above new
options.
FYI, [~lhofhansl], [~apurtell], [~mvanwely].
was (Author: jamestaylor):
High level summary of changes: turns off automatic index rebuilding by default,
leaves indexes active upon a write failure, and provides a means of users
replaying a commit after a write failure to ensure the index is consistent with
data table. Would you have any spare cycles to review, [~tdsilva]?
Here's some more detail on the changes:
* Turns off the background partial index rebuild/catchup task by default for a
table. The reason is that users will typically have some kind of retry strategy
themselves (for example, a message queue that retries). They need this as when
a commit exception occurs, some of the data rows may have been written while
others will not have been (regardless of what state the index is in wrt the
data table). What ever retry mechanism is in use, these retries will also get
the index back in sync (see below for a new mechanism for mutable tables).
* Provides a means for the client to retry a commit at the timestamp at which
it was originally submitted. This is important for mutable data as otherwise
the retried commits may overwrite successful commits that occurred later. This
is accomplished by a) including the server timestamp at which the data rows
were (or would have been) committed in CommitException and b) Adds a new
connection property, {{PhoenixRuntime.REPLAY_AT_ATTRIB}}, which specifies a
timestamp at which the commit will occur and tells the system to ignore later
data updates (to ensure your index remains in sync with your data table).
* Provides an option (the default) to keep an index active even after a write
failure occurs. Many use cases are essentially down without the secondary index
in place and would rather the index be behi
[jira] [Comment Edited] (PHOENIX-3811) Do not disable index on write failure by default
[ https://issues.apache.org/jira/browse/PHOENIX-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006920#comment-16006920 ] Matthew Van Wely edited comment on PHOENIX-3811 at 5/11/17 6:08 PM: Thx [~giacomotaylor]. > If you don't set the REPLAY_AT property and earlier batches are processed > before later batches, you run the risk of overwriting a row with an earlier > value Regarding this, we aim to make all upserts idempotent, by reloading and writing the entire row, as a retry mechanism to avoid overlapping and out of order writes. We'll consider if "replay_at" is still needed. Thx. was (Author: mvanw...@gmail.com): Thx [~giacomotaylor]. > If you don't set the REPLAY_AT property and earlier batches are processed > before later batches, you run the risk of overwriting a row with an earlier > value Regarding this, we aim to make all upserts idempotents by reloading and writing the entire row, as a retry mechanism to avoid overlapping and out of order writes. We'll consider if "replay_at" is still needed. Thx. > Do not disable index on write failure by default > > > Key: PHOENIX-3811 > URL: https://issues.apache.org/jira/browse/PHOENIX-3811 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.11.0 > > Attachments: PHOENIX-3811_v1.patch, PHOENIX-3811_v2.patch, > PHOENIX-3811_v3.patch, PHOENIX-3811-wip1.patch, PHOENIX-3811-wip2.patch, > PHOENIX-3811-wip3.patch, PHOENIX-3811-wip4.patch, PHOENIX-3811-wip5.patch, > PHOENIX-3811-wip7.patch > > > We should provide a way to configure the system so that the server takes no > specific action when an index write fails. Since we always throw the write > failure back to the client, the client can often deal with failures more > easily than the server since they have the batch of mutations in memory. > Often times, allowing access to an index that may be one batch behind the > data table is better than disabling it given the negative performance that > will occur while the index cannot be written to. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
