[
https://issues.apache.org/jira/browse/RYA-246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David W. Lotts updated RYA-246:
-------------------------------
Description:
Currently the Fluo query observer framework (FQF) has a great plugin-in-ish way
of doing exports. (See field: {{exporters}} in class:
{{org.apache.rya.indexing.pcj.fluo.app.observers.QueryResultObserver}} )
Exports are query results and the action taken when they occur. Currently
there are two. Despite this awesome framework, there are three interminglings
that can become issues in a production system. Feel free to make sub tasks for
each of these three tasks:
1. Create PCJ independently of alerts/triggers
At query register time (with FQF), optionally create PCJ's and optionally
create alerts (with Alerting) preferably atomically, so if it fails in the
middle it does not leave dangling references and garbage -- currently there is
one method that does everything: CreatePCJ.withRyaInteraction()
2. Enable exporters per query -- currently all queries will be exported on all
globally enabled exporters. When the query is registered (with FQF) specify
the exporters that should be used.
3. Separate export observer queues per exporter -- Currently retrying a failed
export observer will re-export all export types, even if some already
succeeded, causing duplicate triggers.
Also there is a need to refine the terms for java identifiers (for example:
fields, classes) and documentation. Please document here a decision about what
we name different things. The term PCJ are the index tables used by Rya to
optimize it's queries. But sometimes it is used to describe the whole Fluo
framework. Please define a unique term for the following. The starred term
will be used here, but feel free to rename them:
- Rya query optimization PCJ (ex: the Blabla exporter) PCJ*
- Kafka trigger alert notifier (ex: the Foofoo exporter) Alerting*
- Fluo query observer framework (ex: results from the Barbar) FQF*
Unique identifers:
- FQF uses QueryID
- Alerting uses TopicID
- PCJ uses PcjId
Note that currently TopicID is assigned the same as QueryID, but that badly
presupposes queryID's cannot be grouped onto one Topic.
was:
Currently the Fluo query observer framework (FQF) has a great plugin-in-ish way
of doing exports. (See field: {{exporters}} in class:
{{org.apache.rya.indexing.pcj.fluo.app.observers.QueryResultObserver}} )
Exports are query results and the action taken when they occur. Currently
there are two. Despite this awsome framework, there are three interminglings
that can become issues in a production system:
1. Create PCJ independently of alerts/triggers
At query register time (with FQF), optionally create PCJ's and optionally
create alerts (with Alerting) preferably atomically, so if it fails in the
middle it does not leave dangling references and garbage -- currently there is
one method that does everything: CreatePCJ.withRyaInteraction()
2. Enable exporters per query -- currently all queries will be exported on all
globally enabled exporters
3. Separate export observer queues per exporter -- Currently retrying a failed
export observer will re-export all export types, even if some already
succeeded, causing duplicate triggers.
Also there is a need to refine the terms for java identifiers (for example:
fields, classes) and documentation. Please document here a decision about what
we name different things. The term PCJ are the index tables used by Rya to
optimize it's queries. But sometimes it is used to describe the whole Fluo
framework. Please define a unique term for the following. The starred term
will be used here, but feel free to rename them:
- Rya query optimization PCJ (ex: the Blabla exporter) PCJ*
- Kafka trigger alert notifier (ex: the Foofoo exporter) Alerting*
- Fluo query observer framework (ex: results from the Barbar) FQF*
Unique identifers:
- FQF uses QueryID
- Alerting uses TopicID
- PCJ uses PcjId
Note that currently TopicID is assigned the same as QueryID, but that badly
presupposes queryID's cannot be grouped onto one Topic.
> Separate Fluo query result exporters, Rya PCJ intermingled kafka triggers
> -------------------------------------------------------------------------
>
> Key: RYA-246
> URL: https://issues.apache.org/jira/browse/RYA-246
> Project: Rya
> Issue Type: Improvement
> Components: dao
> Environment: Fluo, Kafka, PCJ, Accumulo
> Reporter: David W. Lotts
>
> Currently the Fluo query observer framework (FQF) has a great plugin-in-ish
> way of doing exports. (See field: {{exporters}} in class:
> {{org.apache.rya.indexing.pcj.fluo.app.observers.QueryResultObserver}} )
> Exports are query results and the action taken when they occur. Currently
> there are two. Despite this awesome framework, there are three
> interminglings that can become issues in a production system. Feel free to
> make sub tasks for each of these three tasks:
> 1. Create PCJ independently of alerts/triggers
> At query register time (with FQF), optionally create PCJ's and optionally
> create alerts (with Alerting) preferably atomically, so if it fails in the
> middle it does not leave dangling references and garbage -- currently there
> is one method that does everything: CreatePCJ.withRyaInteraction()
> 2. Enable exporters per query -- currently all queries will be exported on
> all globally enabled exporters. When the query is registered (with FQF)
> specify the exporters that should be used.
> 3. Separate export observer queues per exporter -- Currently retrying a
> failed export observer will re-export all export types, even if some already
> succeeded, causing duplicate triggers.
> Also there is a need to refine the terms for java identifiers (for example:
> fields, classes) and documentation. Please document here a decision about
> what we name different things. The term PCJ are the index tables used by Rya
> to optimize it's queries. But sometimes it is used to describe the whole
> Fluo framework. Please define a unique term for the following. The starred
> term will be used here, but feel free to rename them:
> - Rya query optimization PCJ (ex: the Blabla exporter) PCJ*
> - Kafka trigger alert notifier (ex: the Foofoo exporter) Alerting*
> - Fluo query observer framework (ex: results from the Barbar) FQF*
> Unique identifers:
> - FQF uses QueryID
> - Alerting uses TopicID
> - PCJ uses PcjId
> Note that currently TopicID is assigned the same as QueryID, but that badly
> presupposes queryID's cannot be grouped onto one Topic.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
