[GitHub] incubator-fluo-recipes pull request #130: Updated ExportQ and CFM to use new...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-recipes/pull/130#discussion_r114644943 --- Diff: docs/combine-queue.md --- @@ -0,0 +1,224 @@ + +# Combine Queue Recipe + +## Background + +When many transactions try to modify the same keys, collisions will occur. Too many collisions +cause transactions to fail and throughput to nose dive. For example, consider [phrasecount] +which has many transactions processing documents. Each transaction counts the phrases in a document +and then updates global phrase counts. Since transaction attempts to update many phrases +, the probability of collisions is high. + +## Solution + +The [combine queue recipe][CombineQueue] provides a reusable solution for updating many keys while +avoiding collisions. The recipe also organizes updates into batches in order to improve throughput. + +This recipes queues updates to keys for other transactions to process. In the phrase count example +transactions processing documents queue updates, but do not actually update the counts. Below is an +example of computing phrasecounts using this recipe. + + * TX1 queues `+1` update for phrase `we want lambdas now` + * TX2 queues `+1` update for phrase `we want lambdas now` + * TX3 reads the updates and current value for the phrase `we want lambdas now`. There is no current value and the updates sum to 2, so a new value of 2 is written. + * TX4 queues `+2` update for phrase `we want lambdas now` + * TX5 queues `-1` update for phrase `we want lambdas now` + * TX6 reads the updates and current value for the phrase `we want lambdas now`. The current value is 2 and the updates sum to 1, so a new value of 3 is written. + +Transactions processing updates have the ability to make additional updates. +For example in addition to updating the current value for a phrase, the new +value could also be placed on an export queue to update an external database. + +### Buckets + +A simple implementation of this recipe would have an update queue for each key. However the +implementation is slightly more complex. Each update queue is in a bucket and transactions process +all of the updates in a bucket. This allows more efficient processing of updates for the following +reasons : + + * When updates are queued, notifications are made per bucket(instead of per a key). + * The transaction doing the update can scan the entire bucket reading updates, this avoids a seek for each key being updated. + * Also the transaction can request a batch lookup to get the current value of all the keys being updated. + * Any additional actions taken on update (like adding something to an export queue) can also be batched. + * Data is organized to make reading exiting values for keys in a bucket more efficient. + +Which bucket a key goes to is decided using hash and modulus so that multiple updates for a key go +to the same bucket. + +The initial number of tablets to create when applying table optimizations can be controlled by +setting the buckets per tablet option when configuring a Combine Queue. For example if you +have 20 tablet servers and 1000 buckets and want 2 tablets per tserver initially then set buckets +per tablet to 1000/(2*20)=25. + +## Example Use + +The following code snippets show how to use this recipe for wordcount. The first step is to +configure it before initializing Fluo. When initializing an ID is needed. This ID is used in two +ways. First, the ID is used as a row prefix in the table. Therefore nothing else should use that +row range in the table. Second, the ID is used in generating configuration keys. + +The following snippet shows how to configure a combine queue. + +```java +FluoConfiguration fluoConfig = ...; + +// Set application properties for the combine queue. These properties are read later by +// the observers running on each worker. +CombineQueue.configure(WcObserverProvider.ID) + .keyType(String.class).valueType(Long.class).buckets(119).finish(fluoConfig); --- End diff -- Another option is `store()` but its up to you. I am fine with finish(). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-recipes pull request #130: Updated ExportQ and CFM to use new...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-recipes/pull/130#discussion_r114631218 --- Diff: docs/combine-queue.md --- @@ -0,0 +1,224 @@ + +# Combine Queue Recipe + +## Background + +When many transactions try to modify the same keys, collisions will occur. Too many collisions +cause transactions to fail and throughput to nose dive. For example, consider [phrasecount] +which has many transactions processing documents. Each transaction counts the phrases in a document +and then updates global phrase counts. Since transaction attempts to update many phrases +, the probability of collisions is high. + +## Solution + +The [combine queue recipe][CombineQueue] provides a reusable solution for updating many keys while +avoiding collisions. The recipe also organizes updates into batches in order to improve throughput. + +This recipes queues updates to keys for other transactions to process. In the phrase count example +transactions processing documents queue updates, but do not actually update the counts. Below is an +example of computing phrasecounts using this recipe. + + * TX1 queues `+1` update for phrase `we want lambdas now` + * TX2 queues `+1` update for phrase `we want lambdas now` + * TX3 reads the updates and current value for the phrase `we want lambdas now`. There is no current value and the updates sum to 2, so a new value of 2 is written. + * TX4 queues `+2` update for phrase `we want lambdas now` + * TX5 queues `-1` update for phrase `we want lambdas now` + * TX6 reads the updates and current value for the phrase `we want lambdas now`. The current value is 2 and the updates sum to 1, so a new value of 3 is written. + +Transactions processing updates have the ability to make additional updates. +For example in addition to updating the current value for a phrase, the new +value could also be placed on an export queue to update an external database. + +### Buckets + +A simple implementation of this recipe would have an update queue for each key. However the +implementation is slightly more complex. Each update queue is in a bucket and transactions process +all of the updates in a bucket. This allows more efficient processing of updates for the following +reasons : + + * When updates are queued, notifications are made per bucket(instead of per a key). + * The transaction doing the update can scan the entire bucket reading updates, this avoids a seek for each key being updated. + * Also the transaction can request a batch lookup to get the current value of all the keys being updated. + * Any additional actions taken on update (like adding something to an export queue) can also be batched. + * Data is organized to make reading exiting values for keys in a bucket more efficient. + +Which bucket a key goes to is decided using hash and modulus so that multiple updates for a key go +to the same bucket. + +The initial number of tablets to create when applying table optimizations can be controlled by +setting the buckets per tablet option when configuring a Combine Queue. For example if you +have 20 tablet servers and 1000 buckets and want 2 tablets per tserver initially then set buckets +per tablet to 1000/(2*20)=25. + +## Example Use + +The following code snippets show how to use this recipe for wordcount. The first step is to +configure it before initializing Fluo. When initializing an ID is needed. This ID is used in two +ways. First, the ID is used as a row prefix in the table. Therefore nothing else should use that +row range in the table. Second, the ID is used in generating configuration keys. + +The following snippet shows how to configure a combine queue. + +```java +FluoConfiguration fluoConfig = ...; + +// Set application properties for the combine queue. These properties are read later by +// the observers running on each worker. +CombineQueue.configure(WcObserverProvider.ID) + .keyType(String.class).valueType(Long.class).buckets(119).finish(fluoConfig); --- End diff -- finish() could be called save() --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-recipes pull request #130: Updated ExportQ and CFM to use new...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo-recipes/pull/130#discussion_r114635795
--- Diff: docs/combine-queue.md ---
@@ -0,0 +1,224 @@
+
+# Combine Queue Recipe
+
+## Background
+
+When many transactions try to modify the same keys, collisions will occur.
Too many collisions
+cause transactions to fail and throughput to nose dive. For example,
consider [phrasecount]
+which has many transactions processing documents. Each transaction counts
the phrases in a document
+and then updates global phrase counts. Since transaction attempts to
update many phrases
+, the probability of collisions is high.
+
+## Solution
+
+The [combine queue recipe][CombineQueue] provides a reusable solution for
updating many keys while
+avoiding collisions. The recipe also organizes updates into batches in
order to improve throughput.
+
+This recipes queues updates to keys for other transactions to process. In
the phrase count example
+transactions processing documents queue updates, but do not actually
update the counts. Below is an
+example of computing phrasecounts using this recipe.
+
+ * TX1 queues `+1` update for phrase `we want lambdas now`
+ * TX2 queues `+1` update for phrase `we want lambdas now`
+ * TX3 reads the updates and current value for the phrase `we want lambdas
now`. There is no current value and the updates sum to 2, so a new value of 2
is written.
+ * TX4 queues `+2` update for phrase `we want lambdas now`
+ * TX5 queues `-1` update for phrase `we want lambdas now`
+ * TX6 reads the updates and current value for the phrase `we want lambdas
now`. The current value is 2 and the updates sum to 1, so a new value of 3 is
written.
+
+Transactions processing updates have the ability to make additional
updates.
+For example in addition to updating the current value for a phrase, the new
+value could also be placed on an export queue to update an external
database.
+
+### Buckets
+
+A simple implementation of this recipe would have an update queue for each
key. However the
+implementation is slightly more complex. Each update queue is in a bucket
and transactions process
+all of the updates in a bucket. This allows more efficient processing of
updates for the following
+reasons :
+
+ * When updates are queued, notifications are made per bucket(instead of
per a key).
+ * The transaction doing the update can scan the entire bucket reading
updates, this avoids a seek for each key being updated.
+ * Also the transaction can request a batch lookup to get the current
value of all the keys being updated.
+ * Any additional actions taken on update (like adding something to an
export queue) can also be batched.
+ * Data is organized to make reading exiting values for keys in a bucket
more efficient.
+
+Which bucket a key goes to is decided using hash and modulus so that
multiple updates for a key go
+to the same bucket.
+
+The initial number of tablets to create when applying table optimizations
can be controlled by
+setting the buckets per tablet option when configuring a Combine Queue.
For example if you
+have 20 tablet servers and 1000 buckets and want 2 tablets per tserver
initially then set buckets
+per tablet to 1000/(2*20)=25.
+
+## Example Use
+
+The following code snippets show how to use this recipe for wordcount.
The first step is to
+configure it before initializing Fluo. When initializing an ID is needed.
This ID is used in two
+ways. First, the ID is used as a row prefix in the table. Therefore
nothing else should use that
+row range in the table. Second, the ID is used in generating
configuration keys.
+
+The following snippet shows how to configure a combine queue.
+
+```java
+FluoConfiguration fluoConfig = ...;
+
+// Set application properties for the combine queue. These properties
are read later by
+// the observers running on each worker.
+CombineQueue.configure(WcObserverProvider.ID)
+
.keyType(String.class).valueType(Long.class).buckets(119).finish(fluoConfig);
+
+fluoConfig.setObserverProvider(WcObserverProvider.class);
+
+// initialize Fluo using fluoConfig
+```
+
+Assume the following observer is triggered when a documents is updated.
It examines new
+and old document content and determines changes in word counts. These
changes are pushed to a
+combine queue.
+
+```java
+public class DocumentObserver implements StringObserver {
+ // word count combine queue
+ private CombineQueue<String, Long> wccq;
+
+ public static final Column NEW_COL = new Column("content", "new");
+
[GitHub] incubator-fluo-recipes pull request #129: Reduced creation of intermediate S...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo-recipes/pull/129#discussion_r113977876
--- Diff:
modules/core/src/main/java/org/apache/fluo/recipes/core/map/CollisionFreeMap.java
---
@@ -194,12 +199,11 @@ void process(TransactionBase tx, Bytes ntfyRow,
Column col) throws Exception {
tx.setWeakNotification(ntfyRow, col);
}
-byte[] dataPrefix = ntfyRow.toArray();
--- End diff --
Does `ntfyRow` contain `mapId`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo-recipes pull request #128: Updated ExportQ and CFM to use new...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo-recipes/pull/128#discussion_r113275106
--- Diff:
modules/core/src/main/java/org/apache/fluo/recipes/core/map/ICombiner.java ---
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.recipes.core.map;
+
+import java.util.Iterator;
+import java.util.Optional;
+import java.util.stream.Stream;
+
+/**
+ * This class was created as an alternative to {@link Combiner}. It
supports easy and efficient use
+ * of java streams when implementing combiners using lambdas.
+ *
+ * @since 1.1.0
+ */
+public interface ICombiner<K, V> {
--- End diff --
Could be called `ValueCombiner`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo-recipes pull request #128: Updated ExportQ and CFM to use new...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-recipes/pull/128#discussion_r113274323 --- Diff: docs/cfm.md --- @@ -52,85 +46,93 @@ value could also be placed on an export queue to update an external database. ### Buckets -A simple implementation of this recipe would be to have an update queue for -each key. However the implementation does something slightly more complex. -Each update queue is in a bucket and transactions that process updates, process -all of the updates in a bucket. This allows more efficient processing of -updates for the following reasons : +A simple implementation of this recipe would have an update queue for each key. However the +implementation is slightly more complex. Each update queue is in a bucket and transactions process +all of the updates in a bucket. This allows more efficient processing of updates for the following +reasons : * When updates are queued, notifications are made per bucket(instead of per a key). - * The transaction doing the update can scan the entire bucket reading updates, this avoids a seek for each key being updated. + * The transaction doing the update can scan the entire bucket reading updates, this avoids a seek for each key being updated. * Also the transaction can request a batch lookup to get the current value of all the keys being updated. * Any additional actions taken on update (like adding something to an export queue) can also be batched. * Data is organized to make reading exiting values for keys in a bucket more efficient. -Which bucket a key goes to is decided using hash and modulus so that multiple -updates for the same key always go to the same bucket. +Which bucket a key goes to is decided using hash and modulus so that multiple updates for a key go +to the same bucket. -The initial number of tablets to create when applying table optimizations can be -controlled by setting the buckets per tablet option when configuring a Collision -Free Map. For example if you have 20 tablet servers and 1000 buckets and want -2 tablets per tserver initially then set buckets per tablet to 1000/(2*20)=25. +The initial number of tablets to create when applying table optimizations can be controlled by +setting the buckets per tablet option when configuring a Collision Free Map. For example if you +have 20 tablet servers and 1000 buckets and want 2 tablets per tserver initially then set buckets +per tablet to 1000/(2*20)=25. ## Example Use -The following code snippets show how to setup and use this recipe for -wordcount. The first step in using this recipe is to configure it before -initializing Fluo. When initializing an ID will need to be provided. This ID -is used in two ways. First, the ID is used as a row prefix in the table. -Therefore nothing else should use that row range in the table. Second, the ID -is used in generating configuration keys associated with the instance of the -Collision Free Map. +The following code snippets show how to use this recipe for wordcount. The first step is to +configure it before initializing Fluo. When initializing an ID is needed. This ID is used in two +ways. First, the ID is used as a row prefix in the table. Therefore nothing else should use that +row range in the table. Second, the ID is used in generating configuration keys. -The following snippet shows how to setup a collision free map. +The following snippet shows how to configure a collision free map. ```java FluoConfiguration fluoConfig = ...; int numBuckets = 119; + int numTablets = 20; - WordCountMap.configure(fluoConfig, 119); + String mapId = WcObserverProvider.ID; - //initialize Fluo using fluoConfig + // Create a Java Object that encapsulates the configuration + CollisionFreeMap.Options cfmOpts = + new CollisionFreeMap.Options(mapId, String.class, Long.class, numBuckets) + .setBucketsPerTablet(numBuckets / numTablets); + + // Set application properties for the collision free map. These properties are read later by + // observers. + CollisionFreeMap.configure(fluoConfig, cfmOpts); --- End diff -- `CollisionFreeMap` could be named `CombineQueue` or `Combiner` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-recipes pull request #128: Updated ExportQ and CFM to use new...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-recipes/pull/128#discussion_r113265960 --- Diff: docs/accumulo-export-queue.md --- @@ -19,8 +19,8 @@ limitations under the License. ## Background The [Export Queue Recipe][1] provides a generic foundation for building export mechanism to any -external data store. The [AccumuloExporter] provides an implementation of this recipe for -Accumulo. The [AccumuloExporter] is located the `fluo-recipes-accumulo` module and provides the +external data store. The [AccumuloConsumer] provides an export consumer for writing to --- End diff -- You could keep AccumuloExporter name but move to a new package --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-recipes pull request #128: Updated ExportQ and CFM to use new...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-recipes/pull/128#discussion_r113275022 --- Diff: modules/core/src/main/java/org/apache/fluo/recipes/core/map/ICombiner.java --- @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ + +package org.apache.fluo.recipes.core.map; --- End diff -- This could be in `org.apache.fluo.recipes.core.combiner` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo pull request #822: fixes #500 Made scanning for notifications...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/822#discussion_r112063445
--- Diff:
modules/core/src/main/java/org/apache/fluo/core/worker/finder/hash/PartitionNotificationFinder.java
---
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.core.worker.finder.hash;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.fluo.core.impl.Environment;
+import org.apache.fluo.core.impl.FluoConfigurationImpl;
+import org.apache.fluo.core.impl.Notification;
+import org.apache.fluo.core.worker.NotificationFinder;
+import org.apache.fluo.core.worker.NotificationProcessor;
+import org.apache.fluo.core.worker.TxResult;
+
+public class PartitionNotificationFinder implements NotificationFinder {
+
+ private ParitionManager paritionManager;
+ private Thread scanThread;
+ private NotificationProcessor processor;
+ private Environment env;
+ private AtomicBoolean stopped;
+
+ @Override
+ public void init(Environment env, NotificationProcessor processor) {
+this.processor = processor;
+this.env = env;
+this.stopped = new AtomicBoolean(false);
+
+ }
+
+ @Override
+ public void start() {
+long minSleepTime =
+
env.getConfiguration().getInt(FluoConfigurationImpl.MIN_SLEEP_TIME_PROP,
--- End diff --
Could use better name than `MIN_SLEEP_TIME`. Same for `MAX_SLEEP_TIME`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #822: fixes #500 Made scanning for notifications...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/822#discussion_r112224718
--- Diff:
modules/core/src/main/java/org/apache/fluo/core/worker/finder/hash/PartitionInfo.java
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.core.worker.finder.hash;
+
+import java.util.List;
+
+class PartitionInfo {
+
+ private final int groupId;
+ private final int idInGroup;
+ private final int groups;
--- End diff --
could be `totalGroups`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #822: fixes #500 Made scanning for notifications...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/822#discussion_r112224636
--- Diff:
modules/core/src/main/java/org/apache/fluo/core/worker/finder/hash/PartitionInfo.java
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.core.worker.finder.hash;
+
+import java.util.List;
+
+class PartitionInfo {
+
+ private final int groupId;
--- End diff --
could be `myGroupId`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #822: fixes #500 Made scanning for notifications...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/822#discussion_r112227421
--- Diff:
modules/core/src/main/java/org/apache/fluo/core/worker/finder/hash/PartitionInfo.java
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.core.worker.finder.hash;
+
+import java.util.List;
+
+class PartitionInfo {
+
+ private final int groupId;
+ private final int idInGroup;
+ private final int groups;
+ private final int groupSize;
+ private final int workers;
+ private final TabletSet groupsTablets;
+
+ PartitionInfo(int myId, int myGroupId, int myGroupSize, int totalGroups,
int totalWorkers,
+ List groupsTablets) {
+this.idInGroup = myId;
+this.groupId = myGroupId;
+this.groupSize = myGroupSize;
+this.groups = totalGroups;
+this.workers = totalWorkers;
+this.groupsTablets = new TabletSet(groupsTablets);
+ }
+
+ public int getGroupId() {
+return groupId;
+ }
+
+ public int getIdInGroup() {
--- End diff --
Could create comments on methods about what these values are being returned
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #822: fixes #500 Made scanning for notifications...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/822#discussion_r112219019
--- Diff:
modules/core/src/main/java/org/apache/fluo/core/worker/finder/hash/ParitionManager.java
---
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.core.worker.finder.hash;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Random;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.accumulo.core.data.ArrayByteSequence;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.PathChildrenCache;
+import
org.apache.curator.framework.recipes.cache.PathChildrenCache.StartMode;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
+import
org.apache.curator.framework.recipes.cache.PathChildrenCacheListener;
+import org.apache.curator.framework.recipes.nodes.PersistentEphemeralNode;
+import
org.apache.curator.framework.recipes.nodes.PersistentEphemeralNode.Mode;
+import org.apache.curator.utils.ZKPaths;
+import org.apache.fluo.accumulo.iterators.NotificationHashFilter;
+import org.apache.fluo.accumulo.util.NotificationUtil;
+import org.apache.fluo.accumulo.util.ZookeeperPath;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.core.impl.Environment;
+import org.apache.fluo.core.impl.FluoConfigurationImpl;
+import org.apache.fluo.core.impl.Notification;
+import org.apache.fluo.core.util.ByteUtil;
+import org.apache.fluo.core.util.FluoThreadFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * This class manages partitioning of notifications across workers
coordinating in ZooKeeper.
+ * Workers are divided into groups. Each group is given a subset of the
Accumulo table. All workers
+ * in a group scan that subset and use hash partitioning to equally divide
notifications.
+ *
+ *
+ * Grouping workers was a compromise between every worker scanning the
entire table OR each worker
+ * having a dedicated part of a table. This scheme allows multiple workers
to share popular parts of
+ * a table. However, it limits the number of workers that will scan a
portion of a table for
+ * notifications. This limitation is important for scaling, even if there
are 1,000 workers there
+ * will never be more than 7 to 13 workers scanning a portion of the table.
+ */
+public class ParitionManager {
+
+ private static final Logger log =
LoggerFactory.getLogger(ParitionManager.class);
+
+ private final PathChildrenCache childrenCache;
+ private final PersistentEphemeralNode myESNode;
+ private final int groupSize;
+ private long paritionSetTime;
+ private PartitionInfo partitionInfo;
+ private final ScheduledExecutorService schedExecutor;
+
+ private CuratorFramework curator;
+
+ private Environment env;
+
+ private final long minSleepTime;
+ private final long maxSleepTime;
+ private long retrySleepTime;
+
+ private static final long STABILIZE_TIME = TimeUnit.SECONDS.toMillis(60);
+
+ private class FindersListener implements PathChildrenCacheListener {
+
+@Override
+public void childEvent(CuratorFramework client, PathChildrenCacheEvent
event) throws Exception {
+ switch (event.getType()) {
+case CHILD_ADDED:
+
[GitHub] incubator-fluo pull request #822: fixes #500 Made scanning for notifications...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/822#discussion_r112224886
--- Diff:
modules/core/src/main/java/org/apache/fluo/core/worker/finder/hash/PartitionInfo.java
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.core.worker.finder.hash;
+
+import java.util.List;
+
+class PartitionInfo {
+
+ private final int groupId;
+ private final int idInGroup;
+ private final int groups;
+ private final int groupSize;
--- End diff --
could be `myGroupSize`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #822: fixes #500 Made scanning for notifications...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/822#discussion_r112225184
--- Diff:
modules/core/src/main/java/org/apache/fluo/core/worker/finder/hash/PartitionInfo.java
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.core.worker.finder.hash;
+
+import java.util.List;
+
+class PartitionInfo {
+
+ private final int groupId;
+ private final int idInGroup;
+ private final int groups;
+ private final int groupSize;
+ private final int workers;
+ private final TabletSet groupsTablets;
+
+ PartitionInfo(int myId, int myGroupId, int myGroupSize, int totalGroups,
int totalWorkers,
--- End diff --
Looks like you already used better naming here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r107483818
--- Diff:
modules/core/src/main/java/org/apache/fluo/core/observer/v2/ObserverStoreV2.java
---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.core.observer.v2;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import com.google.common.base.Preconditions;
+import com.google.gson.Gson;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.fluo.api.config.FluoConfiguration;
+import org.apache.fluo.api.data.Column;
+import org.apache.fluo.api.exceptions.FluoException;
+import org.apache.fluo.api.observer.Observer.NotificationType;
+import org.apache.fluo.api.observer.ObserverProvider;
+import org.apache.fluo.core.impl.Environment;
+import org.apache.fluo.core.observer.RegisteredObservers;
+import org.apache.fluo.core.observer.Observers;
+import org.apache.fluo.core.observer.ObserverStore;
+import org.apache.fluo.core.util.CuratorUtil;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static
org.apache.fluo.accumulo.util.ZookeeperPath.CONFIG_FLUO_OBSERVERS2;
+
+/*
+ * Support for observers configured the new way.
+ */
+public class ObserverStoreV2 implements ObserverStore {
+
+ @Override
+ public boolean handles(FluoConfiguration config) {
+return !config.getObserverProvider().isEmpty();
+ }
+
+ @Override
+ public void update(CuratorFramework curator, FluoConfiguration config)
throws Exception {
+String obsProviderClass = config.getObserverProvider();
+
+ObserverProvider observerProvider =
newObserverProvider(obsProviderClass);
+
+Map<Column, NotificationType> obsCols = new HashMap<>();
+BiConsumer<Column, NotificationType> obsColConsumer = (col, nt) -> {
+ Objects.requireNonNull(col, "Observed column must be non-null");
+ Objects.requireNonNull(nt, "Notification type must be non-null");
+ Preconditions.checkArgument(!obsCols.containsKey(col), "Duplicate
observed column %s", col);
+ obsCols.put(col, nt);
+};
+
+observerProvider.provideColumns(obsColConsumer,
+new ObserverProviderContextImpl(config.getAppConfiguration()));
+
+Gson gson = new Gson();
+String json = gson.toJson(new JsonObservers(obsProviderClass,
obsCols));
+CuratorUtil.putData(curator, CONFIG_FLUO_OBSERVERS2,
json.getBytes(UTF_8),
+CuratorUtil.NodeExistsPolicy.OVERWRITE);
+
+ }
+
+ static ObserverProvider newObserverProvider(String obsProviderClass) {
+ObserverProvider observerProvider;
+try {
+ observerProvider =
+
Class.forName(obsProviderClass).asSubclass(ObserverProvider.class).newInstance();
+} catch (ClassNotFoundException e1) {
+ throw new FluoException("ObserverFactory class '" + obsProviderClass
+ "' was not "
+ + "found. Check for class name misspellings or failure to
include "
+ + "the observer factory jar.", e1);
--- End diff --
should be `observer provider`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r107483725
--- Diff:
modules/core/src/main/java/org/apache/fluo/core/observer/v2/ObserverStoreV2.java
---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.core.observer.v2;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import com.google.common.base.Preconditions;
+import com.google.gson.Gson;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.fluo.api.config.FluoConfiguration;
+import org.apache.fluo.api.data.Column;
+import org.apache.fluo.api.exceptions.FluoException;
+import org.apache.fluo.api.observer.Observer.NotificationType;
+import org.apache.fluo.api.observer.ObserverProvider;
+import org.apache.fluo.core.impl.Environment;
+import org.apache.fluo.core.observer.RegisteredObservers;
+import org.apache.fluo.core.observer.Observers;
+import org.apache.fluo.core.observer.ObserverStore;
+import org.apache.fluo.core.util.CuratorUtil;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static
org.apache.fluo.accumulo.util.ZookeeperPath.CONFIG_FLUO_OBSERVERS2;
+
+/*
+ * Support for observers configured the new way.
+ */
+public class ObserverStoreV2 implements ObserverStore {
+
+ @Override
+ public boolean handles(FluoConfiguration config) {
+return !config.getObserverProvider().isEmpty();
+ }
+
+ @Override
+ public void update(CuratorFramework curator, FluoConfiguration config)
throws Exception {
+String obsProviderClass = config.getObserverProvider();
+
+ObserverProvider observerProvider =
newObserverProvider(obsProviderClass);
+
+Map<Column, NotificationType> obsCols = new HashMap<>();
+BiConsumer<Column, NotificationType> obsColConsumer = (col, nt) -> {
+ Objects.requireNonNull(col, "Observed column must be non-null");
+ Objects.requireNonNull(nt, "Notification type must be non-null");
+ Preconditions.checkArgument(!obsCols.containsKey(col), "Duplicate
observed column %s", col);
+ obsCols.put(col, nt);
+};
+
+observerProvider.provideColumns(obsColConsumer,
+new ObserverProviderContextImpl(config.getAppConfiguration()));
+
+Gson gson = new Gson();
+String json = gson.toJson(new JsonObservers(obsProviderClass,
obsCols));
+CuratorUtil.putData(curator, CONFIG_FLUO_OBSERVERS2,
json.getBytes(UTF_8),
+CuratorUtil.NodeExistsPolicy.OVERWRITE);
+
+ }
+
+ static ObserverProvider newObserverProvider(String obsProviderClass) {
+ObserverProvider observerProvider;
+try {
+ observerProvider =
+
Class.forName(obsProviderClass).asSubclass(ObserverProvider.class).newInstance();
+} catch (ClassNotFoundException e1) {
+ throw new FluoException("ObserverFactory class '" + obsProviderClass
+ "' was not "
--- End diff --
should be `ObserverProvider`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r107480187
--- Diff:
modules/core/src/main/java/org/apache/fluo/core/observer/v2/ObserverStoreV2.java
---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.core.observer.v2;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import com.google.common.base.Preconditions;
+import com.google.gson.Gson;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.fluo.api.config.FluoConfiguration;
+import org.apache.fluo.api.data.Column;
+import org.apache.fluo.api.exceptions.FluoException;
+import org.apache.fluo.api.observer.Observer.NotificationType;
+import org.apache.fluo.api.observer.ObserverProvider;
+import org.apache.fluo.core.impl.Environment;
+import org.apache.fluo.core.observer.RegisteredObservers;
+import org.apache.fluo.core.observer.Observers;
+import org.apache.fluo.core.observer.ObserverStore;
+import org.apache.fluo.core.util.CuratorUtil;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static
org.apache.fluo.accumulo.util.ZookeeperPath.CONFIG_FLUO_OBSERVERS2;
+
+/*
+ * Support for observers configured the new way.
+ */
+public class ObserverStoreV2 implements ObserverStore {
+
+ @Override
+ public boolean handles(FluoConfiguration config) {
+return !config.getObserverProvider().isEmpty();
+ }
+
+ @Override
+ public void update(CuratorFramework curator, FluoConfiguration config)
throws Exception {
+String obsFactoryClass = config.getObserverProvider();
--- End diff --
should be provider
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r107480429
--- Diff:
modules/core/src/main/java/org/apache/fluo/core/observer/v2/ObserversV2.java ---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.core.observer.v2;
+
+import java.lang.reflect.Method;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.Sets;
+import com.google.common.collect.Sets.SetView;
+import org.apache.fluo.api.data.Column;
+import org.apache.fluo.api.exceptions.FluoException;
+import org.apache.fluo.api.observer.Observer;
+import org.apache.fluo.api.observer.Observer.NotificationType;
+import org.apache.fluo.api.observer.ObserverProvider;
+import org.apache.fluo.api.observer.ObserverProvider.Registry;
+import org.apache.fluo.api.observer.StringObserver;
+import org.apache.fluo.core.impl.Environment;
+import org.apache.fluo.core.observer.Observers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class ObserversV2 implements Observers {
+
+ private static final Logger log =
LoggerFactory.getLogger(ObserversV2.class);
+
+ Map<Column, Observer> observers;
+
+ public ObserversV2(Environment env, JsonObservers jco, Set
strongColumns,
+ Set weakColumns) {
+observers = new HashMap<>();
+
+ObserverProvider obsProvider =
+
ObserverStoreV2.newObserverProvider(jco.getObserverProviderClass());
+
+ObserverProviderContextImpl ctx = new ObserverProviderContextImpl(env);
+
+Registry or = new Registry() {
+
+ @Override
+ public void register(Column col, NotificationType nt, Observer obs) {
+try {
+ Method closeMethod = obs.getClass().getMethod("close");
+ if (!closeMethod.getDeclaringClass().equals(Observer.class)) {
+log.warn(
+"Observer {} implements close(). Close is not called on
Observers created using ObserverProvider."
++ " Close is only called on Observers configured the
old way.", obs.getClass()
+.getName());
+ }
+} catch (NoSuchMethodException | SecurityException e) {
+ throw new RuntimeException("Failed to check if close() is
implemented", e);
+}
+
+if (nt == NotificationType.STRONG && !strongColumns.contains(col))
{
+ throw new IllegalArgumentException("Column " + col
+ + " not previously configured for strong notifications");
+}
+
+if (nt == NotificationType.WEAK && !weakColumns.contains(col)) {
+ throw new IllegalArgumentException("Column " + col
+ + " not previously configured for weak notifications");
+}
+
+if (observers.containsKey(col)) {
+ throw new IllegalArgumentException("Duplicate observed column "
+ col);
+}
+
+observers.put(col, obs);
+ }
+
+ @Override
+ public void registers(Column col, NotificationType nt,
StringObserver obs) {
+register(col, nt, obs);
+ }
+};
+
+obsProvider.provide(or, ctx);
+
+// the following check ensures the observers factory provides
observers for all previously
--- End diff --
should be provider
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r107480296
--- Diff:
modules/core/src/main/java/org/apache/fluo/core/observer/v2/ObserverStoreV2.java
---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.core.observer.v2;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Set;
+import java.util.function.BiConsumer;
+
+import com.google.common.base.Preconditions;
+import com.google.gson.Gson;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.fluo.api.config.FluoConfiguration;
+import org.apache.fluo.api.data.Column;
+import org.apache.fluo.api.exceptions.FluoException;
+import org.apache.fluo.api.observer.Observer.NotificationType;
+import org.apache.fluo.api.observer.ObserverProvider;
+import org.apache.fluo.core.impl.Environment;
+import org.apache.fluo.core.observer.RegisteredObservers;
+import org.apache.fluo.core.observer.Observers;
+import org.apache.fluo.core.observer.ObserverStore;
+import org.apache.fluo.core.util.CuratorUtil;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static
org.apache.fluo.accumulo.util.ZookeeperPath.CONFIG_FLUO_OBSERVERS2;
+
+/*
+ * Support for observers configured the new way.
+ */
+public class ObserverStoreV2 implements ObserverStore {
+
+ @Override
+ public boolean handles(FluoConfiguration config) {
+return !config.getObserverProvider().isEmpty();
+ }
+
+ @Override
+ public void update(CuratorFramework curator, FluoConfiguration config)
throws Exception {
+String obsFactoryClass = config.getObserverProvider();
+
+ObserverProvider observerProvider =
newObserverProvider(obsFactoryClass);
+
+Map<Column, NotificationType> obsCols = new HashMap<>();
+BiConsumer<Column, NotificationType> obsColConsumer = (col, nt) -> {
+ Objects.requireNonNull(col, "Observed column must be non-null");
+ Objects.requireNonNull(nt, "Notification type must be non-null");
+ Preconditions.checkArgument(!obsCols.containsKey(col), "Duplicate
observed column %s", col);
+ obsCols.put(col, nt);
+};
+
+observerProvider.provideColumns(obsColConsumer,
+new ObserverProviderContextImpl(config.getAppConfiguration()));
+
+Gson gson = new Gson();
+String json = gson.toJson(new JsonObservers(obsFactoryClass, obsCols));
+CuratorUtil.putData(curator, CONFIG_FLUO_OBSERVERS2,
json.getBytes(UTF_8),
+CuratorUtil.NodeExistsPolicy.OVERWRITE);
+
+ }
+
+ static ObserverProvider newObserverProvider(String obsFactoryClass) {
--- End diff --
should be provider
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r107480083
--- Diff:
modules/core/src/main/java/org/apache/fluo/core/observer/v2/JsonObservers.java
---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.core.observer.v2;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.fluo.api.data.Column;
+import org.apache.fluo.api.observer.Observer.NotificationType;
+
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.Collectors.toMap;
+
+/**
+ * this class created for json serialization
+ */
+class JsonObservers {
+ String obsProviderClass;
+ List observedColumns;
+
+ JsonObservers(String obsFactoryClass, Map<Column, NotificationType>
columns) {
--- End diff --
should be provider
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r107479937
--- Diff:
modules/api/src/main/java/org/apache/fluo/api/config/FluoConfiguration.java ---
@@ -344,6 +362,37 @@ private int getNextObserverId() {
return max + 1;
}
+ /**
+ * Configure the observer factory that Fluo workers will use.
+ *
+ * @since 1.1.0
+ *
+ * @param className Name of a class that implements {@link
ObserverProvider}. Must be non-null and
+ *non-empty.
+ */
+ public void setObserverProvider(String className) {
+setNonEmptyString(OBSERVER_PROVIDER, className);
+ }
+
+ /**
+ * Calls {@link #setObserverProvider(String)} with the class name.
+ *
+ * @since 1.1.0
+ */
+ public void setObserverProvider(Class clazz)
{
+setObserverProvider(clazz.getName());
+ }
+
+ /**
+ * @return The configured {@link ObserverProvider} class name. If one
was not configured, returns
+ * {@value #OBSERVERS_FACTORY_DEFAULT}
--- End diff --
should be provider
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo/pull/817#discussion_r107479787 --- Diff: modules/api/src/main/java/org/apache/fluo/api/config/FluoConfiguration.java --- @@ -86,8 +88,19 @@ /** The properties below get loaded into/from Zookeeper */ // Observer + @Deprecated public static final String OBSERVER_PREFIX = FLUO_PREFIX + ".observer."; + /** + * @since 1.1.0 + */ + public static final String OBSERVER_PROVIDER = FLUO_PREFIX + ".observer.provider"; + + /** + * @since 1.1.0 + */ + public static final String OBSERVERS_FACTORY_DEFAULT = ""; --- End diff -- Should be provider --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r107466983
--- Diff:
modules/api/src/main/java/org/apache/fluo/api/config/FluoConfiguration.java ---
@@ -344,6 +362,37 @@ private int getNextObserverId() {
return max + 1;
}
+ /**
+ * Configure the observer factory that Fluo workers will use.
+ *
+ * @since 1.1.0
+ *
+ * @param className Name of a class that implements {@link
ObserverProvider}. Must be non-null and
+ *non-empty.
+ */
+ public void setObserverProvider(String className) {
+setNonEmptyString(OBSERVERS_FACTORY, className);
--- End diff --
should use provider terminlogy
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r107467396
--- Diff:
modules/integration/src/test/java/org/apache/fluo/integration/impl/CollisionIT.java
---
@@ -81,27 +77,22 @@ public void load(TransactionBase tx, Context context)
throws Exception {
}
}
- public static class TotalObserver extends AbstractObserver {
-
+ public static class CollisionObserverProvider implements
ObserverProvider {
@Override
-public Observer.ObservedColumn getObservedColumn() {
- return new Observer.ObservedColumn(STAT_CHANGED,
NotificationType.WEAK);
-}
-
-@Override
-public void process(TransactionBase tx, Bytes rowBytes, Column col)
throws Exception {
- String row = rowBytes.toString();
- int total = Integer.parseInt(tx.gets(row, STAT_TOTAL));
- int processed = TestUtil.getOrDefault(tx, row, STAT_PROCESSED, 0);
-
- tx.set(row, STAT_PROCESSED, total + "");
- TestUtil.increment(tx, "all", STAT_TOTAL, total - processed);
+public void provide(Registry consumer, Context ctx) {
--- End diff --
could look for `consumer` and replace with `registry`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo/pull/817#discussion_r107466714 --- Diff: modules/api/src/main/java/org/apache/fluo/api/config/FluoConfiguration.java --- @@ -86,8 +88,19 @@ /** The properties below get loaded into/from Zookeeper */ // Observer + @Deprecated public static final String OBSERVER_PREFIX = FLUO_PREFIX + ".observer."; + /** + * @since 1.1.0 + */ + public static final String OBSERVERS_FACTORY = FLUO_PREFIX + ".observers.factory"; --- End diff -- should use provider terminology --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r107466485
--- Diff: docs/applications.md ---
@@ -114,25 +114,45 @@ To create an observer, follow these steps:
// invert row and value
tx.set(value, new Column("inv", "data"), row);
}
+}
+```
+2. Create a class that implements [ObserverProvider] like the example
below. The purpose of this
+class is associate a set Observers with columns that trigger the
observers. The class can
+create multiple observers.
+```java
+class AppObserverProvider implements ObserverProvider {
@Override
- public ObservedColumn getObservedColumn() {
-return new ObservedColumn(new Column("obs", "data"),
NotificationType.STRONG);
+ public void provide(Registry or, Context ctx) {
+//setup InvertObserver to be triggered when the column obs:data is
modified
+or.register(new Column("obs", "data"),
+ NotificationType.STRONG,
+ new InvertObserver());
+
+//Observer is a Functional interface. So Obsevers can be written
as lambdas.
+or.register(new Column("new","data"),
+ NotificationType.WEAK,
+ (tx,row,col) -> {
+ Bytes combined = combineNewAndOld(tx,row);
+ tx.set(row, new Column("current","data"),
combined);
+ });
}
}
```
-2. Build a jar containing this class and include this jar in the `lib/`
directory of your Fluo
+
+3. Build a jar containing thses classes and include this jar in the
`lib/` directory of your Fluo
application.
-3. Configure your Fluo instance to use this observer by modifying the
Observer section of
+4. Configure your Fluo instance to use this observer factory by modifying
the Observer section of
--- End diff --
should be `observer provider`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r107179300
--- Diff:
modules/api/src/main/java/org/apache/fluo/api/observer/ObserverFactory.java ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.api.observer;
+
+import java.util.function.BiConsumer;
+
+import org.apache.fluo.api.client.FluoClient;
+import org.apache.fluo.api.config.FluoConfiguration;
+import org.apache.fluo.api.config.SimpleConfiguration;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+import org.apache.fluo.api.metrics.MetricsReporter;
+import org.apache.fluo.api.observer.Observer.NotificationType;
+
+/**
+ * Fluo Workers use this class to create {@link Observer}s to process
notifications. Implementations
+ * of this class should emit zero or more {@link Observer}s.
+ *
+ *
+ * When Fluo is initialized {@link #getObservedColumns(BiConsumer,
Context)} is called. The columns
+ * it emits are stored in Zookeeper. Transactions will use the columns
stored in Zookeeper to
+ * determine when to set notifications. When Workers call
+ * {@link #createObservers(ObserverConsumer, Context)}, the columns
emitted must be the same as
+ * those emitted during initialization. If this is not the case, then the
worker will fail to start.
+ *
+ * @see FluoConfiguration#setObserversFactory(String)
+ * @since 1.1.0
+ */
+public interface ObserverFactory {
+
+ /**
+ * @since 1.1.0
+ */
+ interface Context {
+/**
+ * @return A configuration object with application configuration like
that returned by
+ * {@link FluoClient#getAppConfiguration()}
+ */
+SimpleConfiguration getAppConfiguration();
+
+/**
+ * @return A {@link MetricsReporter} to report application metrics
from observers.
+ */
+MetricsReporter getMetricsReporter();
+ }
+
+ /**
+ * Observers are emitted to the worker using this interface. This
interface also allows
+ * {@link Observer}s to be related to the columns that trigger them.
+ *
+ * @since 1.1.0
+ */
+ interface ObserverConsumer {
--- End diff --
this should be called `ObserverRegistry`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r107180164
--- Diff:
modules/api/src/main/java/org/apache/fluo/api/observer/ObserverFactory.java ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.api.observer;
+
+import java.util.function.BiConsumer;
+
+import org.apache.fluo.api.client.FluoClient;
+import org.apache.fluo.api.config.FluoConfiguration;
+import org.apache.fluo.api.config.SimpleConfiguration;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+import org.apache.fluo.api.metrics.MetricsReporter;
+import org.apache.fluo.api.observer.Observer.NotificationType;
+
+/**
+ * Fluo Workers use this class to create {@link Observer}s to process
notifications. Implementations
+ * of this class should emit zero or more {@link Observer}s.
+ *
+ *
+ * When Fluo is initialized {@link #getObservedColumns(BiConsumer,
Context)} is called. The columns
+ * it emits are stored in Zookeeper. Transactions will use the columns
stored in Zookeeper to
+ * determine when to set notifications. When Workers call
+ * {@link #createObservers(ObserverConsumer, Context)}, the columns
emitted must be the same as
+ * those emitted during initialization. If this is not the case, then the
worker will fail to start.
+ *
+ * @see FluoConfiguration#setObserversFactory(String)
+ * @since 1.1.0
+ */
+public interface ObserverFactory {
+
+ /**
+ * @since 1.1.0
+ */
+ interface Context {
+/**
+ * @return A configuration object with application configuration like
that returned by
+ * {@link FluoClient#getAppConfiguration()}
+ */
+SimpleConfiguration getAppConfiguration();
+
+/**
+ * @return A {@link MetricsReporter} to report application metrics
from observers.
+ */
+MetricsReporter getMetricsReporter();
+ }
+
+ /**
+ * Observers are emitted to the worker using this interface. This
interface also allows
+ * {@link Observer}s to be related to the columns that trigger them.
+ *
+ * @since 1.1.0
+ */
+ interface ObserverConsumer {
+void accept(Column observedColumn, NotificationType ntfyType, Observer
observer);
+
+/**
+ * This method was created to allow Observers written as lambda to be
passed {@link String}
+ * instead of {@link Bytes}.
+ *
+ *
+ *
+ * void createObservers(ObserverConsumer obsConsumer, Context ctx) {
+ * obsConsumer.accepts(someColumn, WEAK, (tx,row,col) -> {
+ * //row is of type String
+ * };
+ * }
+ *
+ *
+ */
+void accepts(Column observedColumn, NotificationType ntfyType,
StringObserver observer);
+ }
+
+ /**
+ * This is method is called by Fluo Workers to create observers to
process notifications.
+ *
+ *
+ * Observers emitted may be called concurrently by multiple threads to
process different
+ * notifications. Observers should be tolerant of this.
+ *
+ * @param obsConsumer Emit an applications observers to this consumer.
+ */
+ void createObservers(ObserverConsumer obsConsumer, Context ctx);
--- End diff --
this could be `registerObservers` or `setupObservers`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r107179559
--- Diff:
modules/api/src/main/java/org/apache/fluo/api/observer/ObserverFactory.java ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.api.observer;
+
+import java.util.function.BiConsumer;
+
+import org.apache.fluo.api.client.FluoClient;
+import org.apache.fluo.api.config.FluoConfiguration;
+import org.apache.fluo.api.config.SimpleConfiguration;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+import org.apache.fluo.api.metrics.MetricsReporter;
+import org.apache.fluo.api.observer.Observer.NotificationType;
+
+/**
+ * Fluo Workers use this class to create {@link Observer}s to process
notifications. Implementations
+ * of this class should emit zero or more {@link Observer}s.
+ *
+ *
+ * When Fluo is initialized {@link #getObservedColumns(BiConsumer,
Context)} is called. The columns
+ * it emits are stored in Zookeeper. Transactions will use the columns
stored in Zookeeper to
+ * determine when to set notifications. When Workers call
+ * {@link #createObservers(ObserverConsumer, Context)}, the columns
emitted must be the same as
+ * those emitted during initialization. If this is not the case, then the
worker will fail to start.
+ *
+ * @see FluoConfiguration#setObserversFactory(String)
+ * @since 1.1.0
+ */
+public interface ObserverFactory {
+
+ /**
+ * @since 1.1.0
+ */
+ interface Context {
+/**
+ * @return A configuration object with application configuration like
that returned by
+ * {@link FluoClient#getAppConfiguration()}
+ */
+SimpleConfiguration getAppConfiguration();
+
+/**
+ * @return A {@link MetricsReporter} to report application metrics
from observers.
+ */
+MetricsReporter getMetricsReporter();
+ }
+
+ /**
+ * Observers are emitted to the worker using this interface. This
interface also allows
+ * {@link Observer}s to be related to the columns that trigger them.
+ *
+ * @since 1.1.0
+ */
+ interface ObserverConsumer {
+void accept(Column observedColumn, NotificationType ntfyType, Observer
observer);
--- End diff --
This method could be `add` or `register`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r106992791
--- Diff:
modules/api/src/main/java/org/apache/fluo/api/observer/StringObserver.java ---
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.api.observer;
+
+import org.apache.fluo.api.client.TransactionBase;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+
+/**
+ * @since 1.1.0
+ */
+public interface StringObserver extends Observer {
--- End diff --
Could add `@FunctionalInterface` annotation to class
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo/pull/817#discussion_r107001150 --- Diff: pom.xml --- @@ -83,6 +83,11 @@ 1.32 +com.google.code.gson +gson +2.2.4 --- End diff -- Are you using `2.2.4` due to Hadoop using that version? Could add comment if Hadoop restricts use of latest version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo pull request #817: fixes #816 introduced better way to setup ...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/817#discussion_r106990516
--- Diff: docs/applications.md ---
@@ -114,25 +114,45 @@ To create an observer, follow these steps:
// invert row and value
tx.set(value, new Column("inv", "data"), row);
}
+}
+```
+2. Create a class that implements [ObserversFactory] like the example
below. The purpose of this
+class is associate a set Observers with columns that trigger the
observers. The class can
+create multiple observers.
+```java
+class AppObserversFactory implements ObserversFactory {
@Override
- public ObservedColumn getObservedColumn() {
-return new ObservedColumn(new Column("obs", "data"),
NotificationType.STRONG);
+ public void createObservers(ObserverConsumer obsConsumer, Context
ctx) {
+//setup InvertObserver to be triggered when the column obs:data is
modified
+obsConsumer.accept(new Column("obs", "data"),
+ NotificationType.STRONG,
+ new InvertObserver());
+
+//Observer is a Functional interface. So Obsevers can be written
as lambdas.
+obsConcumer.accept(new Column("new","data"),
+ NotificationType.WEAK,
+ (tx,row,col) -> {
--- End diff --
Very nice to have a functional interface now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo-website pull request #51: Added post about Spark+Fluo
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/51#discussion_r93491918 --- Diff: _posts/blog/2016-12-20-spark-load.md --- @@ -0,0 +1,245 @@ +--- +title: "Loading data using Apache Spark" +date: 2016-12-20 11:43:00 + +author: Keith Turner +--- + +[Apache Spark][spark] can be used to preprocess and load batches of data into Fluo. For example +Spark could be used to group data within a batch and then Fluo transactions could load groups of +related data. This blog post offers some tips to help you get started wrting to Fluo from Spark. --- End diff -- should be `writing` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #45: Update (most) old fluo-io links
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/45#discussion_r90491139 --- Diff: _posts/release/2015-06-09-fluo-1.0.0-beta-1.md --- @@ -31,18 +31,18 @@ This release contains many new features that makes it easier to run, develop, an Simplified Fluo administration on a local machine or EC2 cluster Developers can now run Fluo and its dependencies on their local machine ([#92][92]) or an AWS EC2 cluster ([#356][356]) using a few simple commands. -This was done by creating two administration tools called [Fluo-dev][fluo-dev] and [Fluo-deploy][fluo-deploy] whose scripts and configuration reside in repos +This was done by creating two administration tools called [Uno] and [Muchos] whose scripts and configuration reside in repos --- End diff -- I like this change as it does reduce confusion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo issue #799: Use faster byte comparison
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo/pull/799 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo pull request #796: Improved documentation for viewing Fluo lo...
GitHub user mikewalch opened a pull request: https://github.com/apache/incubator-fluo/pull/796 Improved documentation for viewing Fluo logs You can merge this pull request into a Git repository by running: $ git pull https://github.com/mikewalch/incubator-fluo view-logs-docs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-fluo/pull/796.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #796 commit 3d5ee034b8554a7e41f8ac2bd40b182d11d163a8 Author: Mike Walch <mwa...@apache.org> Date: 2016-11-01T19:10:03Z Improved documentation for viewing Fluo logs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website issue #35: added Kryo to powered by, it used by recip...
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo-website/pull/35 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #37: Site redesign
GitHub user mikewalch opened a pull request: https://github.com/apache/incubator-fluo-website/pull/37 Site redesign You can merge this pull request into a Git repository by running: $ git pull https://github.com/mikewalch/incubator-fluo-website site-redesign Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-fluo-website/pull/37.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #37 commit 5350bbee265e7458a5606868ac52a6378ab55101 Author: Mike Walch <mwa...@apache.org> Date: 2016-10-27T20:35:07Z Removed code that should be served from CDN * Removed Boostrap CSS/JS * Removed JQuery * Moved fluo.scss to css dir commit 43f5b268310fadbed65042f5c67c393e1fdb7df0 Author: Mike Walch <mwa...@apache.org> Date: 2016-10-28T00:25:10Z Update site to use Bootswatch Cosmo Theme * Switched using Fluo logo for dark backgrounds * Added bootwatch to powered by * Moved GitHub button to jumbotron and added download button --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo issue #788: Add 'Apache' to Maven module names
GitHub user mikewalch opened an issue: https://github.com/apache/incubator-fluo/issues/788 Add 'Apache' to Maven module names Below are the current names. All should be prefixed with Apache. ``` ./modules/accumulo/pom.xml: Fluo Accumulo Extensions ./modules/api/pom.xml: Fluo API ./modules/cluster/pom.xml: Fluo Cluster ./modules/core/pom.xml: Fluo Core ./modules/distribution/pom.xml: Fluo Distribution ./modules/integration/pom.xml: Fluo Integration ./modules/mapreduce/pom.xml: Fluo MapReduce ./modules/mini/pom.xml: Fluo Mini ./pom.xml: Fluo Project ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website issue #30: Misc website updates
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo-website/pull/30 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #30: Misc website updates
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/30#discussion_r84301838 --- Diff: pages/getinvolved.md --- @@ -8,7 +8,14 @@ Below are some ways that you can get involved with Apache Fluo: ### Mailing list -Join our public mailing list by sending an email to ```dev-subscr...@fluo.incubator.apache.org```. Discussions regarding issues and pull requests should remain on our GitHub pages ([Fluo][f], [Fluo Recipes][r], [Fluo website][w]). Our email list is for announcing releases, organizing meet ups, or any discussion that does not make sense for GitHub. +The `dev` lists is for general disscussion, announcing releases, organizing meet ups, etc. For --- End diff -- Could be "The `dev` mailing list is" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #30: Misc website updates
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/30#discussion_r84302779 --- Diff: pages/related-projects.md --- @@ -5,10 +5,14 @@ permalink: /related-projects/ --- This page list external projects that build on Apache Fluo and may be useful. -These projects are not affiliated with or endorsed by the +These projects are not necessarily affiliated with or endorsed by the [Apache Software Foundation][asf]. If you would like to add a project to this list, please open an issue or submit a pull request [on Github][web-ghr]. +### Projects Using Fluo + +* [Rya] - Rya uses Fluo to keep precomputed joins update to date as new data arrives. --- End diff -- Should be `Apache Rya` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #30: Misc website updates
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/30#discussion_r84302514 --- Diff: pages/getinvolved.md --- @@ -8,7 +8,14 @@ Below are some ways that you can get involved with Apache Fluo: ### Mailing list -Join our public mailing list by sending an email to ```dev-subscr...@fluo.incubator.apache.org```. Discussions regarding issues and pull requests should remain on our GitHub pages ([Fluo][f], [Fluo Recipes][r], [Fluo website][w]). Our email list is for announcing releases, organizing meet ups, or any discussion that does not make sense for GitHub. +The `dev` lists is for general disscussion, announcing releases, organizing meet ups, etc. For +discussion regarding specific issues or pull request consider using Github ([Fluo][f], [Fluo +Recipes][r], [Fluo website][w]). The `notifications` and `commit` list are mailed to by automated --- End diff -- Missing s in commits. Could be "`commits` mailing lists receive email from automated services" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #29: Updated javadocs to point to extern...
GitHub user mikewalch opened a pull request: https://github.com/apache/incubator-fluo-website/pull/29 Updated javadocs to point to external website You can merge this pull request into a Git repository by running: $ git pull https://github.com/mikewalch/incubator-fluo-website web-28 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-fluo-website/pull/29.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #29 commit 44df6c32fe15eee3bc10c5dfd24b265fd176ac82 Author: Mike Walch <mwa...@apache.org> Date: 2016-10-17T14:18:47Z #28 - Removed Fluo & Fluo Recipes javadoc commit 22e3a3d49924f829b728fc33f2d719a59792ae1e Author: Mike Walch <mwa...@apache.org> Date: 2016-10-17T16:38:57Z Fixes #28 - Updated all javadocs links * All javadocs links point to external site * Updated all historical docs, releases, etc * Updated scripts that convert documentation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website issue #28: Link to external Javadoc
GitHub user mikewalch opened an issue: https://github.com/apache/incubator-fluo-website/issues/28 Link to external Javadoc Remove existing javadoc and link to external javadoc instead --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #27: Major refactoring of release posts
GitHub user mikewalch opened a pull request: https://github.com/apache/incubator-fluo-website/pull/27 Major refactoring of release posts * Combined all posts about each release into one * The 'release-notes' category no longer exists * Removed all 'blog' posts about releases. * The blog archive nows shows all releases and was renamed news. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mikewalch/incubator-fluo-website merge-release-posts Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-fluo-website/pull/27.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #27 commit 26199dd57c7e911ac35c7db85457a241dcf392dc Author: Mike Walch <mwa...@apache.org> Date: 2016-10-13T21:03:49Z Major refactoring of release posts * Combined all posts about each release into one * The 'release-notes' category no longer exists * Removed all 'blog' posts about releases. * The blog archive nows shows all releases and was renamed news. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website issue #25: Add link to 'Latest Release' on main page
GitHub user mikewalch opened an issue: https://github.com/apache/incubator-fluo-website/issues/25 Add link to 'Latest Release' on main page From discussion in https://github.com/apache/incubator-fluo-website/pull/23 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/23#discussion_r83278587 --- Diff: docs/fluo/1.0.0-incubating/install.md --- @@ -0,0 +1,210 @@ +--- +layout: fluo-doc +title: Fluo Install Instructions +version: 1.0.0-incubating +--- + +Instructions for installing Apache Fluo and starting a Fluo application in YARN on a cluster where +Accumulo, Hadoop & Zookeeper are running. If you need help setting up these dependencies, see the +[related projects page][related] for external projects that may help. + +## Requirements + +Before you install Fluo, the following software must be installed and running on your local machine +or cluster: + +| Software| Recommended Version | Minimum Version | +|-|-|-| +| [Accumulo] | 1.7.2 | 1.6.1 | +| [Hadoop]| 2.7.2 | 2.6.0 | +| [Zookeeper] | 3.4.8 | | +| [Java] | JDK 8 | JDK 8 | + +## Obtain a distribution + +Before you can install Fluo, you will need to obtain a distribution tarball. It is recommended that +you download the [latest release][release]. You can also build a distribution from the master +branch by following these steps which create a tarball in `modules/distribution/target`: + +git clone https://github.com/apache/incubator-fluo.git +cd fluo/ +mvn package + +## Install Fluo + +After you obtain a Fluo distribution tarball, follow these steps to install Fluo. + +1. Choose a directory with plenty of space and untar the distribution: + +tar -xvzf fluo-1.0.0-incubating-bin.tar.gz + +2. Copy the example configuration to the base of your configuration directory to create the default +configuration for your Fluo install: + +cp conf/examples/* conf/ + +The default configuration will be used as the base configuration for each new application. + +3. Modify [fluo.properties] for your environment. However, you should not configure any +application settings (like observers). + +NOTE - All properties that have a default are set with it. Uncomment a property if you want +to use a value different than the default. Properties that are unset and uncommented must be +set by the user. + +4. Fluo needs to build its classpath using jars from the versions of Hadoop, Accumulo, and +Zookeeper that you are using. Choose one of the two ways below to make these jars available +to Fluo: + +* Set `HADOOP_PREFIX`, `ACCUMULO_HOME`, and `ZOOKEEPER_HOME` in your environment or configure +these variables in [fluo-env.sh]. Fluo will look in these locations for jars. +* Run `./lib/fetch.sh ahz` to download Hadoop, Accumulo, and Zookeeper jars to `lib/ahz` and +configure [fluo-env.sh] to look in this directory. By default, this command will download the +default versions set in [lib/ahz/pom.xml]. If you are not using the default versions, you can +override them: + +./lib/fetch.sh ahz -Daccumulo.version=1.7.2 -Dhadoop.version=2.7.2 -Dzookeeper.version=3.4.8 + +5. Fluo needs more dependencies than what is available from Hadoop, Accumulo, and Zookeeper. These + extra dependencies need to be downloaded to `lib/` using the command below: + +./lib/fetch.sh extra + +You are now ready to use the Fluo command script. + +## Fluo command script + +The Fluo command script is located at `bin/fluo` of your Fluo installation. All Fluo commands are +invoked by this script. + +Modify and add the following to your `~/.bashrc` if you want to be able to execute the fluo script +from any directory: + +export PATH=/path/to/fluo-1.0.0-incubating/bin:$PATH + +Source your `.bashrc` for the changes to take effect and test the script + +source ~/.bashrc +fluo + +Running the script without any arguments prints a description of all commands. + +./bin/fluo + +## Configure a Fluo application + +You are now ready to configure a Fluo application. Use the command below to create the +configuration necessary for a new application. Feel free to pick a different name (other than +`myapp`) for your application: + +fluo new myapp + +This command will create a directory for your application at `apps/myapp` of your Fluo install which +will contain a `conf` and `lib`. + +The `apps/myapp/conf` directory contains a copy of the `fluo.properties` from your default +configuration. This should be configured for your applica
[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/23#discussion_r83278509 --- Diff: _posts/blog/2016-09-27-1.0.0-incubating-is-released.md --- @@ -0,0 +1,12 @@ +--- +title: "Fluo 1.0.0-incubating is released!" +date: 2016-09-27 15:30:00 +--- + +Apache Fluo is happy to announce a [1.0.0-incubating][rel] release. This is the first +release of Fluo as an Apache project and the fourth release for the project. Starting +with this release, Fluo will follow [semver][semver] for all future API changes. See +the [release page][rel] for more information. --- End diff -- This was removed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/23#discussion_r83278537 --- Diff: _posts/release-notes/2016-09-27-fluo-1.0.0-incubating.md --- @@ -1,16 +1,19 @@ --- title: Fluo 1.0.0-incubating Release Notes -date: 2016-10-04 15:30:00 -version: 1.0.0-incubating +date: 2016-09-27 15:30:00 +version: fluo-1.0.0-incubating --- Apache Fluo 1.0.0-incubating is the first release of Fluo as an Apache project and the fourth -release for the project. Since the beta-2 release there were [167 commits][commits] by -[cjmctague], [ctubbsii], [Garvit244], [keith-turner], [mikewalch]. Information about all --- End diff -- This was removed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/23#discussion_r83278560 --- Diff: docs/fluo/1.0.0-incubating/metrics.md --- @@ -0,0 +1,117 @@ +--- +layout: fluo-doc +title: Fluo Metrics +version: 1.0.0-incubating +--- + +A Fluo application can be configured (in [fluo.properties]) to report metrics. When metrics are +configured, Fluo will report some 'default' metrics about an application that help users monitor its +performance. Users can also write code to report 'application-specific' metrics from their +applications. Both 'application-specific' and 'default' metrics share the same reporter configured +by [fluo.properties] and are described in detail below. + +## Configuring reporters + +Fluo metrics are not published by default. To publish metrics, configure a reporter in the 'metrics' +section of [fluo.properties]. There are several different reporter types (i.e Console, CSV, +Graphite, JMX, SLF4J) that are implemented using [Dropwizard]. The choice of which reporter to use +depends on the visualization tool used. If you are not currently using a visualization tool, there +is [documentation][grafana] for reporting Fluo metrics to Grafana/InfluxDB. + +## Metrics names + +When Fluo metrics are reported, they are published using a naming scheme that encodes additional +information. This additional information is represented using all caps variables (i.e `METRIC`) +below. + +Default metrics start with `fluo.class` or `fluo.system` and have following naming schemes: + +fluo.class.APPLICATION.REPORTER_ID.METRIC.CLASS +fluo.system.APPLICATION.REPORTER_ID.METRIC + +Application metrics start with `fluo.app` and have following scheme: + +fluo.app.REPORTER_ID.METRIC + +The variables below describe the additional information that is encoded in metrics names. + +1. `APPLICATION` - Fluo application name +2. `REPORTER_ID` - Unique ID of the Fluo oracle, worker, or client that is reporting the metric. +When running in YARN, this ID is of the format `worker-INSTANCE_ID` or `oracle-INSTANCE_ID` +where `INSTANCE_ID` corresponds to instance number. When not running in YARN, this ID consists +of a hostname and a base36 long that is unique across all fluo processes. +3. `METRIC` - Name of the metric. For 'default' metrics, this is set by Fluo. For 'application' +metrics, this is set by user. Name should be unique and avoid using period '.' in name. +4. `CLASS` - Name of Fluo observer or loader class that produced metric. This allows things like +transaction collisions to be tracked per class. + +## Application-specific metrics + +Application metrics are implemented by retrieving a [MetricsReporter] from an [Observer], [Loader], +or [FluoClient]. These metrics are named using the format `fluo.app.REPORTER_ID.METRIC`. + +## Default metrics + +Default metrics report for a particular Observer/Loader class or system-wide. + +Below are metrics that are reported from each Observer/Loader class that is configured in a Fluo +application. These metrics are reported after each transaction and named using the format +`fluo.class.APPLICATION.REPORTER_ID.METRIC.CLASS`. + +* tx_lock_wait_time - [Timer] +- Time transaction spent waiting on locks held by other transactions. +- Only updated for transactions that have non-zero lock time. +* tx_execution_time - [Timer] +- Time transaction took to execute. +- Updated for failed and successful transactions. +- This does not include commit time, only the time from start until commit is called. +* tx_with_collision - [Meter] +- Rate of transactions with collisions. +* tx_collisions - [Meter] +- Rate of collisions. +* tx_entries_set - [Meter] +- Rate of row/columns set by transaction +* tx_entries_read - [Meter] +- Rate of row/columns read by transaction that existed. +- There is currently no count of all reads (including non-existent data) +* tx_locks_timedout - [Meter] +- Rate of timedout locks rolled back by transaction. +- These are locks that are held for very long periods by another transaction that appears to be + alive based on zookeeper. +* tx_locks_dead - [Meter] +- Rate of dead locks rolled by a transaction. +- These are locks held by a process that appears to be dead according to zookeeper. +* tx_status_`` - [Meter] +- Rate of different ways (i.e ``) a transaction can terminate + +Below are system-wide metrics that are reported for the entire Fluo application. These metrics are +named using the format
[GitHub] incubator-fluo-recipes issue #109: Expose Fluo MetricsReporter in Exporter
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo-recipes/issues/109 Closing this after talking to @keith-turner offline. For implemenations of `Exporter`, the observer context can be retrieved from the `init()` method. For implementations of `AccumuloExporter`, the `init()` method can also be overridden to get observer context if `super.init(context)` is called. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-recipes issue #109: Expose Fluo MetricsReporter in Exporter
Github user mikewalch closed the issue at: https://github.com/apache/incubator-fluo-recipes/issues/109 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website issue #19: added more imports needed by tour
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo-website/pull/19 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-recipes issue #109: Expose Fluo MetricsReporter in Exporter
GitHub user mikewalch opened an issue: https://github.com/apache/incubator-fluo-recipes/issues/109 Expose Fluo MetricsReporter in Exporter It would be nice to be able to report metrics from Exporters. The MetricsReporter should also be exposed to AccumuloExporter. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo pull request #778: Fixes #777 - Set Fluo version in fluo.prop...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo/pull/778#discussion_r80141498 --- Diff: modules/distribution/pom.xml --- @@ -63,6 +63,9 @@ maven-assembly-plugin + + \ --- End diff -- Fixed in edd4e1f118e46 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo pull request #778: Fixes #777 - Set Fluo version in fluo.prop...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo/pull/778#discussion_r80140482 --- Diff: modules/distribution/src/main/assembly/bin.xml --- @@ -94,4 +94,11 @@ + + + src/main/config/fluo.properties + conf/examples --- End diff -- Fixed in 84e90dda52d --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo pull request #778: Fixes #777 - Set Fluo version in fluo.prop...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo/pull/778#discussion_r80138894
--- Diff: modules/distribution/src/main/config/fluo.properties ---
@@ -56,7 +56,7 @@ fluo.admin.hdfs.root=hdfs://localhost:1
# every tablet server. For the default jars below, `fluo init` will place
them
# in HDFS. If you add more jars to the classpath below, you will need to
# add them to HDFS.
-fluo.admin.accumulo.classpath=${fluo.admin.hdfs.root}/fluo/lib/fluo-api-1.0.0-incubating-SNAPSHOT.jar,${fluo.admin.hdfs.root}/fluo/lib/fluo-accumulo-1.0.0-incubating-SNAPSHOT.jar
+fluo.admin.accumulo.classpath=${fluo.admin.hdfs.root}/fluo/lib/fluo-api-${project.version}.jar,${fluo.admin.hdfs.root}/fluo/lib/fluo-accumulo-${project.version}.jar
--- End diff --
Fixed in c95f4ba6cb30c
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo-recipes issue #107: Added standard way to setup per exporter ...
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo-recipes/pull/107 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-recipes pull request #107: Added standard way to setup per ex...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-recipes/pull/107#discussion_r80048501 --- Diff: docs/accumulo-export-queue.md --- @@ -46,10 +46,17 @@ Exporting to Accumulo is easy. Follow the steps below: String password = // Accumulo user password String exportTable =// Name of table to export to -// Configure accumulo export queue -AccumuloExportQueue.configure(fluoConfig, new ExportQueue.Options(EXPORT_QUEUE_ID, -SimpleExporter.class.getName(), String.class.getName(), String.class.getName(), numMapBuckets), -new AccumuloExportQueue.Options(instance, zookeepers, user, password, exportTable)); + +// Create config for export table. +AccumuloExporter.Configuration exportTableCfg = +new AccumuloExporter.Configuration(instance, zookeepers, user, password, exportTable); + +// Create config for export queue. +ExportQueue.Options eqOpts = new ExportQueue.Options(EXPORT_QUEUE_ID, SimpleExporter.class, +String.class, String.class, numMapBuckets).setExporterConfiguration(exportTableCfg); --- End diff -- Nice.. I like this change to the API --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website issue #18: turned quickstart into fluo tour
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo-website/pull/18 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #18: turned quickstart into fluo tour
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo-website/pull/18#discussion_r79886585
--- Diff: src/main/java/ft/Main.java ---
@@ -0,0 +1,44 @@
+package ft;
+
+import java.io.File;
+import java.nio.file.Files;
+
+// Normaly using * with imports is a bad practice, however in this case it
makes experimenting with
+// Fluo easier.
+import org.apache.fluo.api.client.*;
+import org.apache.fluo.api.config.*;
+import org.apache.fluo.api.data.*;
+import org.apache.fluo.api.mini.MiniFluo;
+import org.apache.fluo.api.observer.*;
+
+public class Main {
+ public static void main(String[] args) throws Exception {
+
+String tmpDir = Files.createTempDirectory(new File("target").toPath(),
"mini").toString();
--- End diff --
My comment was that you can replace `new File("target").toPath()` with
`Paths.get("target")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo-recipes pull request #103: Fixes #102 - Simplify Accumulo exp...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo-recipes/pull/103#discussion_r79861790
--- Diff:
modules/accumulo/src/main/java/org/apache/fluo/recipes/accumulo/export/AccumuloExporter.java
---
@@ -16,60 +16,58 @@
package org.apache.fluo.recipes.accumulo.export;
import java.util.ArrayList;
+import java.util.Collection;
import java.util.Iterator;
import org.apache.accumulo.core.data.Mutation;
import org.apache.fluo.api.config.FluoConfiguration;
-import org.apache.fluo.api.config.SimpleConfiguration;
import org.apache.fluo.api.observer.Observer.Context;
+import org.apache.fluo.recipes.core.export.ExportQueue;
import org.apache.fluo.recipes.core.export.Exporter;
import org.apache.fluo.recipes.core.export.SequencedExport;
/**
- * An {@link Exporter} that takes {@link AccumuloExport} objects and
writes mutations to Accumulo
+ * An Accumulo-specific {@link Exporter} that writes mutations to Accumulo
using a
+ * {@link AccumuloWriter}
*
- * @param Export queue key type
* @since 1.0.0
*/
-public class AccumuloExporter extends Exporter<K, AccumuloExport> {
+public abstract class AccumuloExporter<K, V> extends Exporter<K, V> {
- private SharedBatchWriter sbw;
+ private AccumuloWriter accumuloWriter;
- @Override
- public void init(String queueId, Context context) throws Exception {
-
-SimpleConfiguration appConf = context.getAppConfiguration();
-
-String instanceName = appConf.getString("recipes.accumuloExporter." +
queueId + ".instance");
-String zookeepers = appConf.getString("recipes.accumuloExporter." +
queueId + ".zookeepers");
-String user = appConf.getString("recipes.accumuloExporter." + queueId
+ ".user");
-// TODO look into using delegation token
-String password = appConf.getString("recipes.accumuloExporter." +
queueId + ".password");
-String table = appConf.getString("recipes.accumuloExporter." + queueId
+ ".table");
-
-sbw = SharedBatchWriter.getInstance(instanceName, zookeepers, user,
password, table);
+ /**
+ * Configures AccumuloExporter
+ *
+ * @param config Fluo configuration
+ * @param opts Export queue options
+ * @param writerConf Accumulo writer config
+ */
+ public static void configure(FluoConfiguration config,
ExportQueue.Options opts,
+ AccumuloWriter.Config writerConf) {
+ExportQueue.configure(config, opts);
+AccumuloWriter.setConfig(config.getAppConfiguration(),
opts.getQueueId(), writerConf);
}
- public static void setExportTableInfo(FluoConfiguration fconf, String
queueId, TableInfo ti) {
-SimpleConfiguration appConf = fconf.getAppConfiguration();
-appConf.setProperty("recipes.accumuloExporter." + queueId +
".instance", ti.instanceName);
-appConf.setProperty("recipes.accumuloExporter." + queueId +
".zookeepers", ti.zookeepers);
-appConf.setProperty("recipes.accumuloExporter." + queueId + ".user",
ti.user);
-appConf.setProperty("recipes.accumuloExporter." + queueId +
".password", ti.password);
-appConf.setProperty("recipes.accumuloExporter." + queueId + ".table",
ti.table);
+ @Override
+ public void init(String queueId, Context context) throws Exception {
+accumuloWriter =
AccumuloWriter.getInstance(context.getAppConfiguration(), queueId);
}
@Override
- protected void processExports(Iterator<SequencedExport<K,
AccumuloExport>> exports) {
+ protected void processExports(Iterator<SequencedExport<K, V>> exports) {
+
ArrayList buffer = new ArrayList<>();
while (exports.hasNext()) {
- SequencedExport<K, AccumuloExport> export = exports.next();
- buffer.addAll(export.getValue().toMutations(export.getKey(),
export.getSequence()));
+ SequencedExport<K, V> export = exports.next();
+ buffer.addAll(processExport(export));
}
if (buffer.size() > 0) {
- sbw.write(buffer);
+ accumuloWriter.write(buffer);
}
}
+
+ protected abstract Collection processExport(SequencedExport<K,
V> export);
--- End diff --
That makes sense
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo-website pull request #18: turned quickstart into fluo tour
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/18#discussion_r79828012 --- Diff: src/main/java/ft/Main.java --- @@ -0,0 +1,44 @@ +package ft; + +import java.io.File; +import java.nio.file.Files; + +// Normaly using * with imports is a bad practice, however in this case it makes experimenting with --- End diff -- Should be `Normally` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #18: turned quickstart into fluo tour
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo-website/pull/18#discussion_r79831447
--- Diff: src/main/java/ft/Main.java ---
@@ -0,0 +1,44 @@
+package ft;
+
+import java.io.File;
+import java.nio.file.Files;
+
+// Normaly using * with imports is a bad practice, however in this case it
makes experimenting with
+// Fluo easier.
+import org.apache.fluo.api.client.*;
+import org.apache.fluo.api.config.*;
+import org.apache.fluo.api.data.*;
+import org.apache.fluo.api.mini.MiniFluo;
+import org.apache.fluo.api.observer.*;
+
+public class Main {
+ public static void main(String[] args) throws Exception {
+
+String tmpDir = Files.createTempDirectory(new File("target").toPath(),
"mini").toString();
--- End diff --
Could use Paths instead of File:
```java
String tmpDir = Files.createTempDirectory(Paths.get("target"),
"mini-").toString();
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-fluo-website pull request #18: turned quickstart into fluo tour
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/18#discussion_r79826320 --- Diff: README.md --- @@ -1,4 +1,17 @@ Fluo Tour - +The git repository provides a barebones Maven+Java environment for the [Fluo Tour][tour]. As you go +through the Tour edit [Main.java] and then use the following command to execute Main. This command --- End diff -- could be `use the following command to get all of the correct dependencies on the classpath and execute Main` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-recipes pull request #103: Fixes #102 - Simplify Accumulo exp...
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/incubator-fluo-recipes/pull/103#discussion_r79475688
--- Diff:
modules/accumulo/src/main/java/org/apache/fluo/recipes/accumulo/export/AccumuloWriter.java
---
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.recipes.accumulo.export;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.function.Predicate;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.BatchWriterConfig;
+import org.apache.accumulo.core.client.ClientConfiguration;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.TableExistsException;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.client.ZooKeeperInstance;
+import org.apache.accumulo.core.client.security.tokens.PasswordToken;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.security.ColumnVisibility;
+import org.apache.fluo.api.config.FluoConfiguration;
+import org.apache.fluo.api.config.SimpleConfiguration;
+import org.apache.fluo.api.data.Bytes;
+import org.apache.fluo.api.data.Column;
+import org.apache.fluo.api.data.RowColumn;
+import org.apache.fluo.recipes.core.transaction.LogEntry;
+import org.apache.fluo.recipes.core.transaction.TxLog;
+
+/**
+ * Writes mutations to Accumulo using a shared batch writer
+ *
+ * @since 1.0.0
+ */
+public class AccumuloWriter {
+
+ private static class Mutations {
+List mutations;
+CountDownLatch cdl = new CountDownLatch(1);
+
+Mutations(Collection mutations) {
+ this.mutations = new ArrayList<>(mutations);
+}
+ }
+
+ /**
+ * Sets AccumuloWriter config in app configuration
+ */
+ public static void setConfig(SimpleConfiguration conf, String id, Config
ti) {
+String prefix = "recipes.accumulo.writer." + id;
+conf.setProperty(prefix + ".instance", ti.instanceName);
+conf.setProperty(prefix + ".zookeepers", ti.zookeepers);
+conf.setProperty(prefix + ".user", ti.user);
+conf.setProperty(prefix + ".password", ti.password);
+conf.setProperty(prefix + ".table", ti.table);
+ }
+
+ /**
+ * Gets Accumulo Config from app configuration
+ */
+ public static Config getConfig(SimpleConfiguration conf, String id) {
+String prefix = "recipes.accumulo.writer." + id;
+String instanceName = conf.getString(prefix + ".instance");
+String zookeepers = conf.getString(prefix + ".zookeepers");
+String user = conf.getString(prefix + ".user");
+String password = conf.getString(prefix + ".password");
+String table = conf.getString(prefix + ".table");
+return new Config(instanceName, zookeepers, user, password, table);
+ }
+
+ private static class ExportTask implements Runnable {
+
+private BatchWriter bw;
+
+ExportTask(String instanceName, String zookeepers, String user, String
password, String table)
+throws TableNotFoundException, AccumuloException,
AccumuloSecurityException {
+ ZooKeeperInstance zki =
+ new ZooKeeperInstance(new
ClientConfiguration().withInstance(instanceName).withZkHosts(
+ zookeepers));
+
+ // TODO need to close batch writer
[GitHub] incubator-fluo pull request #767: Fixes #534 - Enable reporting application ...
Github user mikewalch closed the pull request at: https://github.com/apache/incubator-fluo/pull/767 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo pull request #767: Fixes #534 - Enable reporting application ...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo/pull/767#discussion_r79247465 --- Diff: docs/metrics.md --- @@ -1,69 +1,112 @@ # Fluo Metrics -Fluo core is instrumented using [dropwizard metrics][1]. This allows fluo users to easily gather -information about Fluo by configuring different reporters. While dropwizard can be configured to -report Fluo metrics to many different tools, below are some tools that have been used with Fluo. +A Fluo application can be configured (in [fluo.properties]) to report metrics. When metrics are +configured, Fluo will report will some 'default' metrics about an application that helps users +monitor its performance. Users can also write code to report 'application' metrics from their --- End diff -- fixed in 7e0c08fec69e --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo pull request #767: Fixes #534 - Enable reporting application ...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo/pull/767#discussion_r79247441 --- Diff: docs/metrics.md --- @@ -1,69 +1,112 @@ # Fluo Metrics -Fluo core is instrumented using [dropwizard metrics][1]. This allows fluo users to easily gather -information about Fluo by configuring different reporters. While dropwizard can be configured to -report Fluo metrics to many different tools, below are some tools that have been used with Fluo. +A Fluo application can be configured (in [fluo.properties]) to report metrics. When metrics are +configured, Fluo will report will some 'default' metrics about an application that helps users +monitor its performance. Users can also write code to report 'application' metrics from their +applications. Both 'application' and 'default' metrics share the same reporter configured by +[fluo.properties] and are described in detail below. -1. [Grafana/InfluxDB][3] - Fluo has [documentation][3] for sending metrics to InfluxDB and viewing -them in Grafana. +## Configuring reporters -2. JMX - Fluo can be configured to reports metrics via JMX which can be viewed in jconsole or -jvisualvm. +Fluo metrics are not published by default. To publish metrics, configure a reporter in the 'metrics' +section of [fluo.properties]. There are several different reporter types (i.e Console, CSV, +Graphite, JMX, SLF4J) that are implemented using [Dropwizard]. The choice of which reporter to use +depends on the visualization tool used. If you are not currently using a visualization tool, there +is [documentation][grafana] for reporting Fluo metrics to Grafana/InfluxDB. -3. CSV - Fluo can be configured to output metrics as CSV to a specified directory. +## Metrics names -## Configuring Reporters +When Fluo metrics are reported, they are published using a naming scheme that encodes additional +information. -In order to configure metrics reporters, look at the metrics section in an applications -`fluo.properties` file. This sections has a lot of commented out options for configuring reporters. +Default metrics start with `fluo.class` or `fluo.system` and have following naming schemes: -fluo.metrics.reporter.console.enable=false -fluo.metrics.reporter.console.frequency=30 +fluo.class.APPLICATION.REPORTER_ID.METRIC.CLASS --- End diff -- i tried adding greater/less than but i think it makes it harder to read. i made it more explicit that all caps are variables in 7e0c08fec69e --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo pull request #767: Fixes #534 - Enable reporting application ...
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo/pull/767#discussion_r79247279 --- Diff: docs/metrics.md --- @@ -1,69 +1,112 @@ # Fluo Metrics -Fluo core is instrumented using [dropwizard metrics][1]. This allows fluo users to easily gather -information about Fluo by configuring different reporters. While dropwizard can be configured to -report Fluo metrics to many different tools, below are some tools that have been used with Fluo. +A Fluo application can be configured (in [fluo.properties]) to report metrics. When metrics are +configured, Fluo will report will some 'default' metrics about an application that helps users --- End diff -- fixed in 7e0c08fec69e --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo issue #767: Fixes #534 - Enable reporting application metrics
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo/pull/767 This PR doesn't have to be included in 1.0.0 but if can get in that would be great. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-recipes issue #101: Fixed CompactTransient, it was not pickin...
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo-recipes/pull/101 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-recipes issue #98: Update for changes from apache/incubator-f...
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo-recipes/pull/98 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo issue #761: Found and fixed two bugs while working on the Flu...
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo/pull/761 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo issue #761: Found and fixed two bugs while working on the Flu...
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo/pull/761 I was talking about TravisCI which looks like it has tests that are failing and not dep issues. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo issue #756: Made some methods in API private
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo/pull/756 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo issue #757: fix scanner() javadoc
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo/pull/757 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo issue #755: Fixed bug in suggested InfluxDB config
Github user mikewalch commented on the issue: https://github.com/apache/incubator-fluo/pull/755 yes.. they were being stored incorrectly so dashboards did not work. Bug started when metrics prefix was renamed from `io.fluo` to `fluo` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo pull request #755: Fixed bug in suggested InfluxDB config
GitHub user mikewalch opened a pull request: https://github.com/apache/incubator-fluo/pull/755 Fixed bug in suggested InfluxDB config * Bug was causing metrics to be stored incorrectly You can merge this pull request into a Git repository by running: $ git pull https://github.com/mikewalch/incubator-fluo metrics-bug-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-fluo/pull/755.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #755 commit 0ba6eda188f426bc18805b4c08cf8105b5a9a7bc Author: Mike Walch <mwa...@gmail.com> Date: 2016-08-08T15:33:48Z Fixed bug in suggested InfluxDB config * Bug was causing metrics to be stored incorrectly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #15: Added place holder tour page
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/15#discussion_r73745205 --- Diff: pages/tour.md --- @@ -0,0 +1,10 @@ +--- +layout: page +title: Apache Fluo Tour +permalink: /tour/ +--- + +Shortly after Fluo 1.0.0-incubating is released, a step by step introduction with hands on examples will be placed here. For now this page is a palce holder so that the documentation that will ship with Fluo can be modified to point here before release. --- End diff -- should be `place` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
