[GitHub] storm pull request: STORM-1075 add external module storm-cassandra
Github user jnioche commented on a diff in the pull request:
https://github.com/apache/storm/pull/827#discussion_r45128717
--- Diff:
external/storm-cassandra/src/main/java/org/apache/storm/cassandra/Murmur3StreamGrouping.java
---
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.storm.cassandra;
+
+import backtype.storm.generated.GlobalStreamId;
+import backtype.storm.grouping.CustomStreamGrouping;
+import backtype.storm.task.WorkerTopologyContext;
+import backtype.storm.topology.FailedException;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import com.google.common.hash.Hashing;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.List;
+
+/**
+ *
+ * Simple {@link backtype.storm.grouping.CustomStreamGrouping} that uses
Murmur3 algorithm to choose the target task of a tuple.
+ *
+ * This stream grouping may be used to optimise writes to Apache Cassandra.
+ */
+public class Murmur3StreamGrouping implements CustomStreamGrouping {
--- End diff --
Would it make sense to provide this class as part of the code module? It
was mentioned in a [discussion on the user
list](https://mail-archives.apache.org/mod_mbox/incubator-storm-user/201510.mbox/%3ccac19-6akwc8ch681muaq1lsfkwiofqhaffjkesfu04lpvor...@mail.gmail.com%3E)
not long ago and it sounded like it gives better results in same cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1075 add external module storm-cassandra
Github user jnioche commented on a diff in the pull request:
https://github.com/apache/storm/pull/827#discussion_r45129715
--- Diff:
external/storm-cassandra/src/main/java/org/apache/storm/cassandra/bolt/BatchCassandraWriterBolt.java
---
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.storm.cassandra.bolt;
+
+import backtype.storm.Config;
+import backtype.storm.task.OutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.tuple.Tuple;
+import backtype.storm.utils.Time;
+import com.datastax.driver.core.Statement;
+import org.apache.storm.cassandra.executor.AsyncResultHandler;
+import org.apache.storm.cassandra.executor.impl.BatchAsyncResultHandler;
+import org.apache.storm.cassandra.query.CQLStatementTupleMapper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+public class BatchCassandraWriterBolt extends
BaseCassandraBolt> {
+
+private final static Logger LOG =
LoggerFactory.getLogger(BatchCassandraWriterBolt.class);
+
+public static final int DEFAULT_EMIT_FREQUENCY = 2;
+
+private static final int QUEUE_MAX_SIZE = 1000;
+
+private LinkedBlockingQueue queue;
+
+private int tickFrequencyInSeconds;
+
+private long lastModifiedTimesMillis;
+
+private String componentID;
+
+private AsyncResultHandler> asyncResultHandler;
+
+/**
+ * Creates a new {@link CassandraWriterBolt} instance.
+ *
+ * @param tupleMapper
+ */
+public BatchCassandraWriterBolt(CQLStatementTupleMapper tupleMapper) {
+this(tupleMapper, DEFAULT_EMIT_FREQUENCY);
+}
+
+/**
+ * Creates a new {@link CassandraWriterBolt} instance.
+ *
+ * @param tupleMapper
+ */
+public BatchCassandraWriterBolt(CQLStatementTupleMapper tupleMapper,
int tickFrequencyInSeconds) {
+super(tupleMapper);
+this.tickFrequencyInSeconds = tickFrequencyInSeconds;
+}
+/**
+ * {@inheritDoc}
+ */
+@Override
+public void prepare(Map stormConfig, TopologyContext topologyContext,
OutputCollector outputCollector) {
+super.prepare(stormConfig, topologyContext, outputCollector);
+this.componentID = topologyContext.getThisComponentId();
+this.queue = new LinkedBlockingQueue<>(QUEUE_MAX_SIZE);
+this.lastModifiedTimesMillis = now();
+}
+
+@Override
+protected AsyncResultHandler> getAsyncHandler() {
+if( asyncResultHandler == null) {
+asyncResultHandler = new
BatchAsyncResultHandler(getResultHandler());
+}
+return asyncResultHandler;
+}
+
+/**
+ * {@inheritDoc}
+ */
+@Override
+protected void process(Tuple input) {
+if( ! queue.offer(input) ) {
+LOG.info(logPrefix() + "The message queue is full, preparing
batch statement...");
+prepareAndExecuteStatement();
+queue.add(input);
+}
+}
+/**
+ * {@inheritDoc}
+ */
+@Override
+protected void tick() {
+prepareAndExecuteStatement();
+}
+
+public void prepareAndExecuteStatement() {
+int size = queue.size();
+if( size > 0 ) {
+List inputs = new ArrayList<>(size);
+queue.drainTo(inputs);
+try {
+List psl = buildStatement(inputs);
+
+int sinceLastModified =
updateAndGetSecondsSinceLastModified();
+LOG.debug(logPrefix() + String.format("Exec
[GitHub] storm pull request: STORM-1075 add external module storm-cassandra
Github user jnioche commented on a diff in the pull request:
https://github.com/apache/storm/pull/827#discussion_r45130329
--- Diff:
external/storm-cassandra/src/main/java/org/apache/storm/cassandra/client/CassandraConf.java
---
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.storm.cassandra.client;
+
+import com.datastax.driver.core.ConsistencyLevel;
+import com.google.common.base.Objects;
+
+import java.io.Serializable;
+import java.util.Map;
+
+/**
+ * Configuration used by cassandra storm components.
+ */
+public class CassandraConf implements Serializable {
+
+public static final String CASSANDRA_USERNAME =
"cassandra.username";
+public static final String CASSANDRA_PASSWORD =
"cassandra.password";
+public static final String CASSANDRA_KEYSPACE =
"cassandra.keyspace";
+public static final String CASSANDRA_CONSISTENCY_LEVEL =
"cassandra.output.consistencyLevel";
+public static final String CASSANDRA_NODES =
"cassandra.nodes";
+public static final String CASSANDRA_PORT =
"cassandra.port";
+public static final String CASSANDRA_BATCH_SIZE_ROWS=
"cassandra.batch.size.rows";
+
+/**
+ * The authorized cassandra username.
+ */
+private String username;
+/**
+ * The authorized cassandra password
+ */
+private String password;
+/**
+ * The cassandra keyspace.
+ */
+private String keyspace;
+/**
+ * List of contacts nodes.
+ */
+private String[] nodes = {"localhost"};
+
+/**
+ * The port used to connect to nodes.
+ */
+private int port = 9092;
+
+/**
+ * Consistency level used to write statements.
+ */
+private ConsistencyLevel consistencyLevel = ConsistencyLevel.ONE;
+/**
+ * The maximal numbers of rows per batch.
+ */
+private int batchSizeRows = 100;
+
+/**
+ * Creates a new {@link CassandraConf} instance.
+ */
+public CassandraConf() {
+super();
+}
+
+/**
+ * Creates a new {@link CassandraConf} instance.
+ *
+ * @param conf The storm configuration.
+ */
+public CassandraConf(Map conf) {
+this.username = getOrElse(conf, CASSANDRA_USERNAME, null);
+this.password = getOrElse(conf, CASSANDRA_PASSWORD, null);
+this.keyspace = get(conf, CASSANDRA_KEYSPACE);
+this.consistencyLevel = ConsistencyLevel.valueOf(getOrElse(conf,
CASSANDRA_CONSISTENCY_LEVEL, ConsistencyLevel.ONE.name()));
+this.nodes= getOrElse(conf, CASSANDRA_NODES,
"localhost").split(",");
+this.batchSizeRows = getOrElse(conf, CASSANDRA_BATCH_SIZE_ROWS,
100);
+this.port = conf.get(CASSANDRA_PORT) != null ?
Integer.valueOf((String)conf.get(CASSANDRA_PORT)) : 9042;
+}
+
+public String getUsername() {
+return username;
+}
+
+public String getPassword() {
+return password;
+}
+
+public String getKeyspace() {
+return keyspace;
+}
+
+public String[] getNodes() {
+return nodes;
+}
+
+public ConsistencyLevel getConsistencyLevel() {
+return consistencyLevel;
+}
+
+public int getBatchSizeRows() {
+return batchSizeRows;
+}
+
+public int getPort() {
+return this.port;
+}
+
+private T get(Map conf, String key) {
+Object o = conf.get(key);
+if(o == null) {
+throw new IllegalArgumentException("No '" + key + "' value
found in configur
[GitHub] storm pull request: storm-hdfs : change visibility of create and c...
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/1122 storm-hdfs : change visibility of create and closeOutputFile methods to protected When extending HdfsBolt it can be useful to be able to override the methods createOutputFile() or closeOutputFile(), for instance to add a header or footer to the files created. This is currently not possible when the class is outside the `org.apache.storm.hdfs.bolt` package. This patch makes it possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/DigitalPebble/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1122.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1122 commit bcb40e481f189cc9f77ddcaeddf7d4daa556113b Author: Julien Nioche Date: 2016-02-18T16:25:40Z storm-hdfs : change visibility of create and closeOutputFile methods to protected --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: storm-hdfs : change visibility of create and c...
Github user jnioche commented on the pull request: https://github.com/apache/storm/pull/1122#issuecomment-186435209 @dossett good idea - I have just committed your suggestion @revans2 and @ptgoetz \: will the content of the master branch become 2.x and if so could this be committed to the next 1.x release? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: storm-hdfs : change visibility of create and c...
Github user jnioche commented on the pull request: https://github.com/apache/storm/pull/1122#issuecomment-186561314 @ptgoetz perfect, thanks. BTW I really like the HDFS module, great work! Would be good to have compression for HdfsBolt, I might have a go at implementing that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update doc for rebalance command
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/1130 Update doc for rebalance command Trivial update for the rebalance command - now similar to the output of 'storm help rebalance' You can merge this pull request into a Git repository by running: $ git pull https://github.com/DigitalPebble/storm asf-site Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1130.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1130 commit e9eb307dac90a4fae40113235a006ffb2e81d383 Author: Julien Nioche Date: 2016-02-20T21:10:48Z Update doc for rebalance command --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: storm-hdfs : change visibility of create and c...
Github user jnioche commented on the pull request: https://github.com/apache/storm/pull/1122#issuecomment-188872715 Anything blocking this from being committed (apart from committers' availability)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: storm-hdfs : change visibility of create and c...
Github user jnioche commented on the pull request: https://github.com/apache/storm/pull/1122#issuecomment-188994648 thanks for having taken the time to do it @revans2, much appreciated --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update doc for rebalance command
Github user jnioche commented on the pull request: https://github.com/apache/storm/pull/1130#issuecomment-196814390 Sorry for bumping it, but it is a trivial commit and I think it would be good to update the doc - especially prior to the new releases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Fix logging for LoggingMetricsConsumer on bran...
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/1222 Fix logging for LoggingMetricsConsumer on branch 0.10.x STORM-584 Port of #1221 to branch 0.10 You can merge this pull request into a Git repository by running: $ git pull https://github.com/DigitalPebble/storm 0.10.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1222.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1222 commit db5fbbde84e456de4594edf0a28e8982b5d00f12 Author: Julien Nioche Date: 2016-03-16T11:22:52Z Fix logging for LoggingMetricsConsumer on branch 0.10.x STORM-584 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Fix logging for LoggingMetricsConsumer STORM-5...
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/1221 Fix logging for LoggingMetricsConsumer STORM-584 Currently the metrics.log file is always empty and it declared in the cluster.xml file whereas it should be in the worker one. The metrics end up in the log file which is not a huge issue but it would be cleaner to have them in a separate file, as originally intended. This patch addresses this and makes so that there is one metrics file per worker, as per the other log files, which should address the issues mentioned in [https://issues.apache.org/jira/browse/STORM-584]. You can merge this pull request into a Git repository by running: $ git pull https://github.com/DigitalPebble/storm metricsFile Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1221.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1221 commit 4d75ec8494eeda24d6510641259d4c469cff3ee2 Author: Julien Nioche Date: 2016-03-16T11:14:24Z Fix logging for LoggingMetricsConsumer STORM-584 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1466: Update worker.xml
Github user jnioche commented on the issue: https://github.com/apache/storm/pull/1466 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Fixed incubator URLs
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/828 Fixed incubator URLs Replaces links to 'storm.incubator.apache.org' with 'storm.apache.org' and 'github.com/apache/incubator-storm' with 'github.com/apache/storm' This also removes the DISCLAIMER file now that the project has become top level. You can merge this pull request into a Git repository by running: $ git pull https://github.com/DigitalPebble/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/828.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #828 commit acca57205c731a48b5285bfe85c5ea55e6222108 Author: Julien Nioche Date: 2015-10-29T12:37:18Z Fixed incubation URLs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Fixed incubator URLs
Github user jnioche commented on the pull request: https://github.com/apache/storm/pull/828#issuecomment-152205114 thanks for your comments @kishorvpatil This particular document is outdated - instead of searching for the relevant line in the master branch I fixed the URLs by pointing them to the branch 0.7.1 which was relevant at the time this document was written. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1148 - Removed references to incubator i...
Github user jnioche commented on the pull request: https://github.com/apache/storm/pull/853#issuecomment-153671366 Duplicates [https://github.com/apache/storm/pull/828] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1896: STORM-2326 Upgrade log4j and slf4j
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/1896 STORM-2326 Upgrade log4j and slf4j Ran ` mvn clean install -DskipTests` without problems You can merge this pull request into a Git repository by running: $ git pull https://github.com/jnioche/storm STORM-2326 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1896.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1896 commit 5f9bffbbb15f9964612b48e91db9b24c710a251b Author: Julien Nioche Date: 2017-01-26T12:23:35Z STORM-2326 Upgrade log4j and slf4j --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1896: STORM-2326 Upgrade log4j and slf4j
Github user jnioche commented on the issue: https://github.com/apache/storm/pull/1896 As far as I understand, Storm uses SLF4J + LOG4J. I could not find a way to run a topology where one of the dependencies relies on a more recent version of log4j (Elasticsearch 5). A similar problem was reported on https://issues.apache.org/jira/browse/STORM-1386. Upgrading the dependencies in Storm solved the problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1900: Port STORM-2326 Upgrade log4j and slf4j to branch ...
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/1900 Port STORM-2326 Upgrade log4j and slf4j to branch 1.0.x See #1896 You can merge this pull request into a Git repository by running: $ git pull https://github.com/jnioche/storm STORM-2326-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1900.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1900 commit b1258a486194332a7dfa79ba3dbda64365811634 Author: Julien Nioche Date: 2017-01-27T11:39:51Z Port STORM-2326 Upgrade log4j and slf4j to branch 1.0.x --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1900: Port STORM-2326 Upgrade log4j and slf4j to branch ...
Github user jnioche closed the pull request at: https://github.com/apache/storm/pull/1900 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1900: Port STORM-2326 Upgrade log4j and slf4j to branch 1.0.x
Github user jnioche commented on the issue: https://github.com/apache/storm/pull/1900 Needs more work. I am getting the following in supervisor.log ``` 2017-01-27 11:45:55.788 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd:Exception in thread "main" java.lang.ExceptionInInitializerError 2017-01-27 11:45:55.788 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.ThreadContextAccess.getThreadContextMap(ThreadContextAccess.java:45) 2017-01-27 11:45:55.788 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.core.impl.ContextDataInjectorFactory.createDefaultInjector(ContextDataInjectorFactory.java:83) 2017-01-27 11:45:55.788 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.core.impl.ContextDataInjectorFactory.createInjector(ContextDataInjectorFactory.java:67) 2017-01-27 11:45:55.788 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.core.lookup.ContextMapLookup.(ContextMapLookup.java:34) 2017-01-27 11:45:55.788 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.core.lookup.Interpolator.(Interpolator.java:116) 2017-01-27 11:45:55.788 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.core.config.AbstractConfiguration.(AbstractConfiguration.java:120) 2017-01-27 11:45:55.788 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.core.config.NullConfiguration.(NullConfiguration.java:32) 2017-01-27 11:45:55.788 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.core.LoggerContext.(LoggerContext.java:72) 2017-01-27 11:45:55.788 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.core.selector.BasicContextSelector.(BasicContextSelector.java:32) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at java.lang.reflect.Constructor.newInstance(Constructor.java:423) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.util.LoaderUtil.newInstanceOf(LoaderUtil.java:158) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.util.LoaderUtil.newInstanceOf(LoaderUtil.java:180) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.util.LoaderUtil.newCheckedInstanceOf(LoaderUtil.java:201) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.util.LoaderUtil.newCheckedInstanceOfProperty(LoaderUtil.java:226) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.core.impl.Log4jContextFactory.createContextSelector(Log4jContextFactory.java:97) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at org.apache.logging.log4j.core.impl.Log4jContextFactory.(Log4jContextFactory.java:58) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a21163fd: at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 2017-01-27 11:45:55.789 Thread-12 o.a.s.u.Utils [INFO] Worker Process 9318e633-f5fc-476f-825c-2f68a2
[GitHub] storm pull request #1896: STORM-2326 Upgrade log4j and slf4j
Github user jnioche closed the pull request at: https://github.com/apache/storm/pull/1896 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1896: STORM-2326 Upgrade log4j and slf4j
Github user jnioche commented on the issue: https://github.com/apache/storm/pull/1896 See issue with #1900 - closing for now --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1897: STORM-2327 ConfigurableTopology
Github user jnioche commented on the issue: https://github.com/apache/storm/pull/1897 Thanks @HeartSaVioR. Good idea, I'll have a look at the topology classes in storm-starter. Just came across the (StormRunner)[https://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/org/apache/storm/starter/util/StormRunner.java] class, which the ConfigurableTopology could replace. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1897: STORM-2327 ConfigurableTopology
Github user jnioche commented on the issue: https://github.com/apache/storm/pull/1897 @HeartSaVioR I removed StormRunner and modified some of the topologies so that they extend ConfigurableTopology. Please let me know your thoughts on this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1897: STORM-2327 ConfigurableTopology
Github user jnioche commented on a diff in the pull request:
https://github.com/apache/storm/pull/1897#discussion_r98649259
--- Diff:
examples/storm-starter/src/jvm/org/apache/storm/starter/RollingTopWords.java ---
@@ -17,114 +17,89 @@
*/
package org.apache.storm.starter;
-import org.apache.storm.Config;
-import org.apache.storm.testing.TestWordSpout;
-import org.apache.storm.topology.TopologyBuilder;
-import org.apache.storm.tuple.Fields;
-import org.apache.log4j.Logger;
import org.apache.storm.starter.bolt.IntermediateRankingsBolt;
import org.apache.storm.starter.bolt.RollingCountBolt;
import org.apache.storm.starter.bolt.TotalRankingsBolt;
-import org.apache.storm.starter.util.StormRunner;
+import org.apache.storm.testing.TestWordSpout;
+import org.apache.storm.topology.ConfigurableTopology;
+import org.apache.storm.topology.TopologyBuilder;
+import org.apache.storm.tuple.Fields;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
- * This topology does a continuous computation of the top N words that the
topology has seen in terms of cardinality.
- * The top N computation is done in a completely scalable way, and a
similar approach could be used to compute things
- * like trending topics or trending images on Twitter.
+ * This topology does a continuous computation of the top N words that the
+ * topology has seen in terms of cardinality. The top N computation is
done in a
+ * completely scalable way, and a similar approach could be used to compute
+ * things like trending topics or trending images on Twitter.
*/
-public class RollingTopWords {
+public class RollingTopWords extends ConfigurableTopology {
- private static final Logger LOG =
Logger.getLogger(RollingTopWords.class);
- private static final int DEFAULT_RUNTIME_IN_SECONDS = 60;
+ private static final Logger LOG =
LoggerFactory.getLogger(RollingTopWords.class);
private static final int TOP_N = 5;
- private final TopologyBuilder builder;
- private final String topologyName;
- private final Config topologyConfig;
- private final int runtimeInSeconds;
-
- public RollingTopWords(String topologyName) throws InterruptedException {
-builder = new TopologyBuilder();
-this.topologyName = topologyName;
-topologyConfig = createTopologyConfiguration();
-runtimeInSeconds = DEFAULT_RUNTIME_IN_SECONDS;
-
-wireTopology();
- }
-
- private static Config createTopologyConfiguration() {
-Config conf = new Config();
-conf.setDebug(true);
-return conf;
- }
-
- private void wireTopology() throws InterruptedException {
-String spoutId = "wordGenerator";
-String counterId = "counter";
-String intermediateRankerId = "intermediateRanker";
-String totalRankerId = "finalRanker";
-builder.setSpout(spoutId, new TestWordSpout(), 5);
-builder.setBolt(counterId, new RollingCountBolt(9, 3),
4).fieldsGrouping(spoutId, new Fields("word"));
-builder.setBolt(intermediateRankerId, new
IntermediateRankingsBolt(TOP_N), 4).fieldsGrouping(counterId, new Fields(
-"obj"));
-builder.setBolt(totalRankerId, new
TotalRankingsBolt(TOP_N)).globalGrouping(intermediateRankerId);
- }
-
- public void runLocally() throws InterruptedException {
-StormRunner.runTopologyLocally(builder.createTopology(), topologyName,
topologyConfig, runtimeInSeconds);
+ private RollingTopWords() {
}
- public void runRemotely() throws Exception {
-StormRunner.runTopologyRemotely(builder.createTopology(),
topologyName, topologyConfig);
+ public static void main(String[] args) throws Exception {
+ConfigurableTopology.start(new RollingTopWords(), args);
}
/**
* Submits (runs) the topology.
*
- * Usage: "RollingTopWords [topology-name] [local|remote]"
+ * Usage: "RollingTopWords [topology-name] [-local]"
*
- * By default, the topology is run locally under the name
"slidingWindowCounts".
+ * By default, the topology is run locally under the name
+ * "slidingWindowCounts".
*
* Examples:
*
* ```
*
- * # Runs in local mode (LocalCluster), with topology name
"slidingWindowCounts"
- * $ storm jar storm-starter-jar-with-dependencies.jar
org.apache.storm.starter.RollingTopWords
+ * # Runs in local mode (LocalCluster), with topology name
+ * "slidingWindowCounts" $ storm jar
storm-starter-jar-with-dependencies.jar
--- End diff --
done!
---
If your project is set up for it, you can reply to this
[GitHub] storm pull request #1907: STORM-2327 ConfigurableTopology
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/1907 STORM-2327 ConfigurableTopology Squashed #1897 into a single commit. @HeartSaVioR You can merge this pull request into a Git repository by running: $ git pull https://github.com/jnioche/storm STORM-2327-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1907.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1907 commit 8685807b9a9c77dad3ecb8507e985906bce974b5 Author: Julien Nioche Date: 2017-01-26T14:44:01Z STORM-2327 ConfigurableTopology --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1897: STORM-2327 ConfigurableTopology
Github user jnioche closed the pull request at: https://github.com/apache/storm/pull/1897 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1897: STORM-2327 ConfigurableTopology
Github user jnioche commented on the issue: https://github.com/apache/storm/pull/1897 I made a mess with this branch and have pushed a clean, squashed PR as a replacement -> #1907 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1907: STORM-2327 ConfigurableTopology
Github user jnioche commented on the issue: https://github.com/apache/storm/pull/1907 Thanks @HeartSaVioR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1900: Port STORM-2326 Upgrade log4j and slf4j to branch 1.0.x
Github user jnioche commented on the issue: https://github.com/apache/storm/pull/1900 FYI this is fixed in the following version of log4j, see https://blogs.apache.org/logging/entry/log4j-2-8-released. I will test with 2.8 and open a new PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1946: STORM-2326 Upgrade log4j and slf4j for branch 1.0....
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/1946 STORM-2326 Upgrade log4j and slf4j for branch 1.0.x Compiled and packaged with ``` mvn clean install -DskipTests=true cd storm-dist/binary && mvn package ``` Works fine in both local and deployed mode. You can merge this pull request into a Git repository by running: $ git pull https://github.com/DigitalPebble/storm STORM-2326 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1946.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1946 commit 7159a1930f0b46d04e777a195f2239efe99c00ec Author: Julien Nioche Date: 2017-02-20T17:22:55Z STORM-2326 Upgrade log4j and slf4j --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1947: STORM-2326 Upgrade log4j and slf4j for branch 1.x
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/1947 STORM-2326 Upgrade log4j and slf4j for branch 1.x See #1946 for 1.0.x equivalent You can merge this pull request into a Git repository by running: $ git pull https://github.com/DigitalPebble/storm STORM-2326-1.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1947.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1947 commit 593d523f874b70ceddcf67fe5dd4fa9af6c8436b Author: Julien Nioche Date: 2017-02-20T17:32:06Z STORM-2326 Upgrade log4j and slf4j --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1969: STORM-2326 Upgrade log4j and slf4j for branch 2.x
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/1969 STORM-2326 Upgrade log4j and slf4j for branch 2.x See #1946 for 1.0.x equivalent ` mvn clean install -DskipTests=true` ran without problems You can merge this pull request into a Git repository by running: $ git pull https://github.com/DigitalPebble/storm STORM-2326-2.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1969.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1969 commit 71195780e872b45f592c5d30b87626852bd71d3e Author: Julien Nioche Date: 2017-02-27T11:31:47Z STORM-2326 Upgrade log4j and slf4j --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2258: addition of isDebugEnabled() check to improve the perform...
Github user jnioche commented on the issue: https://github.com/apache/storm/pull/2258 -1 Why not simply use parameterized messages instead? See https://www.slf4j.org/faq.html#logging_performance --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2607: STORM-3011 Use default bin path in flight.bash if ...
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/2607 STORM-3011 Use default bin path in flight.bash if $JAVA_HOME is undefined You can merge this pull request into a Git repository by running: $ git pull https://github.com/jnioche/storm 1.x-branch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2607.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2607 commit f2535dfb6ecbb1416c9527c0ce17a5e7818e5198 Author: Julien Nioche Date: 2018-03-27T13:42:28Z STORM-3011 Use default bin path in flight.bash if /usr/lib/jvm/java-8-oracle is undefined ---
[GitHub] storm pull request #2609: STORM-3011 Use default bin path in flight.bash if ...
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/2609 STORM-3011 Use default bin path in flight.bash if JAVA_HOME is undefined port of #2607 to master branch You can merge this pull request into a Git repository by running: $ git pull https://github.com/jnioche/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2609.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2609 commit 15a5757a6109f0840861a5004763b8611864b26c Author: Julien Nioche Date: 2018-03-28T07:54:42Z STORM-3011 Use default bin path in flight.bash if JAVA_HOME is undefined ---
[GitHub] storm issue #2607: STORM-3011 Use default bin path in flight.bash if $JAVA_H...
Github user jnioche commented on the issue: https://github.com/apache/storm/pull/2607 thanks, see #2609 ---
[GitHub] storm pull request #2607: STORM-3011 Use default bin path in flight.bash if ...
Github user jnioche closed the pull request at: https://github.com/apache/storm/pull/2607 ---
[GitHub] storm issue #2607: STORM-3011 Use default bin path in flight.bash if $JAVA_H...
Github user jnioche commented on the issue: https://github.com/apache/storm/pull/2607 thanks @revans2 ---
[GitHub] storm pull request #2897: bugfix ConfigurableTopology completely overwrites ...
GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/2897 bugfix ConfigurableTopology completely overwrites preexisting config The ConfigurableTopology can take configuration files where the values are put under a single config element to be compatible with Flux. This PR fixes a bug when multiple configs are set, the last one will overwrite any preexisting value as everything is put under the same key 'config' You can merge this pull request into a Git repository by running: $ git pull https://github.com/jnioche/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2897.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2897 commit a88e4491c39e7dded5a16974ed2fcacc33892ae7 Author: Julien Nioche Date: 2018-10-26T15:41:03Z bugfix ConfigurableTopology completely overwrites preexisting config ---
[GitHub] storm issue #2908: STORM-3276: Updated Flux to deal with storm local correct...
Github user jnioche commented on the issue: https://github.com/apache/storm/pull/2908 Tested on a StormCrawler topology, works fine in local and remote mode. one quick question though: shouldn't --sleep be deprecated? --local-ttl is used for all the local jobs instead ---
[GitHub] storm pull request #2908: STORM-3276: Updated Flux to deal with storm local ...
Github user jnioche commented on a diff in the pull request:
https://github.com/apache/storm/pull/2908#discussion_r237004334
--- Diff: flux/flux-core/src/main/java/org/apache/storm/flux/Flux.java ---
@@ -52,17 +52,22 @@
public class Flux {
private static final Logger LOG = LoggerFactory.getLogger(Flux.class);
+@Deprecated
--- End diff --
IMHO this it is acceptable for a major release. Projects need a drastic
cleanup once in a while.
---
[GitHub] storm pull request #2908: STORM-3276: Updated Flux to deal with storm local ...
Github user jnioche commented on a diff in the pull request:
https://github.com/apache/storm/pull/2908#discussion_r237006771
--- Diff: flux/flux-core/src/main/java/org/apache/storm/flux/Flux.java ---
@@ -52,17 +52,22 @@
public class Flux {
private static final Logger LOG = LoggerFactory.getLogger(Flux.class);
+@Deprecated
--- End diff --
Got you! Your comment was pretty clear, I just need another coffee :-)
---
