[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user asfgit closed the pull request at: https://github.com/apache/nutch/pull/138 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r76333743
--- Diff: src/java/org/apache/nutch/fetcher/FetcherThreadPublisher.java ---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nutch.fetcher;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.nutch.publisher.NutchPublishers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class handles the publishing of the events to the queue
implementation.
+ *
+ */
+public class FetcherThreadPublisher {
+
+ private static NutchPublishers publisher;
+ private static final Logger LOG =
LoggerFactory.getLogger(FetcherThreadPublisher.class);
+
+ /**
+ * Configure all registered publishers
+ * @param conf
--- End diff --
Look at the Javadoc specification for JDK8.
You cannot simply enter @param $paramName, you need to add a description of
the $pqramName after.
There are plenty of examples peppered throughout the code base already.
Good work Sujen.
On Thursday, August 25, 2016, Sujen Shah wrote:
> In src/java/org/apache/nutch/fetcher/FetcherThreadPublisher.java
> :
>
> > +import org.apache.nutch.publisher.NutchPublishers;
> > +import org.slf4j.Logger;
> > +import org.slf4j.LoggerFactory;
> > +
> > +/**
> > + * This class handles the publishing of the events to the queue
implementation.
> > + *
> > + */
> > +public class FetcherThreadPublisher {
> > +
> > + private static NutchPublishers publisher;
> > + private static final Logger LOG =
LoggerFactory.getLogger(FetcherThreadPublisher.class);
> > +
> > + /**
> > + * Configure all registered publishers
> > + * @param conf
>
> @lewismc , I addressed the other comments in
> the PR, what are you trying to tell me here, I did not understand.
> Thanks.
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
>
,
> or mute the thread
>
> .
>
--
*Lewis*
Dr. Lewis J. McGibbney Ph.D, B.Sc
Director
Phone: +1(626)498-3090
Skype: lewis.john.mcgibbney
Email: lewis.mcgibb...@gmail.com
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user sujen1412 commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r76332711
--- Diff: src/java/org/apache/nutch/fetcher/FetcherThreadPublisher.java ---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nutch.fetcher;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.nutch.publisher.NutchPublishers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class handles the publishing of the events to the queue
implementation.
+ *
+ */
+public class FetcherThreadPublisher {
+
+ private static NutchPublishers publisher;
+ private static final Logger LOG =
LoggerFactory.getLogger(FetcherThreadPublisher.class);
+
+ /**
+ * Configure all registered publishers
+ * @param conf
--- End diff --
@lewismc, I addressed the other comments in the PR, what are you trying to
tell me here, I did not understand.
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r75551907
--- Diff:
src/plugin/publish-rabbitmq/src/java/org/apache/nutch/publisher/rabbitmq/RabbitMQPublisherImpl.java
---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nutch.publisher.rabbitmq;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.nutch.publisher.NutchPublisher;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+
+public class RabbitMQPublisherImpl implements NutchPublisher{
+
+ private static String EXCHANGE_SERVER;
+ private static String EXCHANGE_TYPE;
+ private static String HOST;
+ private static final Logger LOG =
LoggerFactory.getLogger(RabbitMQPublisherImpl.class);
+ private static Channel channel;
+
+ @Override
+ public boolean setConfig(Configuration conf) {
+// TODO Auto-generated method stub
+try{
+EXCHANGE_SERVER = conf.get("rabbitmq.exchange.server", "fetcher_log");
+EXCHANGE_TYPE = conf.get("rabbitmq.exchange.type", "fanout");
+HOST = conf.get("rabbitmq.host", "localhost");
+ConnectionFactory factory = new ConnectionFactory();
+factory.setHost(HOST);
+
+ Connection connection = factory.newConnection();
+ channel = connection.createChannel();
+ channel.exchangeDeclare(EXCHANGE_SERVER, EXCHANGE_TYPE);
+ LOG.info("Configured RabbitMQ publisher");
+ return true;
+}catch(Exception e) {
+ LOG.error("Could not initialize RabbitMQ publisher - {}",
StringUtils.stringifyException(e));
+ return false;
+}
+
+ }
+
+ @Override
+ public void publish(Object event, Configuration conf) {
+// TODO Auto-generated method stub
+String rountingKey = conf.get("rabbitmq.queue.routingkey", "");
+try {
+ channel.basicPublish(EXCHANGE_SERVER, rountingKey, null,
getJSONString(event).getBytes());
+} catch (Exception e) {
+ // TODO Auto-generated catch block
+ LOG.error("Error occured while publishing - {}",
StringUtils.stringifyException(e));
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request: https://github.com/apache/nutch/pull/138#discussion_r75551855 --- Diff: src/plugin/publish-rabbitmq/ivy.xml --- @@ -0,0 +1,42 @@ + + + + + + + +http://nutch.apache.org"/> + +Apache Nutch + + + + + + + + + + + + + + --- End diff -- Please upgrade to 3.6.5 if possible and not API breaking. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r75551918
--- Diff:
src/plugin/publish-rabbitmq/src/java/org/apache/nutch/publisher/rabbitmq/RabbitMQPublisherImpl.java
---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nutch.publisher.rabbitmq;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.nutch.publisher.NutchPublisher;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+
+public class RabbitMQPublisherImpl implements NutchPublisher{
+
+ private static String EXCHANGE_SERVER;
+ private static String EXCHANGE_TYPE;
+ private static String HOST;
+ private static final Logger LOG =
LoggerFactory.getLogger(RabbitMQPublisherImpl.class);
+ private static Channel channel;
+
+ @Override
+ public boolean setConfig(Configuration conf) {
+// TODO Auto-generated method stub
+try{
+EXCHANGE_SERVER = conf.get("rabbitmq.exchange.server", "fetcher_log");
+EXCHANGE_TYPE = conf.get("rabbitmq.exchange.type", "fanout");
+HOST = conf.get("rabbitmq.host", "localhost");
+ConnectionFactory factory = new ConnectionFactory();
+factory.setHost(HOST);
+
+ Connection connection = factory.newConnection();
+ channel = connection.createChannel();
+ channel.exchangeDeclare(EXCHANGE_SERVER, EXCHANGE_TYPE);
+ LOG.info("Configured RabbitMQ publisher");
+ return true;
+}catch(Exception e) {
+ LOG.error("Could not initialize RabbitMQ publisher - {}",
StringUtils.stringifyException(e));
+ return false;
+}
+
+ }
+
+ @Override
+ public void publish(Object event, Configuration conf) {
+// TODO Auto-generated method stub
+String rountingKey = conf.get("rabbitmq.queue.routingkey", "");
+try {
+ channel.basicPublish(EXCHANGE_SERVER, rountingKey, null,
getJSONString(event).getBytes());
+} catch (Exception e) {
+ // TODO Auto-generated catch block
+ LOG.error("Error occured while publishing - {}",
StringUtils.stringifyException(e));
+}
+ }
+
+ private String getJSONString(Object obj) {
+ObjectMapper mapper = new ObjectMapper();
+try {
+ return mapper.writeValueAsString(obj);
+} catch (JsonProcessingException e) {
+ // TODO Auto-generated catch block
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r75551899
--- Diff:
src/plugin/publish-rabbitmq/src/java/org/apache/nutch/publisher/rabbitmq/RabbitMQPublisherImpl.java
---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nutch.publisher.rabbitmq;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.nutch.publisher.NutchPublisher;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+
+public class RabbitMQPublisherImpl implements NutchPublisher{
+
+ private static String EXCHANGE_SERVER;
+ private static String EXCHANGE_TYPE;
+ private static String HOST;
+ private static final Logger LOG =
LoggerFactory.getLogger(RabbitMQPublisherImpl.class);
+ private static Channel channel;
+
+ @Override
+ public boolean setConfig(Configuration conf) {
+// TODO Auto-generated method stub
+try{
+EXCHANGE_SERVER = conf.get("rabbitmq.exchange.server", "fetcher_log");
+EXCHANGE_TYPE = conf.get("rabbitmq.exchange.type", "fanout");
+HOST = conf.get("rabbitmq.host", "localhost");
+ConnectionFactory factory = new ConnectionFactory();
+factory.setHost(HOST);
+
+ Connection connection = factory.newConnection();
+ channel = connection.createChannel();
+ channel.exchangeDeclare(EXCHANGE_SERVER, EXCHANGE_TYPE);
+ LOG.info("Configured RabbitMQ publisher");
+ return true;
+}catch(Exception e) {
+ LOG.error("Could not initialize RabbitMQ publisher - {}",
StringUtils.stringifyException(e));
+ return false;
+}
+
+ }
+
+ @Override
+ public void publish(Object event, Configuration conf) {
+// TODO Auto-generated method stub
--- End diff --
Remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r75551731
--- Diff: src/java/org/apache/nutch/publisher/NutchPublishers.java ---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nutch.publisher;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.nutch.plugin.PluginRepository;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class NutchPublishers extends Configured implements NutchPublisher{
+
+ private static final Logger LOG =
LoggerFactory.getLogger(NutchPublishers.class);
+ private NutchPublisher[] publishers;
+
+ public NutchPublishers(Configuration conf) {
+// TODO Auto-generated constructor stub
+this.publishers = (NutchPublisher[])PluginRepository.get(conf).
+getOrderedPlugins(NutchPublisher.class,
+NutchPublisher.X_POINT_ID, "publisher.order");
+ }
+
+ @Override
+ public boolean setConfig(Configuration conf) {
+// TODO Auto-generated method stub
+boolean success = false;
+try {
+ for(int i=0; i
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r75551738
--- Diff: src/java/org/apache/nutch/publisher/NutchPublishers.java ---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nutch.publisher;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.nutch.plugin.PluginRepository;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class NutchPublishers extends Configured implements NutchPublisher{
+
+ private static final Logger LOG =
LoggerFactory.getLogger(NutchPublishers.class);
+ private NutchPublisher[] publishers;
+
+ public NutchPublishers(Configuration conf) {
+// TODO Auto-generated constructor stub
+this.publishers = (NutchPublisher[])PluginRepository.get(conf).
+getOrderedPlugins(NutchPublisher.class,
+NutchPublisher.X_POINT_ID, "publisher.order");
+ }
+
+ @Override
+ public boolean setConfig(Configuration conf) {
+// TODO Auto-generated method stub
+boolean success = false;
+try {
+ for(int i=0; i
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r75551658
--- Diff: src/java/org/apache/nutch/publisher/NutchPublishers.java ---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nutch.publisher;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.nutch.plugin.PluginRepository;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class NutchPublishers extends Configured implements NutchPublisher{
+
+ private static final Logger LOG =
LoggerFactory.getLogger(NutchPublishers.class);
+ private NutchPublisher[] publishers;
+
+ public NutchPublishers(Configuration conf) {
+// TODO Auto-generated constructor stub
+this.publishers = (NutchPublisher[])PluginRepository.get(conf).
+getOrderedPlugins(NutchPublisher.class,
+NutchPublisher.X_POINT_ID, "publisher.order");
+ }
+
+ @Override
+ public boolean setConfig(Configuration conf) {
+// TODO Auto-generated method stub
+boolean success = false;
+try {
+ for(int i=0; i
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r75551719
--- Diff: src/java/org/apache/nutch/publisher/NutchPublishers.java ---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nutch.publisher;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.nutch.plugin.PluginRepository;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class NutchPublishers extends Configured implements NutchPublisher{
+
+ private static final Logger LOG =
LoggerFactory.getLogger(NutchPublishers.class);
+ private NutchPublisher[] publishers;
+
+ public NutchPublishers(Configuration conf) {
+// TODO Auto-generated constructor stub
+this.publishers = (NutchPublisher[])PluginRepository.get(conf).
+getOrderedPlugins(NutchPublisher.class,
+NutchPublisher.X_POINT_ID, "publisher.order");
+ }
+
+ @Override
+ public boolean setConfig(Configuration conf) {
+// TODO Auto-generated method stub
+boolean success = false;
+try {
+ for(int i=0; i
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r75551687
--- Diff: src/java/org/apache/nutch/publisher/NutchPublishers.java ---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nutch.publisher;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.nutch.plugin.PluginRepository;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class NutchPublishers extends Configured implements NutchPublisher{
+
+ private static final Logger LOG =
LoggerFactory.getLogger(NutchPublishers.class);
+ private NutchPublisher[] publishers;
+
+ public NutchPublishers(Configuration conf) {
+// TODO Auto-generated constructor stub
+this.publishers = (NutchPublisher[])PluginRepository.get(conf).
+getOrderedPlugins(NutchPublisher.class,
+NutchPublisher.X_POINT_ID, "publisher.order");
+ }
+
+ @Override
+ public boolean setConfig(Configuration conf) {
+// TODO Auto-generated method stub
+boolean success = false;
+try {
+ for(int i=0; i
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r75551671
--- Diff: src/java/org/apache/nutch/publisher/NutchPublishers.java ---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nutch.publisher;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.nutch.plugin.PluginRepository;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class NutchPublishers extends Configured implements NutchPublisher{
+
+ private static final Logger LOG =
LoggerFactory.getLogger(NutchPublishers.class);
+ private NutchPublisher[] publishers;
+
+ public NutchPublishers(Configuration conf) {
+// TODO Auto-generated constructor stub
+this.publishers = (NutchPublisher[])PluginRepository.get(conf).
+getOrderedPlugins(NutchPublisher.class,
+NutchPublisher.X_POINT_ID, "publisher.order");
+ }
+
+ @Override
+ public boolean setConfig(Configuration conf) {
+// TODO Auto-generated method stub
--- End diff --
Remove this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request: https://github.com/apache/nutch/pull/138#discussion_r75551621 --- Diff: src/java/org/apache/nutch/metadata/Nutch.java --- @@ -95,4 +95,11 @@ public static final String ARG_SEGMENTDIR = "segment_dir"; /** Argument key to specify the location of individual segment for the REST endpoints **/ public static final String ARG_SEGMENT = "segment"; + + /** Metadata used for the FetcherThreadEvent for publishing to subscribers*/ + public static final String FETCH_EVENT_TITLE = "title"; + public static final String FETCH_EVENT_CONTENTTYPE = "content-type"; + public static final String FETCH_EVENT_SCORE = "score"; --- End diff -- Javadoc for all of these. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r75551602
--- Diff: src/java/org/apache/nutch/fetcher/FetcherThreadPublisher.java ---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nutch.fetcher;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.nutch.publisher.NutchPublishers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class handles the publishing of the events to the queue
implementation.
+ *
+ */
+public class FetcherThreadPublisher {
+
+ private static NutchPublishers publisher;
+ private static final Logger LOG =
LoggerFactory.getLogger(FetcherThreadPublisher.class);
+
+ /**
+ * Configure all registered publishers
+ * @param conf
--- End diff --
Please complete Javadoc. If and when we upgrade to JDK 1.8 this will fail
the Javadoc generation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r75551553
--- Diff: src/java/org/apache/nutch/fetcher/FetcherThreadEvent.java ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nutch.fetcher;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.nutch.parse.Outlink;
+
+/**
+ * This class is used to capture the various events occurring
+ * at fetch time. These events are sent to a queue implementing the
publisher
+ *
+ */
+@SuppressWarnings("serial")
+public class FetcherThreadEvent implements Serializable{
+
+ public static enum PublishEventType {START, END, REPORT}
+
+
+ private PublishEventType eventType;
+ private Map eventData;
+ private String url;
+ private Long timestamp;
+
+ public FetcherThreadEvent(PublishEventType eventType, String url) {
+this.eventType = eventType;
+this.url = url;
+this.timestamp = System.currentTimeMillis();
+ }
+
+ public PublishEventType getEventType() {
+return eventType;
+ }
+ public void setEventType(PublishEventType eventType) {
+this.eventType = eventType;
+ }
+ public Map getEventData() {
+return eventData;
+ }
+ public void setEventData(Map eventData) {
+this.eventData = eventData;
+ }
+ public String getUrl() {
+return url;
+ }
+ public void setUrl(String url) {
--- End diff --
Need Javadoc for all of these.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r75551531
--- Diff: src/java/org/apache/nutch/fetcher/FetcherThreadEvent.java ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nutch.fetcher;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.nutch.parse.Outlink;
+
+/**
+ * This class is used to capture the various events occurring
+ * at fetch time. These events are sent to a queue implementing the
publisher
+ *
+ */
+@SuppressWarnings("serial")
+public class FetcherThreadEvent implements Serializable{
+
+ public static enum PublishEventType {START, END, REPORT}
--- End diff --
Javadoc throughout.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/138#discussion_r75551512
--- Diff: src/java/org/apache/nutch/fetcher/FetcherThreadEvent.java ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nutch.fetcher;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.nutch.parse.Outlink;
+
+/**
+ * This class is used to capture the various events occurring
+ * at fetch time. These events are sent to a queue implementing the
publisher
+ *
+ */
+@SuppressWarnings("serial")
--- End diff --
What warning are you suppressing/ Do you have examples @sujen1412
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
Github user lewismc commented on a diff in the pull request: https://github.com/apache/nutch/pull/138#discussion_r75551452 --- Diff: ivy/ivy.xml --- @@ -127,6 +127,9 @@ + + + --- End diff -- Indentation is off. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nutch pull request #138: Fix for NUTCH-2132: Publisher/Subscriber model for ...
GitHub user sujen1412 opened a pull request: https://github.com/apache/nutch/pull/138 Fix for NUTCH-2132: Publisher/Subscriber model for Nutch to emit events This PR is still in progress and needs a review to get the plugin system working. It is not ready to commit as of yet. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sujen1412/nutch NUTCH-2132 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nutch/pull/138.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #138 commit 5a13301a7808d852b88d8724c3c3b7783fa9d2be Author: Sujen Shah Date: 2015-10-02T08:03:35Z Added dependency for RabbitMQ commit 3029ef4055834b72d63e7fc516eca448c2efe32a Author: Sujen Shah Date: 2015-10-02T08:04:39Z Code for FetcherThreadPublisher commit ebfd7728e650cc7648a3939d3985826eefde93f3 Author: Sujen Shah Date: 2015-10-02T08:05:11Z Added property descriptions in nutch-default.xml commit 445fcc2d766ddef7cc36783ebcaecb552e4f2819 Author: Sujen Shah Date: 2015-10-02T08:28:52Z Added support for queue routing key commit 44498308634dda99f543d5e18724ad8cfeb16343 Author: Sujen Shah Date: 2015-10-19T08:44:29Z Added properties to make publisher optional in nutch-default.xml commit ad88c94fc274576aacaa2c17b1f55a087f7a04f9 Author: Sujen Shah Date: 2015-10-27T02:40:45Z Added routingkey support commit e380de803c8c129f0dfb7d8c31a8596b4ceae8bf Author: Sujen Shah Date: 2015-10-29T20:12:55Z Better exception handling when RMQ server is down commit e4f5e13cc5675f9e7d37ebda39bf230c08baf4b8 Author: Sujen Shah Date: 2016-08-02T15:01:48Z Created plugin system for pub/sub implementation in Nutch commit 2c484ec4789c84f7bf9e592e15c96cf788ef5967 Author: Sujen Shah Date: 2016-08-02T15:12:28Z Removed Rabbitmq dependency from ivy.xml and remove author tags --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
