Re: Exclude Storm jars from an assembly

[Ramin Farajollah (BLOOMBERG/ 731 LEX)]

Thanks for the reply. I was able to exclude Storm and log4j related jars.

Here's a related problem:

Since Storm will now use its log4j jars in its path/jar, you cannot control how 
log4j is used.
Specifically, I'm not able to use log4j2 and my own log4j2.xml to configure 
logging.
(I'm able to do that, if I start the topology from a main function.)

So, the question is: 
- How do I have Storm use log4j2 and apply my log4j2.xml?


BTW, "exclude" as was suggested worked for storm-core, but not for log4j 
related jars.
Instead, I just changed the scope of the dependencies to "provided".


Thank you,
Ramin Farajollah
212 617-0014

From: user@storm.apache.org 
Subject: Re: Exclude Storm jars from an assembly

Hello Ramin,

You can use maven-shade-plugin for creating uber-JAR as well. For example, we 
use following configuration:


   org.apache.maven.plugins
   maven-shade-plugin
   
  
 package
 
shade
 
 
  false
 
  
   
   
  
 
org.apache.storm:storm-core
 
  
   


​Cheers! 

Stefan

On Fri, Mar 24, 2017 at 3:03 PM, Ramin Farajollah (BLOOMBERG/ 731 LEX) 
 wrote:

Hi,

The following link describes how to create a Maven assembly and that the Storm 
jars should be excluded:
http://storm.apache.org/releases/1.0.3/Running-topologies-on-a-production-cluster.html

I have been able to generate both a jar of my own classes and another with 
dependencies. Neither are acceptable to "storm jar" command.

*** The question: How to exclude Storm jars from jar-with-dependencies?

On the web there are plenty examples of using an assembly xml but I did not 
find one using a plugin as suggested in the above link.
Would you paste the portion of a pom.xml that includes both the plugin for 
maven-assembly-plugin artifact and the exclusions?

Thank you!

Thank you,
Ramin Farajollah
212 617-0014


-- 
Mgr. Stefan Rakonjac
(+420) 774 692 671stefanrakon...@gmail.com

[storm-kafka] where is stored Kafka Spout consummer's offset?

[Alexandre Vermeerbergen]

Hello,

I need to programmatically monitor the offset of my Kafka Spouts (based on
storm-kafka, not storm-kafka-client).

I already have a probe checking the offsets of all consumers of my Kafka
Brokers cluster. It takes into account both Zookeeper-based old consumers,
but also newer ones managing their offsets directly in Kafka brokers.

But my probe do not "see" any of the consumers corresponding to Kafka
spouts.

So my question is: where does Kafka spouts (from storm-kafka) store their
offsets as Kafka consumers?

Tested configurations:
* Storm 1.0.1, Storm 1.0.3 and Storm 1.1.0 rc3,
* Kafka Brokers : 0.1.10.1


Best regards,
Alexandre Vermeerbergen

Re: [storm-kafka] where is stored Kafka Spout consummer's offset?

[Harsh Choudhary]

The storm stores its offset in the Zookeeper, it is connected to. So, you
won't find the offset information for the storm clients in the same place
where other clients' offsets are. You must have provided topic id in Spout
Config in Storm, this is where it stores the offset information in its
Zookeeper.

*Cheers!*

Harsh Choudhary / Software Engineer

Blog / express.harshti.me

[image: Facebook]  [image: Twitter]
 [image: Google Plus]

 [image: Linkedin]
 [image: Instagram]

[image: 500px]
 [image: github]


On Tue, Mar 28, 2017 at 11:27 AM, Alexandre Vermeerbergen <
avermeerber...@gmail.com> wrote:

> Hello,
>
> I need to programmatically monitor the offset of my Kafka Spouts (based on
> storm-kafka, not storm-kafka-client).
>
> I already have a probe checking the offsets of all consumers of my Kafka
> Brokers cluster. It takes into account both Zookeeper-based old consumers,
> but also newer ones managing their offsets directly in Kafka brokers.
>
> But my probe do not "see" any of the consumers corresponding to Kafka
> spouts.
>
> So my question is: where does Kafka spouts (from storm-kafka) store their
> offsets as Kafka consumers?
>
> Tested configurations:
> * Storm 1.0.1, Storm 1.0.3 and Storm 1.1.0 rc3,
> * Kafka Brokers : 0.1.10.1
>
>
> Best regards,
> Alexandre Vermeerbergen
>
>