Re: build error

cute a > command like the following: > gradle clients:build > > Thanks, > Grant > > On Mon, Nov 16, 2015 at 6:33 PM, Guozhang Wang > wrote: > > > Siyuan, > > > > Which command did you use to build? > > > > Guozhang > > > > On Mon

Questions about new consumer API

The new consumer API looks good. If I understand it correctly you can use it like simple consumer or high-level consumer. But I have couple questions about it's internal implementation First of all does the consumer have any internal fetcher threads like high-level consumer? When you assign multi

build error

I got a build error on both trunk and 0.9.0 branch > docs/producer_config.html (No such file or directory) Do I miss anything before build Thanks, Siyuan

Re: registry / export question

AM, Gour Saha wrote: > It is REST style uri, so if you append the uri path with the export group > name you will get the info you are looking for. > > If that does not answer your question, can you give an example response > that you are expecting to see? > > -Gour > > >

Re: registry / export question

I've noticed the http://hostname/ws/v1/slider/publisher/exports/ only gives you the list of export values, but within each one the entries block are empty. Is it ok have them all embedded in one response so that you can get all

Re: Packaging new apps

Hi Jean, Thanks for the change, using instance tag(is it a new feature in the latest version? I didn't see it in the older slider versions) is a really good idea. it might be good for other's to have a template but not for kafka. Kafka is evolving in quite fast pace. I've seen many property key/v

Re: Component instance level configuration

nt will have their own > configuation? Only port can be different for instance of a component > nowadays. > > 2015-01-28 5:57 GMT+08:00 hsy...@gmail.com : > > > Hi, > > > > I'm just wondering if there is a way to support component instance level > > c

Component instance level configuration

Hi, I'm just wondering if there is a way to support component instance level configuration. For example, in kafka-on-yarn, kafka broker is a component, but each broker instance needs a unique id. It would be better to expose the instance id, so that different instance would override some of the

What configuration is mandatory in slider package?

Hi guys, Slider requires several configurations appConfig, metainfo, resources. What are necessary to be included in package what is only needed when launch the slider application. If I change appConfig do I have to update the package to make it run? Thanks!

Given brokers, is it able to know all the zookeepers that brokers connect to

Hi Guys, If I know the brokers. Is there a way to know the zookeeper host from broker list? Thanks! Siyuan

Re: Is it able to specify the percentage of memory instead of # of MB

rom YARN and YARN only accepts numbers. > > > > > > > > % of total on each node also becomes an issue when nodes can have > > varying > > > > value of total memory available. This is common for heterogeneous > > > clusters > > > >

Re: Questions about new consumer API

> Yes, it will. > > 4. Will simple consumer api and high-level consumer api still be supported? > > Over time, we will phase out the current high-level and simple consumer > since the > 0.9 API supports both. > > Thanks, > Neha > > On Tue, Dec 2, 2014 at 12:07 PM,

Questions about new consumer API

Hi guys, I'm interested in the new Consumer API. http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/ I have couple of question. 1. In this doc it says kafka consumer will automatically do load balance. Is it based on throughtput or same as what we have now balance the cardinali

Is it able to specify the percentage of memory instead of # of MB

Hi guys, For each component instance, Is there a way to specify the % of total memory of each node instead of # of MB? Best, Siyuan

Is there a plan to build a ubiquitous web service API to manage the kafka cluster

Hi guys, Nowadays, all kafka administration work (add, tear down node, topic management, throughput monitor) are done by various different tool talk to brokers, zookeeper etc. Is there a plan for core team to build a central universal server providing webservice API to do all the admin work? Best

I can't specify user by calling Execute method

Hi I specified user when call Execute(..., user='hdp',...) But the container processes are all launched as *yarn* user

Re: Application configuration page is outdated?

and resources.json should be > enough. > > -Sumit > > On Tue, Nov 18, 2014 at 2:00 PM, Sumit Mohanty > wrote: > > > Is it possible for you to share the application package? I can browse > > through it to see what needs to change. > > > > On Tue, Nov 18, 2014 at 1

Application configuration page is outdated?

Hi guys, I just switch from 40 to 60 and I found the application configuration doesn't work any more. http://slider.incubator.apache.org/docs/slider_specs/application_instance_configuration.html For example : { "schema" : "http://example.org/specification/v2.0.0";, "metadata" : { }, "glo

Re: slider stop bug?

I'm using 0.60 On Tue, Nov 18, 2014 at 12:26 PM, Sumit Mohanty wrote: > You can use --force. > > Are you using the RC for 0.60 or 0.40? > > On Tue, Nov 18, 2014 at 12:00 PM, hsy...@gmail.com > wrote: > > > If the application is stuck in ACCEPT status, slider s

slider stop bug?

If the application is stuck in ACCEPT status, slider stop command doesn't stop the yarn application. I didn't setup the scheduler address correctly so the application stuck at ACCEPT status and I couldn't use slider stop to stop the application Best, Siyuan

How to try 1.7?

Hi guys, Is there an easy way to try 1.7, probably with HDP2.2? Thanks!

HDP2.2?

Hi guys, Maybe it is not the right place to ask. If I want to switch to 0.60, it depends on hadoop 2.6 which is not generally available right now. Is there a easy way to setup a virtual cluster with hadoop2.6? I saw HDP2.2 preview is available with sandbox only. Best, Siyuan

Re: Do I have to repackage if I change appConfig.json/resources.json

"thaw" in 0.40) the application then it will use the hostnames that it remembered. What if I use *yarn application -kill* to stop the application? On Fri, Nov 14, 2014 at 5:09 PM, hsy...@gmail.com wrote: > The only problem is I have to upgrade to hadoop 2.6? > > On Fri, Nov 1

Re: Do I have to repackage if I change appConfig.json/resources.json

t;>> add more instances for example > > Flex command can be used while the application is running. > > >>> want the application recover from last snapshot > If you stop and then start (probably it is "freeze" and "thaw" in 0.40) the > application then

Question about behavior after container failures

Hi all, In the document, I got that slider will try to recover on container failure. But in my test application it doesn't I'm using the 0.40 release. build from source Here is what I get. *If I kill the child process * The agent check_process_status method will raise ComponentIsNotRunning and

Re: Do I have to repackage if I change appConfig.json/resources.json

at there are only one file to download for the users.* > > Which branch are you using? (for me to answer the next questions) > > On Fri, Nov 14, 2014 at 3:47 PM, hsy...@gmail.com > wrote: > > > I'm getting confused. After you edit your appConfig.json and > > re

How to change log level for Slider AM?

How to change log level for Slider AM? Thanks!

Re: log4j dir?

g insists > on using 'kafka/logs', so I create a soft link from there to the desired > destination directory: > > # kafka scripts hard-code the logs dir, so point that path to where we want > the logs to be. > ln -s $STREAM_BUFFER_LOGS_DIR kafka/logs > > -Ben > >

Re: Do I have to repackage if I change appConfig.json/resources.json

n package zip file > * Call create with --template appConfig.json --resources > resources.json ... > > -Sumit > > On Fri, Nov 14, 2014 at 2:08 PM, hsy...@gmail.com > wrote: > > > So actually you don't need appConfig.json and resources.json in the ZIP &

Re: Do I have to repackage if I change appConfig.json/resources.json

> instance referencing the new versions of the file from the command line: > > ./slider create —template —resources > > > the app config references the application package in HDFS, which can be > pre-seeded using “slider install-package” > > — Jon > > On Nov

Do I have to repackage if I change appConfig.json/resources.json

Everytime I change appConfig.json and resources.json. Do I have to repackage the zip file and redeploy the file to hdfs? Thanks!

Re: log4j dir?

Anyone has any idea how do I config the log4j file dir? On Thu, Nov 13, 2014 at 4:58 PM, hsy...@gmail.com wrote: > Hi guys, > > Just notice kafka.logs.dir in log4j.properties doesn't take effect > > It's always set to *$base_dir/logs* in kafka-run-class.sh &

log4j dir?

Hi guys, Just notice kafka.logs.dir in log4j.properties doesn't take effect It's always set to *$base_dir/logs* in kafka-run-class.sh LOG_DIR=$base_dir/logs KAFKA_LOG4J_OPTS="-Dkafka.logs.dir=$LOG_DIR $KAFKA_LOG4J_OPTS" Best, Siyuan

Re: Q: How to define component configuration?

ake them available at the container? This seems to > be a good feature to support. > > -Sumit > > On Tue, Nov 11, 2014 at 2:42 PM, hsy...@gmail.com > wrote: > > > Thanks Steve, but I logged all the properties in params.py, I couldn't > > find any key named *myk

Re: Q: How to define component configuration?

rrides the global value, that override > is picked up > > It's essentially a form of prototype-based programming, except only of > properties, not methods: > http://en.wikipedia.org/wiki/Prototype-based_programming > > On 11 November 2014 01:30, hsy...@gmail.com wrote

Re: Q: How to define component configuration?

s": { "COMPONENT1": { *"mykey": "myvalue"* }, "slider-appmaster": { "jvm.heapsize": "256M" } } } Is it able to make the component1 read the value for *mykey* ？ Best, Siyuan On Mon, Nov 10, 2014 at 4:15

Re: Q: How to define component configuration?

If I want to have several instances of some component. But I want to set some of the properties to different value for different instances. How can I do it? Thanks! Best, Siyuan On Mon, Nov 10, 2014 at 1:26 PM, hsy...@gmail.com wrote: > Hi guys, > > Is there an example of

Q: How to define component configuration?

Hi guys, Is there an example of component configuration? Is there a way to give different value to same property for different instances? Siyuan

Re: Maven repository?

lider.git > > > You can checkout develop branch. > > > Cheers > > > On Wed, Nov 5, 2014 at 6:36 PM, hsy...@gmail.com wrote: > > > Is there a public maven repository that I can checkout the slider > library? > > > > Best, > > Siyuan > > >

Maven repository?

Is there a public maven repository that I can checkout the slider library? Best, Siyuan

Re: Is it able to make AM to restart from previous state?

from new node? What if you decrease the instance number? Thanks! Best, Siyuan On Wed, Nov 5, 2014 at 1:44 PM, hsy...@gmail.com wrote: > Thanks Steve, > > Is No 1 a new feature in YARN (Not released yet)? > > And you mentioned slider saves the location in history files. What ar

Re: Is it able to make AM to restart from previous state?

? Thanks On Wed, Nov 5, 2014 at 12:46 PM, Steve Loughran wrote: > On 5 November 2014 20:21, hsy...@gmail.com wrote: > > > Hi guys, > > > > I noticed in the code when a container fails it will try to relaunch from > > the same node. My question is if I restart whole app

Is it able to make AM to restart from previous state?

Hi guys, I noticed in the code when a container fails it will try to relaunch from the same node. My question is if I restart whole application(Ex. AM got killed, or manually restart the app). Does slider try to launch all containers from the nodes where it was running? Thanks! Best, Siyuan

Re: 2 questions

rt an fully functional embedded jetty server in the AM and set the tracking url to this {amhost:port}? Best, Siyuan On Fri, Oct 31, 2014 at 2:59 PM, Steve Loughran wrote: > On 31 October 2014 01:15, hsy...@gmail.com wrote: > > > Hi all, > > > > I have 2 questions, > > 1

2 questions

Hi all, I have 2 questions, 1. Is there a way to extend the REST API(add some customized api)? 2. Is there a way to add/remove component/component instance at the runtime? Thanks! Best, Siyuan

Re: Why command script is written in python?

opment. > > If you do not mind can you open a JIRA at > https://issues.apache.org/jira/browse/SLIDER describing the requirements > of > "debugging" during development time and post development. I will use that > JIRA to drive the issue. > > -Sumit > > On Thu, Oct

Re: Why command script is written in python?

fied to handle various > script types. Of course, today the only supported/tested one is python. > > On Thu, Oct 30, 2014 at 1:55 PM, hsy...@gmail.com > wrote: > > > May I ask why do you choose python to write the command script. And is > > there any other language? I find

How to inject my own web service handler into app master?

Hi, I noticed slider app master has an embedded web service server. Is there a way to inject my own web service call into that server? Another question how to you communication between containers(components) and AM(through agent code)? Is there a way to customize the communication/data between c

Why command script is written in python?

May I ask why do you choose python to write the command script. And is there any other language? I find difficult to debug because each time if I find an issue I have to change file package upload to hdfs, rerun and I'm not very familiar with python Best Siyuan

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Sorry, my classpath should be $HADOOP_HOME/etc/hadoop/, thanks for you guys' help! On Wed, Oct 29, 2014 at 12:06 PM, hsy...@gmail.com wrote: > > > And all I have in slider-err.txt is > > log4j:WARN No appenders could be found for logger > (org.apache.slider.co

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

: hdfs://localhost:9000/user/siyuan/.slider/cluster/cl15, expected: file:/// Looks like there is no log4j log at all. How do I properly setup log4j? Best, Siyuan On Wed, Oct 29, 2014 at 11:47 AM, hsy...@gmail.com wrote: > Hi, > > I installed the apache hadoop with just unzip the tar ba

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

arn classpath variable. > > E.g. > > yarn.application.classpath > */etc/hadoop/conf*,/usr/hdp/current/hadoop-client/* > > On Wed, Oct 29, 2014 at 11:18 AM, hsy...@gmail.com > wrote: > > > @Pushkar > > > > Thanks, but it doesn&#x

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

@Pushkar Thanks, but it doesn't work for me. My slider-client.xml setting is yarn.application.classpath/home/siyuan/hadoop/current/etc/hadoop/*,/home/siyuan/hadoop/current/share/hadoop/tools/lib/*,/home/siyuan/hadoop/current/share/hadoop/mapreduce/lib/*,/home/siyuan/hadoop/current/share/ha

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

Another observation is this error has no stacktrace. I guess it does happen during launching the App master. It happens when app master try to launch container when it tries to localize the resource? Best, Siyuan On Tue, Oct 28, 2014 at 7:02 PM, hsy...@gmail.com wrote: > I try to run kafka

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

I try to run kafka as slider application On Tue, Oct 28, 2014 at 7:01 PM, hsy...@gmail.com wrote: > I had the same problem > > This is my appConfig.json > > { > "schema": "http://example.org/specification/v2.0.0";, > "metadata": { >

Re: Wrong FS: hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: file:/// Issues deploying memcached using slider.

I had the same problem This is my appConfig.json { "schema": "http://example.org/specification/v2.0.0";, "metadata": { }, "global": { "application.def": "hdfs://localhost:9000/user/siyuan/slider_kafka.zip", "java_home": "/usr/lib/jvm/java-7-oracle/", "package_list": "files/kaf

NoClassDefFoundError ? Hadoop classpath

Hi guys, I'm new to slider. I tried to run a java application from slider and get the error. Hadoop classpath has been setup in client.xml Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/client/api/async/AMRMClientAsync$CallbackHandler Do I have to include the

Create topic programmatically

Hi guys, Besides TopicCommand, which I believe is not provided to create topic programmatically, is there any other way to automate creating topic in code? Thanks! Best, Siyuan

Re: Is there a way to run application on certain subset of nodes?

e that the container can consume > most (not just > 50%) of the available memory. > > One anti-affinity is available or Slider adds such a feature itself, you > need not do use the work-around. > > > On Mon, Aug 4, 2014 at 2:04 PM, hsy...@gmail.com wrote: > > > Thanks Sum

Re: Is there a way to run application on certain subset of nodes?

half of available resource (e.g. memory) per host. > > Depending on the timeline of the availability of anti-affinity feature from > YARN, Slider can also add support for such a feature but the desirable path > is to get this through YARN. > > > On Mon, Aug 4, 2014 at 1:48 PM,

Re: Is there a way to run application on certain subset of nodes?

> > > https://issues.apache.org/jira/browse/YARN-796 > > > > > > On Mon, Aug 4, 2014 at 2:37 PM, hsy...@gmail.com > wrote: > > > > > Hi guys, > > > > > > I'm new to slider and try to convert some application into yarn app.

Is there a way to run application on certain subset of nodes?

Hi guys, I'm new to slider and try to convert some application into yarn app. I would like to ask is there a way to specify only a subset of nodes in the cluster to run my app and can slider guarantee every container(of that application) run on different nodes? Thank you very much! Best, Siyuan

Re: Kafka on yarn

gt; solutions. If there is interest I could certainly list the relevant JIRA's > along with some additional JIRA's > > required IMO. > > > > Thanks > > Kam > > > > > > On Wednesday, July 23, 2014 2:37 PM, "hsy...@gmail.com"

Kafka on yarn

Hi guys, Kafka is getting more and more popular and in most cases people run kafka as long-term service in the cluster. Is there a discussion of running kafka on yarn cluster which we can utilize the convenient configuration/resource management and HA. I think there is a big potential and require

Re: How to do an interactive Spark SQL

Anyone has any idea on this? On Tue, Jul 22, 2014 at 7:02 PM, hsy...@gmail.com wrote: > But how do they do the interactive sql in the demo? > https://www.youtube.com/watch?v=dJQ5lV5Tldw > > And if it can work in the local mode. I think it should be able to work in > cluste

Re: How to do an interactive Spark SQL

rent stream after the StreamingContext has started. > > Tobias > > > On Wed, Jul 23, 2014 at 9:55 AM, hsy...@gmail.com > wrote: > >> For example, this is what I tested and work on local mode, what it does >> is it get data and sql query both from kafka and do sql on

Re: How to do an interactive Spark SQL

+ r.mkString(",") + "\n" }) producer.send(new KeyedMessage[String, String](outputTopic, s"SQL: $sqlS \n $result")) }) ssc.start() ssc.awaitTermination() On Tue, Jul 22, 2014 at 5:10 PM, Zongheng Yang wrote: > Can you paste a small code example

Re: How to do an interactive Spark SQL

? What do you mean by "cannot shar the sql to all workers"? > > On Tue, Jul 22, 2014 at 4:03 PM, hsy...@gmail.com > wrote: > > Hi guys, > > > > I'm able to run some Spark SQL example but the sql is static in the > code. I > > would like to know is

How to do an interactive Spark SQL

Hi guys, I'm able to run some Spark SQL example but the sql is static in the code. I would like to know is there a way to read sql from somewhere else (shell for example) I could read sql statement from kafka/zookeeper, but I cannot share the sql to all workers. broadcast seems not working for up

Re: Task not serializable: java.io.NotSerializableException: org.apache.spark.SparkContext

I have the same problem On Sat, Jul 19, 2014 at 12:31 AM, lihu wrote: > Hi, > Everyone. I have a piece of following code. When I run it, > it occurred the error just like below, it seem that the SparkContext is not > serializable, but i do not try to use the SparkContext except the broadca

Re: Difference among batchDuration, windowDuration, slideDuration

Thanks Tathagata, so can I say RDD size(from the stream) is window size. and the overlap between 2 adjacent RDDs are sliding size. But I still don't understand what it batch size, why do we need this since data processing is RDD by RDD right? And does spark chop the data into RDDs at the very beg

Re: Interested in contributing to Kafka?

Is there a scala API doc for the entire kafka library? On Wed, Jul 16, 2014 at 5:34 PM, hsy...@gmail.com wrote: > Hi Jay, > > I would like to take a look at the code base and maybe start working on > some jiras. > > Best, > Siyuan > > > On Wed, Jul 16, 201

Re: Interested in contributing to Kafka?

Hi Jay, I would like to take a look at the code base and maybe start working on some jiras. Best, Siyuan On Wed, Jul 16, 2014 at 3:09 PM, Jay Kreps wrote: > Hey All, > > A number of people have been submitting really nice patches recently. > > If you are interested in contributing and are loo

Difference among batchDuration, windowDuration, slideDuration

When I'm reading the API of spark streaming, I'm confused by the 3 different durations StreamingContext(conf: SparkConf , batchDuration: Duration

Re: SQL + streaming

tors, so you wont find anything in the driver logs! > So try doing a collect, or take on the RDD returned by sql query and print > that. > > TD > > > On Tue, Jul 15, 2014 at 4:28 PM, hsy...@gmail.com > wrote: > >> By the way, have you ever run SQL and stream to

Re: SQL + streaming

By the way, have you ever run SQL and stream together? Do you know any example that works? Thanks! On Tue, Jul 15, 2014 at 4:28 PM, hsy...@gmail.com wrote: > Hi Tathagata, > > I could see the output of count, but no sql results. Run in standalone is > meaningless for me and I ju

Re: SQL + streaming

see output? > Also, I recommend going through the previous step-by-step approach to > narrow down where the problem is. > > TD > > > On Mon, Jul 14, 2014 at 9:15 PM, hsy...@gmail.com > wrote: > >> Actually, I deployed this on yarn cluster(spark-submit) and I could

Re: How to kill running spark yarn application

>> it is terminated as well. Sorry, I cannot reproduce it. >> >> >> On Mon, Jul 14, 2014 at 7:36 PM, hsy...@gmail.com >> wrote: >> >>> Before "yarn application -kill" If you do jps You'll have a list >>> of SparkSubmit and ApplicationM

Re: SQL + streaming

If you can get that to work, then I would test the Spark SQL > stuff. > > TD > > > On Mon, Jul 14, 2014 at 5:25 PM, hsy...@gmail.com > wrote: > >> No errors but no output either... Thanks! >> >> >> On Mon, Jul 14, 2014 at 4:59 PM, Tathagata Das &

Re: SQL + streaming

throwing error? No errors but no output > either? > > TD > > > On Mon, Jul 14, 2014 at 4:06 PM, hsy...@gmail.com > wrote: > >> Hi All, >> >> Couple days ago, I tried to integrate SQL and streaming together. My >> understanding is I can transform RDD from

Re: How to kill running spark yarn application

is is what I did 2 hours > ago. > > Sorry I cannot provide more help. > > > Sent from my iPhone > > On 14 Jul, 2014, at 6:05 pm, "hsy...@gmail.com" wrote: > > yarn-cluster > > > On Mon, Jul 14, 2014 at 2:44 PM, Jerry Lam wrote: > >> Hi Si

SQL + streaming

Hi All, Couple days ago, I tried to integrate SQL and streaming together. My understanding is I can transform RDD from Dstream to schemaRDD and execute SQL on each RDD. But I got no luck Would you guys help me take a look at my code? Thank you very much! object KafkaSpark { def main(args: Arr

Re: How to kill running spark yarn application

yarn-cluster On Mon, Jul 14, 2014 at 2:44 PM, Jerry Lam wrote: > Hi Siyuan, > > I wonder if you --master yarn-cluster or yarn-client? > > Best Regards, > > Jerry > > > On Mon, Jul 14, 2014 at 5:08 PM, hsy...@gmail.com > wrote: > >> Hi all, >

How to kill running spark yarn application

Hi all, A newbie question, I start a spark yarn application through spark-submit How do I kill this app. I can kill the yarn app by "yarn application -kill appid" but the application master is still running. What's the proper way to shutdown the entire app? Best, Siyuan

Difference between SparkSQL and shark

I have a newbie question. What is the difference between SparkSQL and Shark? Best, Siyuan

Re: Some question about SQL and streaming

now if it is the best way, but it works. > > Tobias > > > On Thu, Jul 10, 2014 at 4:21 AM, hsy...@gmail.com > wrote: > >> Hi guys, >> >> I'm a new user to spark. I would like to know is there an example of how >> to user spark SQL and spark streaming together? My use case is I want to do >> some SQL on the input stream from kafka. >> Thanks! >> >> Best, >> Siyuan >> > >

Some question about SQL and streaming

Hi guys, I'm a new user to spark. I would like to know is there an example of how to user spark SQL and spark streaming together? My use case is I want to do some SQL on the input stream from kafka. Thanks! Best, Siyuan

Re: Too Many Open Files Broker Error

I have the same problem. I didn't dig deeper but I saw this happen when I launch kafka in daemon mode. I found the daemon mode is just launch kafka with nohup. Not quite clear why this happen. On Wed, Jul 9, 2014 at 9:59 AM, Lung, Paul wrote: > Yup. In fact, I just ran the test program again wh

Re: Help is processing huge data through Kafka-storm cluster

ue, Jun 17, 2014 at 10:55 AM, Robert Rodgers >> wrote: >> >> > we have been experimenting with Samza which is also worth a look. It's >> > basically a topic-to-topic node on Yarn. >> > >> > >> > >> > On Jun 17, 201

Re: delete topic ?

I'm using 0.8.1.1 I use DeleteTopicCommand to delete topic args[0] = "--topic"; args[1] = the topic you want to delete args[2] = "--zookeeper"; args[3] = kafkaZookeepers; DeleteTopicCommand.main(args); You can write your own script to delete the topic, I guess. And I think it o

Re: Help is processing huge data through Kafka-storm cluster

Hi Shaikh, I heard some throughput bottleneck of storm. It cannot really scale up with kafka. I recommend you to try DataTorrent platform(https://www.datatorrent.com/) The platform itself is not open-source but it has a open-source library ( https://github.com/DataTorrent/Malhar) which contains a

How to get last message

Is there a way to get the last message of a partition for a given topic?

Async producer callback?

Hi guys, So far, is there a way to track the asyn producer callback. My requirement is basically if all nodes of the topic goes down, can I pause the producer and after the broker comes back online, continue to produce from the failure point? Best, Siyuan

Is there a way to delete partition at runtime?

Hi guys, I found there is a tool to add partition on the fly. My question is, is there a way to delete a partition at runtime? Thanks! Best, Siyuan

Re: kafka_2.8.0/0.8.0 pom seems invalid

Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > / > > > On Wed, Dec 4, 2013 at 4:48 PM, hsy...@gmail.com wrote: > > > Hi All, I was trying

kafka_2.8.0/0.8.0 pom seems invalid

Hi All, I was trying to upgrade the kafka to 0.8 but I get an empty jar file for org.apache.kafka kafka_2.8.0 0.8.0 However org.apache.kafka kafka_2.8.2 0.8.0 is good for me. BTW from the download page I can only see kafka_2.8.0_0.8.0. Where can I download the scala

Re: Consuming from a replica

What I did for my project is I have a thread send metadata request to a random broker and monitor the metadata change periodically. The good thing is, to my knowledge, any broker in the cluster know the metadata for all the topics served in this cluster. Another options is you can always query zook

Re: Producer reaches a max of 7Mbps

I think the max 50Mbps is almost the disk bottleneck My guess is IO is the bottle neck for kafka if you set to same type(async without ack) I got throughput at about 30Mb Try to increase if you don't care about latency very much log.flush.interval.messages=1 log.flush.interval.ms=3000 On Tue

High-level consumer load-balancing problem

Hi, I have questions about the load balancing of kafka high-level consumer Suppose I have 4 partition And the producer throughput to these 4 partitions are like this 01 23 10MB/s 10MB/s 1MB/s1MB/s 1kMsg/s,10kMs

Re: will this cause message loss?

Also if you use HEAD, you can create more partitions at runtime, you just need dynamic partitioner class I think On Thu, Nov 14, 2013 at 7:23 AM, Neha Narkhede wrote: > There is no way to delete topics in Kafka yet. You can add partitions to > existing topics, but you may have to use 0.8 HEAD si

<    1   2   3   >