Re: Does withkeys transform enforce a reshuffle?

Also I looked at the code, reshuffle seems doing some groupby work internally. But I don't really need groupby On Fri, Jan 19, 2024 at 9:35 AM hsy...@gmail.com wrote: > ReShuffle is deprecated > > On Fri, Jan 19, 2024 at 8:25 AM XQ Hu via user > wrote: > >> I do

Re: Does withkeys transform enforce a reshuffle?

thKeys > > Have you tried to just add ReShuffle after PubsubLiteIO? > > On Thu, Jan 18, 2024 at 8:54 PM hsy...@gmail.com wrote: > >> Hey guys, >> >> I have a question, does withkeys transformation enforce a reshuffle? >> >> My pipeline basicall

Does withkeys transform enforce a reshuffle?

Hey guys, I have a question, does withkeys transformation enforce a reshuffle? My pipeline basically look like this PubsubLiteIO -> ParDo(..) -> ParDo() -> BigqueryIO.write() The problem is PubsubLiteIO -> ParDo(..) -> ParDo() always fused together. But The ParDo is expensive and I want

ParDo(DoFn) with multiple context.output vs FlatMapElements

Hello I have a question. If I have a transform for each input it will emit 1 or many output (same collection) I can do it with ParDo + DoFun while in processElement method for each input, call context.output multiply times vs doing it with FlatMapElements, is there any difference? Does the

pubsubliteio ack problem

In my application, the pubsubliteio seems never ack the message and the data lateness is building up forever, my question is how does dataflow know when to ack the message, How does the engine even know when it is processed?

How to set flow control for pubsubliteio?

How to change flow control config for pubsubliteio ? I saw the setting has been taken out as part of https://issues.apache.org/jira/browse/BEAM-14129 But without setup flow control correctly, my beam app is running super slow ingesting from pubsbulite and getting NO_CLIENT_TOKEN error on the

Re: pubsubliteio is super slow

, 2023 at 10:17 AM hsy...@gmail.com > wrote: > >> Any one is using pubsublite? I find it super slow 5 messages/sec and >> there is no options for me to tune the performance >> >

pubsubliteio is super slow

Any one is using pubsublite? I find it super slow 5 messages/sec and there is no options for me to tune the performance

Re: Questions about writing to BigQuery using storage api

> On Thu, Dec 7, 2023 at 8:46 AM hsy...@gmail.com wrote: > >> Here is the complete stacktrace It doesn't even hit my code and it >> happens consistently! >> >> Error message from worker: java.lang.RuntimeException: >> java.lang.IllegalStateException >> or

Re: Questions about writing to BigQuery using storage api

ll make this more >> straightforward. >> >> On Wed, Dec 6, 2023 at 11:24 AM hsy...@gmail.com >> wrote: >> >>> I’m just using dataflow engine >>> On Wed, Dec 6, 2023 at 08:23 John Casey via user >>> wrote: >>> >>>> Well, that is odd.

Re: Questions about writing to BigQuery using storage api

alsaud this might be interesting to > you too > > On Tue, Dec 5, 2023 at 9:39 PM hsy...@gmail.com wrote: > >> I'm using version 2.51.0 and The configuration is like this >> >> write >> .withoutValidation() >> .withCreateDisposition(BigQueryIO.Write.

Re: Questions about writing to BigQuery using storage api

hich beam version are you using? > > > On Tue, Dec 5, 2023 at 1:52 PM hsy...@gmail.com wrote: > >> Any one has experience in writing to BQ using storage api >> >> I tried to use it because according to the document it is more efficient >> but I got error below >&

Questions about writing to BigQuery using storage api

Any one has experience in writing to BQ using storage api I tried to use it because according to the document it is more efficient but I got error below 2023-12-05 04:01:29.741 PST Error message from worker: java.lang.RuntimeException: java.lang.IllegalStateException

Watermark never progress for deduplicate transform

Hello, I'm using the https://beam.apache.org/releases/javadoc/2.21.0/org/apache/beam/sdk/transforms/Deduplicate.html transform to help dedup my data but in the monitoring page I see the watermark is not moving forward. Is it common for that transformation? Thanks

Question about metrics

Hi I have questions about metrics. I want to use beam metrics api to send metrics to GCP monitoring. Instead of collecting just some simple numeric values. I also need to send labels along with them. Is there a way to do that? Thanks!

Streaming write to orc problem

Hello all, I’m just trying to build a pipeline reading data from a streaming source and write to orc file. But I don’t see any file that is written to the file system nor any exceptions Here is an example val df = spark.readStream.format(“...") .option( “Topic", "Some

I got weird error from a join

from pyspark.sql import Row A_DF = sc.parallelize( [ Row(id='A123', name='a1'), Row(id='A234', name='a2') ]).toDF() B_DF = sc.parallelize( [ Row(id='A123', pid='A234', ename='e1') ]).toDF() join_df = B_DF.join(A_DF, B_DF.id==A_DF.id).drop(B_DF.id)

Questions about using pyspark 2.1.1 pushing data to kafka

I have questions about using pyspark 2.1.1 pushing data to kafka. I don't see any pyspark streaming api to write data directly to kafka, if there is one or example, please point me to the right page. I implemented my own way which using a global kafka producer and push the data picked from

Question about accumulator

I have a small application like this acc = sc.accumulate(5) def t_f(x,): global acc sleep(5) acc += x def f(x): global acc thread = Thread(target = t_f, args = (x,)) thread.start() # thread.join() # without this it doesn't work rdd = sc.parallelize([1,2,4,1])

Re: [EXTERNAL] kafka

Hey Raja, The setup for secure kafka input operator is not easy. You can follow these steps. 1. Follow kafka document to setup your brokers properly ( https://kafka.apache.org/090/documentation.html#security_overview) 2. You have to manually create the client JAAS file (

Re: Datatorrent fault tolerance

String, > String>("/datalake/corporate/ses_dlpoc/stream_fabbddev/jsingh73/sample-stream:error", > tenant.getVolumeName(), gson.toJson(tenant))); > > } > producer.flush(); > } > } > > > &g

Re: Datatorrent fault tolerance

On Fri, Oct 7, 2016 at 1:27 PM, hsy...@gmail.com <hsy...@gmail.com> wrote: > >> Jaspal, >> >> Topic is a mandatory property you have to set. In mapr, the value should >> be set to the full stream path example: /your/stream/path:streamname >> >> Regards, >>

Re: Datatorrent fault tolerance

Hey Jaspal, Did you add any code to existing KafkaSinglePortExactlyOnceOutputOperator from malhar? If so please make sure the producer you use here is org.apache.kafka.clients.producer.KafkaProducer instead of kafka.javaapi.producer.Producer. That is old api and that is not supported by MapR

Re: Apex and Malhar Java 8 Certified

I think the problem is what people expect when we say "certified". To me, If I see something is certified with java 8, I would assume that I can use java 8 api(new features stream, lambda etc.) to write the operator code, not only just run the code with jre 8 or compile existing code with jdk 8

Re: Emit values from an array

Is you pojo a shared object? I think you need to create new pojo every time. Regards, Siyuan On Thu, Sep 29, 2016 at 3:03 PM, Jaikit Jilka wrote: > Hello, > > I am trying to emit values from an array. I am emitting using an for loop. > Number of records emitted is

Re: Malhar 0.8.1.1 Kafka Operator and chroot

Hey Alex, Does the workaround work? I just want to follow up to see my hypothesis for the root cause is correct. Thanks! Regards, Siyuan On Wed, Aug 24, 2016 at 10:56 AM, hsy...@gmail.com <hsy...@gmail.com> wrote: > Hey Alex, > > Yeah, I think there is a bug for multitenan

Re: Malhar 0.8.1.1 Kafka Operator and chroot

Hey Alex, Do you use ONE_TO_ONE or ONE_TO_MANY partition? Regards, Siyuan On Wed, Aug 24, 2016 at 10:27 AM, McCullough, Alex < alex.mccullo...@capitalone.com> wrote: > Hey Siyuan, > > > > We are using 3.4.0 > > > > Thanks, > > Alex > > *From: *&

Re: can operators emit on a different from the operator itself thread?

Hey Vlad, Thanks for bringing this up. Is there an easy way to detect unexpected use of emit method without hurt the performance. Or at least if we can detect this in debug mode. Regards, Siyuan On Wed, Aug 10, 2016 at 11:27 AM, Vlad Rozov wrote: > The short answer is

Re: Force Fail Application

He Michael, You can throw a ShutdownException. Siyuan On Mon, Jul 18, 2016 at 10:06 AM, Silver, Michael < michael.sil...@capitalone.com> wrote: > > > > > Hello, > > > > I am looking for a solution to force shutdown or fail my application. I > have an operator that checks that a file (which is

Re: Regarding using Scala to develop Apex app.

BTW Akshay, if you are using anonymous function as a field in the operator, it's very likely that your function is stateless? If that's the case, you can try to mark it transient ( http://www.scala-lang.org/api/rc2/scala/transient.html) On Thu, Jul 14, 2016 at 11:06 PM, Akshay S Harale <

Re: A proposal for Malhar

<a...@datatorrent.com> wrote: > > My vote is to do 2&3 > > Thks > Amol > > > On Tue, Jul 12, 2016 at 12:14 PM, Kottapalli, Venkatesh < > vkottapa...@directv.com> wrote: > >> +1 for deprecating the packages listed below. >> >> -Original

Re: A proposal for Malhar

<a...@datatorrent.com> wrote: > > My vote is to do 2&3 > > Thks > Amol > > > On Tue, Jul 12, 2016 at 12:14 PM, Kottapalli, Venkatesh < > vkottapa...@directv.com> wrote: > >> +1 for deprecating the packages listed below. >> >> -Original

Re: A proposal for Malhar

+1 On Tue, Jul 12, 2016 at 11:53 AM, David Yan wrote: > Hi all, > > I would like to renew the discussion of retiring operators in Malhar. > > As stated before, the reason why we would like to retire operators in > Malhar is because some of them were written a long time

Re: A proposal for Malhar

+1 On Tue, Jul 12, 2016 at 11:53 AM, David Yan wrote: > Hi all, > > I would like to renew the discussion of retiring operators in Malhar. > > As stated before, the reason why we would like to retire operators in > Malhar is because some of them were written a long time

Re: DataTorrent with SBT: .apa file not created

I've never used SBT to build Apex application. But I guess you can try 2 things here Use the sbt maven plugin https://github.com/shivawu/sbt-maven-plugin or use sbt assembly plugin https://github.com/sbt/sbt-assembly In the 2nd way, you need to translate the plugin configuration part in pom.xml

How do I turn off INFO log for embedded kafka server

Hey guys, I have some unit test that has an embedded kafka server running. I want to skip all debug and info logs from kafka server. But having this set in log4j.properties does work. Some INFO log still keep showing up like this: 2016-07-08 18:01:14,288 [kafka-request-handler-4] INFO

log4j setting for embedded kafka server

Hi guys, I start server grammatically in my application using KafkaStatableServer.startup() method. And in the log4j.properties setting. I add this log4j.logger.org.apacke.kafka=WARN log4j.logger.kafka=WARN But I always get INFO log, Do you guys know how to enforce the log level here? Thanks!

Re: kafka offset commit

hanks Siyuan. > > So, to confirm, to apex is not storing offsets status at any location ? > Like how Storm stores in Zookeeper ? > > > Regards, > Raja. > > From: "hsy...@gmail.com" <hsy...@gmail.com> > Reply-To: "users@apex.apache.org" <users@

Log level for consumer properties

Hi, Right now, when we initialize kafka consumer, it always log the consumer properties at INFO level, can we put it into DEBUG level? I have to periodically create consumer instance to just pull some metadata of some topic and I don't want to get this noisy log. Regards, Siyuan

Re: javax.persistence.Persistence classNotFoundError

Hey Ananth, Can you unzip your apa file to see if it is in the lib folder? If it is not there, it means the packaging is not correct. Regards, Siyuan On Mon, May 2, 2016 at 4:39 AM, Ananth Gundabattula wrote: > Hello All, > > I was able to get around the issue pasted

Re: Apex engine and malhar maven versions for Kafka 0.9 version operator

Hi Ananth, Could you please try to add malhar-library in test scope? Thanks! Regards, Siyuan On Saturday, April 16, 2016, Ananth Gundabattula wrote: > Hello all, > > I would like to run a unit test code using the new Kafka Operator that > supports the 0.9 version

Exceptions when programmatically start multiple kafka brokers

I'm trying to start 2 brokers in my kafka ingestion unit test and I got exception javax.management.InstanceAlreadyExistsException: kafka.server:type=app-info,id=0 at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) at

Re: how to programatically monitor Kafka availability

Hey Hohl, I use *partitionsFor * method to monitor the partition info for particular topics On Tue, Dec 15, 2015 at 11:27 AM, Hohl, Ken wrote: > We

Where can I find the document for consumer metrics

I can find some broker/producer metrics here http://kafka.apache.org/documentation.html#monitoring but where can I find consumer metrics docs Everytime I have to log this to find out what metrics I want MetricName [name=join-rate, group=consumer-coordinator-metrics, description=The number of

Re: Kafka 0.9 consumer API question

Hi Rajiv, I think it makes sense to return a read-only assignments. What we can improve here is we can have addPartition method for consumer. Then we don't have to do any operations on the assignments returned by assignment method BTW, I think you can implement PartitionAssignor interface to

partitionsFor method doesn't return latest partition metadata

Hi guys, I want to use partitionsFor method of new consumer API periodically to get the monitor partition metadata change, It seems it only issue remote call to the server for the first time. If I add partitions after that, partitionsFor will return stale value. Is there a way to reuse consumer

partitionsFor method doesn't return latest partition metadata

Hi guys, I want to use partitionsFor method of new consumer API periodically to get the monitor partition metadata change, It seems it only issue remote call to the server for the first time. If I add partitions after that, partitionsFor will return stale value. Is there a way to reuse consumer

Is 0.9 new consumer API compatible with 0.8.x.x broker

Is 0.9 new consumer API compatible with 0.8.x.x broker

Re: 0.9.0.0 RC4

In http://kafka.apache.org/090/documentation.html#newconsumerconfigs partition.assignment.strategy should string, not a list of string? On Fri, Nov 20, 2015 at 5:21 PM, Jun Rao wrote: > This is the fourth candidate for release of Apache Kafka 0.9.0.0. This a > major release

Re: 0.9.0.0 RC4

In http://kafka.apache.org/090/documentation.html#newconsumerconfigs partition.assignment.strategy should string, not a list of string? On Fri, Nov 20, 2015 at 5:21 PM, Jun Rao wrote: > This is the fourth candidate for release of Apache Kafka 0.9.0.0. This a > major release

Re: Commit offsets only work for subscribe(), not assign()

nt.io> > wrote: > > > Hey Siyuan, > > > > The commit API should work the same regardless whether subscribe() or > > assign() was used. Does this not appear to be working? > > > > Thanks, > > Jason > > > > On Wed, Nov 18, 2015 at 4:40

Re: Questions about new consumer API

account the > leaders of each partition. Then you could just use subscribe() and let > Kafka manage the group for you. This is similar to how we were thinking of > implementing consumer rack-awareness. > > -Jason > > On Tue, Nov 17, 2015 at 4:04 PM, hsy...@gmail.com <hsy...@gm

Q about PartitionAssignor

Hey guys, I saw the PartitionAssignor is not in public doc API and the package name is internals. Does it mean this API is not stable and could be changed even in minor release? And in the assign method signature, the key for the "subscription" map is memberId, what is memberId, can I manually

Commit offsets only work for subscribe(), not assign()

In the new API, the explicit commit offset method call only works for subscribe consumer, not the assign consumer, correct? Best, Siyuan

Re: Q about PartitionAssignor

gnment has > failed over other new leaders can still access the past assignment history. > > Guozhang > > > > On Wed, Nov 18, 2015 at 9:02 AM, hsy...@gmail.com <hsy...@gmail.com> > wrote: > > > Hey guys, > > > > I saw the PartitionAssignor is n

Re: build error

n, Nov 16, 2015 at 6:41 PM, hsy...@gmail.com <hsy...@gmail.com> > wrote: > > > The actual thing I want to do is I want to build and install in my local > > maven repository so I can include new api in my dependencies. When the > > release is officially out, I can have both my code

Re: build error

On Mon, Nov 16, 2015 at 6:41 PM, hsy...@gmail.com <hsy...@gmail.com> > wrote: > > > The actual thing I want to do is I want to build and install in my local > > maven repository so I can include new api in my dependencies. When the > > release is officially out, I can hav

Re: Questions about new consumer API

ector will close idle ones after some > timeout. So in worst case it will only maintain N+1 sockets in total for N > Kafka brokers at one time. > > Guozhang > > On Mon, Nov 16, 2015 at 4:22 PM, hsy...@gmail.com <hsy...@gmail.com> > wrote: > > > The new consum

Re: Questions about new consumer API

sFor() to find the current partition leaders and assign() to set > the assignment in each thread. > > -Jason > > On Tue, Nov 17, 2015 at 10:25 AM, hsy...@gmail.com <hsy...@gmail.com> > wrote: > > > Thanks Guozhang, > > > > Maybe I should give a few words

build error

I got a build error on both trunk and 0.9.0 branch > docs/producer_config.html (No such file or directory) Do I miss anything before build Thanks, Siyuan

Questions about new consumer API

The new consumer API looks good. If I understand it correctly you can use it like simple consumer or high-level consumer. But I have couple questions about it's internal implementation First of all does the consumer have any internal fetcher threads like high-level consumer? When you assign

Re: build error

did you use to build? > > > > Guozhang > > > > On Mon, Nov 16, 2015 at 4:01 PM, hsy...@gmail.com <javascript:;> < > hsy...@gmail.com <javascript:;>> > > wrote: > > > > > I got a build error on both trunk and 0.9.0 b

Re: Packaging new apps

Hi Jean, Thanks for the change, using instance tag(is it a new feature in the latest version? I didn't see it in the older slider versions) is a really good idea. it might be good for other's to have a template but not for kafka. Kafka is evolving in quite fast pace. I've seen many property

Re: Component instance level configuration

port can be different for instance of a component nowadays. 2015-01-28 5:57 GMT+08:00 hsy...@gmail.com hsy...@gmail.com: Hi, I'm just wondering if there is a way to support component instance level configuration. For example, in kafka-on-yarn, kafka broker is a component, but each

Component instance level configuration

Hi, I'm just wondering if there is a way to support component instance level configuration. For example, in kafka-on-yarn, kafka broker is a component, but each broker instance needs a unique id. It would be better to expose the instance id, so that different instance would override some of the

Given brokers, is it able to know all the zookeepers that brokers connect to

Hi Guys, If I know the brokers. Is there a way to know the zookeeper host from broker list? Thanks! Siyuan

Re: Is it able to specify the percentage of memory instead of # of MB

for available memory and core, etc. On Mon, Dec 1, 2014 at 1:39 PM, hsy...@gmail.com hsy...@gmail.com wrote: Hi guys, For each component instance, Is there a way to specify the % of total memory of each node instead of # of MB? Best, Siyuan

Questions about new consumer API

Hi guys, I'm interested in the new Consumer API. http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/ I have couple of question. 1. In this doc it says kafka consumer will automatically do load balance. Is it based on throughtput or same as what we have now balance the

Re: Questions about new consumer API

consumer api still be supported? Over time, we will phase out the current high-level and simple consumer since the 0.9 API supports both. Thanks, Neha On Tue, Dec 2, 2014 at 12:07 PM, hsy...@gmail.com hsy...@gmail.com wrote: Hi guys, I'm interested in the new Consumer API. http

Is it able to specify the percentage of memory instead of # of MB

Hi guys, For each component instance, Is there a way to specify the % of total memory of each node instead of # of MB? Best, Siyuan

Is there a plan to build a ubiquitous web service API to manage the kafka cluster

Hi guys, Nowadays, all kafka administration work (add, tear down node, topic management, throughput monitor) are done by various different tool talk to brokers, zookeeper etc. Is there a plan for core team to build a central universal server providing webservice API to do all the admin work?

I can't specify user by calling Execute method

Hi I specified user when call Execute(..., user='hdp',...) But the container processes are all launched as *yarn* user

slider stop bug?

If the application is stuck in ACCEPT status, slider stop command doesn't stop the yarn application. I didn't setup the scheduler address correctly so the application stuck at ACCEPT status and I couldn't use slider stop to stop the application Best, Siyuan

Re: slider stop bug?

I'm using 0.60 On Tue, Nov 18, 2014 at 12:26 PM, Sumit Mohanty smoha...@hortonworks.com wrote: You can use --force. Are you using the RC for 0.60 or 0.40? On Tue, Nov 18, 2014 at 12:00 PM, hsy...@gmail.com hsy...@gmail.com wrote: If the application is stuck in ACCEPT status, slider stop

Application configuration page is outdated?

Hi guys, I just switch from 40 to 60 and I found the application configuration doesn't work any more. http://slider.incubator.apache.org/docs/slider_specs/application_instance_configuration.html For example : { schema : http://example.org/specification/v2.0.0;, metadata : { }, global :

Re: Application configuration page is outdated?

...@gmail.com hsy...@gmail.com wrote: Hi guys, I just switch from 40 to 60 and I found the application configuration doesn't work any more. http://slider.incubator.apache.org/docs/slider_specs/application_instance_configuration.html For example : { schema : http://example.org

How to try 1.7?

Hi guys, Is there an easy way to try 1.7, probably with HDP2.2? Thanks!

Do I have to repackage if I change appConfig.json/resources.json

Everytime I change appConfig.json and resources.json. Do I have to repackage the zip file and redeploy the file to hdfs? Thanks!

Re: Do I have to repackage if I change appConfig.json/resources.json

, 2014, at 2:54 PM, hsy...@gmail.com wrote: Everytime I change appConfig.json and resources.json. Do I have to repackage the zip file and redeploy the file to hdfs? Thanks! -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which

Re: Do I have to repackage if I change appConfig.json/resources.json

with --template appConfig.json --resources resources.json ... -Sumit On Fri, Nov 14, 2014 at 2:08 PM, hsy...@gmail.com hsy...@gmail.com wrote: So actually you don't need appConfig.json and resources.json in the ZIP package? On Fri, Nov 14, 2014 at 12:05 PM, Jon Maron jma

How to change log level for Slider AM?

How to change log level for Slider AM? Thanks!

Re: Do I have to repackage if I change appConfig.json/resources.json

.* Which branch are you using? (for me to answer the next questions) On Fri, Nov 14, 2014 at 3:47 PM, hsy...@gmail.com hsy...@gmail.com wrote: I'm getting confused. After you edit your appConfig.json and resources.json. You don't put it back in your package zip file? So the appConfig

Question about behavior after container failures

Hi all, In the document, I got that slider will try to recover on container failure. But in my test application it doesn't I'm using the 0.40 release. build from source Here is what I get. *If I kill the child process * The agent check_process_status method will raise ComponentIsNotRunning and

Re: Do I have to repackage if I change appConfig.json/resources.json

from last snapshot If you stop and then start (probably it is freeze and thaw in 0.40) the application then it will use the hostnames that it remembered. -Sumit On Fri, Nov 14, 2014 at 4:35 PM, hsy...@gmail.com hsy...@gmail.com wrote: I'm actually using 0.40 On Fri, Nov 14, 2014 at 4:21

Re: Do I have to repackage if I change appConfig.json/resources.json

the hostnames that it remembered. What if I use *yarn application -kill* to stop the application? On Fri, Nov 14, 2014 at 5:09 PM, hsy...@gmail.com hsy...@gmail.com wrote: The only problem is I have to upgrade to hadoop 2.6? On Fri, Nov 14, 2014 at 5:08 PM, Sumit Mohanty smoha...@hortonworks.com wrote

Re: log4j dir?

Anyone has any idea how do I config the log4j file dir? On Thu, Nov 13, 2014 at 4:58 PM, hsy...@gmail.com hsy...@gmail.com wrote: Hi guys, Just notice kafka.logs.dir in log4j.properties doesn't take effect It's always set to *$base_dir/logs* in kafka-run-class.sh LOG_DIR=$base_dir/logs

Re: log4j dir?

...@gmail.com hsy...@gmail.com wrote: Anyone has any idea how do I config the log4j file dir? On Thu, Nov 13, 2014 at 4:58 PM, hsy...@gmail.com hsy...@gmail.com wrote: Hi guys, Just notice kafka.logs.dir in log4j.properties doesn't take effect It's always set to *$base_dir/logs

Re: Q: How to define component configuration?

, 2014 at 2:42 PM, hsy...@gmail.com hsy...@gmail.com wrote: Thanks Steve, but I logged all the properties in params.py, I couldn't find any key named *mykey* *config = Script.get_config()* for key in config.keys(): print key: + key for key in config['global'].keys

log4j dir?

Hi guys, Just notice kafka.logs.dir in log4j.properties doesn't take effect It's always set to *$base_dir/logs* in kafka-run-class.sh LOG_DIR=$base_dir/logs KAFKA_LOG4J_OPTS=-Dkafka.logs.dir=$LOG_DIR $KAFKA_LOG4J_OPTS Best, Siyuan

Re: Q: How to define component configuration?

is picked up It's essentially a form of prototype-based programming, except only of properties, not methods: http://en.wikipedia.org/wiki/Prototype-based_programming On 11 November 2014 01:30, hsy...@gmail.com hsy...@gmail.com wrote: Thanks Ted, but back to my first question, how can you

Q: How to define component configuration?

Hi guys, Is there an example of component configuration? Is there a way to give different value to same property for different instances? Siyuan

Re: Q: How to define component configuration?

If I want to have several instances of some component. But I want to set some of the properties to different value for different instances. How can I do it? Thanks! Best, Siyuan On Mon, Nov 10, 2014 at 1:26 PM, hsy...@gmail.com hsy...@gmail.com wrote: Hi guys, Is there an example

Re: Q: How to define component configuration?

...@gmail.com wrote: To my knowledge, there is no direct support for this. You can create different components, each with corresponding properties. Cheers On Mon, Nov 10, 2014 at 4:13 PM, hsy...@gmail.com hsy...@gmail.com wrote: If I want to have several instances of some component. But I want

Is it able to make AM to restart from previous state?

Hi guys, I noticed in the code when a container fails it will try to relaunch from the same node. My question is if I restart whole application(Ex. AM got killed, or manually restart the app). Does slider try to launch all containers from the nodes where it was running? Thanks! Best, Siyuan

Re: Is it able to make AM to restart from previous state?

? Thanks On Wed, Nov 5, 2014 at 12:46 PM, Steve Loughran ste...@hortonworks.com wrote: On 5 November 2014 20:21, hsy...@gmail.com hsy...@gmail.com wrote: Hi guys, I noticed in the code when a container fails it will try to relaunch from the same node. My question is if I restart whole

Re: Is it able to make AM to restart from previous state?

from new node? What if you decrease the instance number? Thanks! Best, Siyuan On Wed, Nov 5, 2014 at 1:44 PM, hsy...@gmail.com hsy...@gmail.com wrote: Thanks Steve, Is No 1 a new feature in YARN (Not released yet)? And you mentioned slider saves the location in history files. What

Maven repository?

Is there a public maven repository that I can checkout the slider library? Best, Siyuan

How to inject my own web service handler into app master?

Hi, I noticed slider app master has an embedded web service server. Is there a way to inject my own web service call into that server? Another question how to you communication between containers(components) and AM(through agent code)? Is there a way to customize the communication/data between

Re: Why command script is written in python?

script types. Of course, today the only supported/tested one is python. On Thu, Oct 30, 2014 at 1:55 PM, hsy...@gmail.com hsy...@gmail.com wrote: May I ask why do you choose python to write the command script. And is there any other language? I find difficult to debug because each time if I

Re: Why command script is written in python?

during development. If you do not mind can you open a JIRA at https://issues.apache.org/jira/browse/SLIDER describing the requirements of debugging during development time and post development. I will use that JIRA to drive the issue. -Sumit On Thu, Oct 30, 2014 at 2:40 PM, hsy...@gmail.com

  1   2   >