Agreed on this.
Thanks,
Fang, Yan
yanfang...@gmail.com
On Tue, Jun 16, 2015 at 10:14 AM, Guozhang Wang wangg...@gmail.com wrote:
Hi all,
We have been running a couple of our jobs against `0.9.1` branch last week
at LinkedIn with some critical bug fixes back-ported, including:
SAMZA-608
/TestKafkaSystemConsumer.scala
2a84328
samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java
b063366
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala
1e936b4
Diff: https://reviews.apache.org/r/34974/diff/
Testing
---
Thanks,
Yan
/TestKafkaSystemConsumer.scala
2a84328
samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java
b063366
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala
1e936b4
Diff: https://reviews.apache.org/r/34974/diff/
Testing
---
Thanks,
Yan Fang
Hi Shekar,
a. How do I detect it?
-- You can go to Yarn's monitoring page, usually, localhost:8088. May see
what are running.
b. How do I kill redundant jobs?
-- Just like what you do for any Yarn jobs. *yarn application -kill
applicationId* . If you are using the similar structure as the
:39 AM, Navina Ramesh
nram...@linkedin.com.invalid wrote:
+1 for adding to the documentation. I didn¹t know about this difference.
This information is perhaps part of the JobCoordinator/CoordinatorStream
documentation - SAMZA-614?
On 6/15/15, 11:27 AM, Yan Fang yanfang...@gmail.com wrote
this.
Fang, Yan
yanfang...@gmail.com
On Fri, Jun 12, 2015 at 5:09 PM, Yan Fang yanfang...@gmail.com wrote:
Hi guys,
I am looking at the code. Do we have any special reason that we want to
keep two web servlets in the AM? One is for the Rest, the other one is for
the AM dashboard. Can we combine them
I think they are doing some maintenance. I even can not access the JIRA...
Fang, Yan
yanfang...@gmail.com
On Fri, Jun 12, 2015 at 11:01 AM, Yi Pan nickpa...@gmail.com wrote:
Hi, all,
Just FYI that the cwiki links are down now. I have filed an infra ticket
for that: INFRA-9806 - Cwiki site
of the last 1 mins/5 mins, right?
samza-api/src/main/java/org/apache/samza/metrics/Meter.java
https://reviews.apache.org/r/35053/#comment139802
timer - meter
also added unit tests for the histagram and meter.
- Yan Fang
On June 4, 2015, 7:22 a.m., Luis De Pombo wrote
Hi Chas,
The actually connection does not happen in the StreamTask, but the
Consumer/Producer layer. One way of understanding this is that,
* Consumers are run in a separate thread, which keeps the connection to the
server (kafka broker, wiki api, etch) and fetches messages from them;
*
Hi Garrett,
I guess you run into the serde issues as you mentioned. If you can check
the Samza log and show us, we will be more helpful. Also, maybe pasting the
config here (if you dont mind), we can help to see if you miss something.
Thanks,
Fang, Yan
yanfang...@gmail.com
On Tue, Jun 2, 2015
-- Hi Jay,
Thanks for forwarding this.
-- Hi TD,
Thanks for pointing this out. That overview was written about one year ago.
It is out-of-date. Good to get some critiques from your side. Will update
it soon. Created SAMZA-698 https://issues.apache.org/jira/browse/SAMZA-698 to
track this. Thank
---
Thanks,
Yan Fang
Hi Renato,
Grant you the privilege to create pages. Should be able to create a page to
track your GSoC progress.
Thanks,
Fang, Yan
yanfang...@gmail.com
On Sun, May 31, 2015 at 12:07 PM, Renato Marroquín Mogrovejo
renatoj.marroq...@gmail.com wrote:
Hi all,
I would like to create a wiki
/SystemConsumersMetrics.scala
https://reviews.apache.org/r/34888/#comment137897
change to registerSystemStreamPartition?
- Yan Fang
On June 1, 2015, 1:27 p.m., József Márton Jung wrote:
---
This is an automatically generated e-mail. To reply, visit
Hi José,
Thank you. If you can contribute a patch for this fix (SAMZA-688
https://issues.apache.org/jira/browse/SAMZA-688), it would be very
helpful. And here
https://cwiki.apache.org/confluence/display/SAMZA/Contributor%27s+Corner
is the guide for contributing.
Cheers,
Fang, Yan
websites when
releasing new version (6 weeks ago) Yan Fang
* 6946f78 - SAMZA-540: Expose checkpointed offset in OffsetManager (6
weeks ago) Renato Marroquín Mogrovejo
* 9427c9e - SAMZA-634; stop job coordinator when process job exits (6
weeks ago) Tommy Becker
* 81f54a2 - SAMZA-616; make
/samza/container/SamzaContainerMetrics.scala
https://reviews.apache.org/r/34564/#comment136334
do we want to create a new Gauge every loop? I think one Gauge is
sufficient, we can update its value. What do you think?
- Yan Fang
On May 21, 2015, 8:53 p.m., Luis De Pombo wrote
Hi Renato,
There maybe a misunderstanding in the concept. Consumer is to feed the msgs
into the Samza, while Producer is to send the msg from Samza to other
systems. So if you implement the Consumer, should be able to see the msgs
in the StreamTask. That's why you confuse Naveen.
Cheers,
Fang,
Hi Shekar,
This website requires signed in. Could you past to another more open place
? There are a lot if you just google it. Sorry for being a little picky.
Also, when you see a lot of jobs in the accepted state, what does the log
say?
Thanks,
Fang, Yan
yanfang...@gmail.com
On Wed, May 20,
On Thu, May 21, 2015 at 2:21 PM, Yan Fang yanfang...@gmail.com wrote:
Hi guys,
Just ask, are there any other bugs that we want to back port to 0.9.1
besides SAMZA-662 https://issues.apache.org/jira/browse/SAMZA-662 ?
If no, I think we can prepare the 0.9.1 release and ask
Hi guys,
Just ask, are there any other bugs that we want to back port to 0.9.1
besides SAMZA-662 https://issues.apache.org/jira/browse/SAMZA-662 ?
If no, I think we can prepare the 0.9.1 release and ask for the vote.
Cheers,
Fang, Yan
yanfang...@gmail.com
://reviews.apache.org/r/33419/diff/
Testing
---
tested with multiple partitions and multiple stores recovery.
Thanks,
Yan Fang
to the slave resource manager node
3. restart yarn on node 1.
I am not sure if I missed anything.
- Shekar
On Thu, May 14, 2015 at 3:06 PM, Yan Fang yanfang...@gmail.com wrote:
Is the HA set correctly? The log looks like it's in the YARN setting
side.
Fang, Yan
yanfang...@gmail.com
Hi Elango,
Currently Samza does not support having two config files, like
--classpath-config-file.
But there are a few workarounds:
1. you already mentioned putting the properties in the samza properties file
2. use key=value pair. so it would look like deploy/samza/bin/run-job.sh
Is the HA set correctly? The log looks like it's in the YARN setting side.
Fang, Yan
yanfang...@gmail.com
On Thu, May 14, 2015 at 12:29 PM, Shekar Tippur ctip...@gmail.com wrote:
Other observation I forgot to mention is that if I kill the rm and nm
process, samza job seem to run properly.
Hi Shekar,
Currently by default, the log4j is using DailyRollingFileAppender. You can
change the log4j.xml to config as you want. Usually, daily base is good
enough.
Thanks,
Fang, Yan
yanfang...@gmail.com
On Thu, May 14, 2015 at 10:56 AM, Shekar Tippur ctip...@gmail.com wrote:
Hello,
How
/configuration-table.html ,
that will be great. Thanks!
- Yan Fang
On May 11, 2015, 10:10 a.m., Luis De Pombo wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34011
please expand on what you mean by get the rule when starting
up StreamTasks
and then localize it.? Do you mean, loading messages into a changelog
stream using a bootstrap job?
Thanks in advance for your help!
Susan
On Tue, May 5, 2015 at 6:02 PM, Yan Fang yanfang...@gmail.com wrote
On May 7, 2015, 8:07 p.m., Yi Pan (Data Infrastructure) wrote:
samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala,
line 94
https://reviews.apache.org/r/33735/diff/2/?file=951495#file951495line94
It would be nicer to try catch the exception here
On May 6, 2015, 12:10 a.m., Yan Fang wrote:
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala,
lines 481-486
https://reviews.apache.org/r/33453/diff/4/?file=950515#file950515line481
1. why does the changeLogSystemStreamPartition matter here?
2
Dir?
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala
https://reviews.apache.org/r/33453/#comment133371
personal opinion: should this logic be another method, such as
readOffsetFromFile, not cleanBaseDir? A little misleading.
- Yan Fang
On May 5, 2015, 6:47
If the rule does not change, we can get the rule when starting up
StreamTasks and then localize it.
Cheers,
Fang, Yan
yanfang...@gmail.com
On Tue, May 5, 2015 at 2:41 PM, Yan Fang yanfang...@gmail.com wrote:
If I understand it correctly the only viable solution at the moment is to
create
it correctly the only viable solution at the moment is to
create a new stream for the rules messages with as many partitions as the
data stream and write each rules update message to all partitions of the
new rules stream.
Cheers,
- ueli
On Tue, May 5, 2015 at 12:06 PM, Yan Fang yanfang
Hi Andreas,
Not quite understand this part
Because the messages coming into the input stream are random (i.e. can hit
any partition and therefore any task), each task will need its own copy of
the data (i.e. the data needs to be duplicated across each task).
Messages come into the input stream
/TestLog4jSystemConfig.java
6314a3e
samza-shell/src/main/bash/state-storage-tool.sh PRE-CREATION
Diff: https://reviews.apache.org/r/33419/diff/
Testing
---
tested with multiple partitions and multiple stores recovery.
Thanks,
Yan Fang
by elastic, what is set, what is read, etc. Also The
ElasticConfig should only be responsible for reading the config and we create
factories in other classes, such as in the Producer class, if needed. What do
you think?
- Yan Fang
On May 1, 2015, 6:23 p.m., Dan Harvey wrote
Hi Percy,
If you have some Samza specific questions, can post to this mailing list.
We would try to help.
Thanks,
Fang, Yan
yanfang...@gmail.com
On Sat, May 2, 2015 at 9:40 AM, Percy Wegmann percy.wegm...@evariant.com
wrote:
Hello,
Apologies if this isn’t the right list, but I’m looking
is not int?
2. add warning/info messages for this setting. Because if this is set too
low accidently, at least we have a place to check.
- Yan Fang
On April 30, 2015, 11:21 p.m., Naveen Somasundaram wrote:
---
This is an automatically
On April 29, 2015, 7:30 p.m., Yan Fang wrote:
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java,
line 149
https://reviews.apache.org/r/32147/diff/7/?file=922534#file922534line149
Will this bootstrappedStreamSet overflow or take
/SamzaContainer.scala
https://reviews.apache.org/r/33674/#comment132539
This config is not used anymore. remove it?
- Yan Fang
On April 29, 2015, 1:17 p.m., Tommy Becker wrote:
---
This is an automatically generated e-mail. To reply, visit
be overwritten when we produce the new config?
samza-core/src/main/scala/org/apache/samza/util/Util.scala
https://reviews.apache.org/r/32147/#comment132577
typo: jobd - job; format is of - format of
- Yan Fang
On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote
://reviews.apache.org/r/33503/#comment131892
name to listKafkaTopics?
gradle/wrapper/gradle-wrapper.properties
https://reviews.apache.org/r/33503/#comment131895
downloading binary is sufficient, which is smaller in terms of the size.
- Yan Fang
On April 24, 2015, 2:40 a.m., Ken Gidley wrote
this limitation. So I think you have to deal
with this situation in the processing side.
Thanks,
Fang, Yan
yanfang...@gmail.com
On Wed, Apr 8, 2015 at 11:05 AM, Yan Fang yanfang...@gmail.com wrote:
-- Hi Bart,
If KV values are immediately committed to the changelog when you do a
db.put are output
Hi Naveen,
Send email to dev-subscr...@samza.apache.org for subscribing.
Thanks,
Fang, Yan
yanfang...@gmail.com
On Thu, Apr 23, 2015 at 5:42 AM, Naveen k navin.k...@gmail.com wrote:
Hi All,
Am very much interested to be part of Samza project. Please add me to the
mail list.
Regards,
wrote:
Hi guys,
I have some time to contribute to the project. Except am still relatively
new to Scala! Any Java APIs i can write ? Most JIRAs seem Scala centric...
Thanks
Asif
On 23 Apr 2015 19:44, Yan Fang yanfang...@gmail.com wrote:
Hi Naveen,
Send email to dev-subscr
accordingly
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala
https://reviews.apache.org/r/33409/#comment131808
using containerId seems better
- Yan Fang
On April 21, 2015, 8:57 p.m., Benjamin Fradet wrote
Hi Jeremy,
Currently Samza does not support accepting from command. After SAMZA-655
https://issues.apache.org/jira/browse/SAMZA-655 and SAMZA-348
https://issues.apache.org/jira/browse/SAMZA-348 , you will have other two
options - environment variable and stream.
*Then How to pass argument from
Hi Jeremy,
Naveen is correct. You can also pass the argument through --config. I
overlooked it. Sorry for the confusion.
Cheers,
Fang, Yan
yanfang...@gmail.com
On Tue, Apr 21, 2015 at 2:10 PM, Naveen S navg...@gmail.com wrote:
Hey Jeremy,
Sorry, it's probably because it's
. I
don't
think that Samza support this now. And the only work around is Yan's
solution which requires coordination between two jobs.
-Yi
On Thu, Apr 16, 2015 at 11:21 AM, Yan Fang yanfang...@gmail.com
wrote:
you are able to call coordinator.shutdown to shut
Hi Jeremy,
If my understanding is correct, whenever you add a new rule, you want to
apply this rule to the historical data. Right?
If you do not care about duplication, you can create a new task that
contains existing rules and new rules. Configure bootstrap. This will apply
all the rules from
think there is a way to do
that?
On Wed, Apr 15, 2015 at 2:56 PM, Yan Fang yanfang...@gmail.com wrote:
Hi Jeremy,
If my understanding is correct, whenever you add a new rule, you want to
apply this rule to the historical data. Right?
If you do not care about duplication, you can
Hi Yuanchi,
There is no out-of-box unit tests provided by Samza. But there are some
ways:
1) If you only want to test the logic in the Task class, normal unit tests
will work. You can create a unit test that tests init(), process(), etc.
2) Create mock systems by implementing SystemAdmin,
to hear that you may be working together.
Lewis
On Fri, Apr 10, 2015 at 11:15 AM, Yan Fang yanfang...@gmail.com wrote:
Hi Lewis,
Thanks for the reminder. I am willing to be Renato's mentor and have
been
in touch with him. Have scrolled the Wish to Mentor button in his
proposal
For Samza Committers who want to be the mentors.
Fang, Yan
yanfang...@gmail.com
-- Forwarded message --
From: Ulrich Stärk u...@apache.org
Date: Fri, Mar 6, 2015 at 11:32 AM
Subject: Google Summer of Code 2015 Mentor Registration
To: ment...@community.apache.org
Dear PMCs,
I'm
---
tested locally. Checked the metrics.
Thanks,
Yan Fang
this group, this is the replay of that first input and it should be
re-emitted to the output stream.
-Bart
-Original Message-
From: Yan Fang [mailto:yanfang...@gmail.com]
Sent: Tuesday, April 7, 2015 4:12 PM
To: dev@samza.apache.org
Subject: Re: consistency between input, output
Hi Bart,
In terms of your assumption,
* Ts = To , this is correction. The code backups this assumption is here:
in RunLoop
https://github.com/apache/samza/blob/master/samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala
,
the commit is called after each process and window methods.
committers/non-committers
to do validation as well.
Cheers,
Chris
On Wed, Mar 25, 2015 at 11:20 AM, Yan Fang yanfang...@gmail.com
wrote:
Hi Chris,
Opps, signed it with another key. Now updated all files in
http://people.apache.org/~yanfang/samza-0.9.0-rc0
locally. Checked the metrics.
Thanks,
Yan Fang
/SamzaContainer.scala
5416dd6
samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala
d18d4c4
Diff: https://reviews.apache.org/r/32528/diff/
Testing
---
tested locally. Checked the metrics.
Thanks,
Yan Fang
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32155/#review76944
---
On March 17, 2015, 9:37 a.m., Yan Fang wrote
/Log4jSystemConfig.java
https://reviews.apache.org/r/32006/#comment123943
what is the reason of getting rid of the system serde?
- Yan Fang
On March 13, 2015, 12:57 a.m., Chris Riccomini wrote:
---
This is an automatically generated e-mail
What about SAMZA-448 ?
Agreed on the release vote date. I think it is a good idea to have a
release before the ApacheCon (April 16-3-17), which can help spread the
words.
Cheers,
Fang, Yan
yanfang...@gmail.com
+1 (206) 849-4108
On Tue, Mar 3, 2015 at 8:01 AM, Chris Riccomini
, Mar 3, 2015 at 11:26 AM, Yan Fang yanfang...@gmail.com wrote:
What about SAMZA-448 ?
Agreed on the release vote date. I think it is a good idea to have a
release before the ApacheCon (April 16-3-17), which can help spread the
words.
Cheers,
Fang, Yan
yanfang...@gmail.com
+1 (206
On Feb. 27, 2015, 7:40 p.m., Yan Fang wrote:
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemConsumer.scala,
line 151
https://reviews.apache.org/r/31520/diff/3/?file=880457#file880457line151
1. besides abdicate, refreshBrokers is also called
it from droopedTopicAndPartitions. release the lock.
* thread 1 goes into the synchronized part, it will still create broker
for topic t1 because val head already has the value.
- Yan Fang
On Feb. 27, 2015, 5:59 p.m., Chris Riccomini wrote
/#comment120763
this is just for draft, right? :)
- Yan Fang
On Feb. 25, 2015, 12:54 a.m., Chris Riccomini wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31392
/TestLoggingEventStringSerde.java
PRE-CREATION
Diff: https://reviews.apache.org/r/31034/diff/
Testing
---
Thanks,
Yan Fang
Is this helpful?
http://samza.apache.org/learn/tutorials/latest/deploy-samza-to-CDH.html
Thanks,
Fang, Yan
yanfang...@gmail.com
+1 (206) 849-4108
On Tue, Feb 17, 2015 at 11:46 AM, Naveen Madhire vmadh...@umail.iu.edu
wrote:
Hi,
Is there a process document which shows how to install Samza
101 - 169 of 169 matches
Mail list logo
