Re: [VOTE] Apache Samza 0.13.0 RC6

+1 (binding) On Wed, Jun 7, 2017 at 8:55 AM, Yi Pan wrote: > +1 (binding) > build and ran all local integration tests on Linux. > > On Tue, Jun 6, 2017 at 4:01 PM, Boris S wrote: > > > +1 (non-binding) > > build and tested on Linux (with python 2.7; 2.4 and 3.5 - didn't work) > > > > On Tue, Ju

[REPORT] Samza - July 2016

## Description: - Apache Samza is a stream processing framework built on top of Apache Hadoop YARN and Apache Kafka. ## Issues: There are no issues requiring board attention at this time. ## Activity: - Good coverage of Apache Samza at Hadoop summit. Several talks about it. - Meetup at L

Re: [VOTE] Samza 0.10.0 Release Candidate 2

+1 binding On Wed, Dec 9, 2015 at 7:16 PM, Boris Shkolnik wrote: > I've opened a ticket to track this rocksdb:test failure: > >1. SAMZA-836 > > > On Wed, Dec 9, 2015 at 7:11 PM, Boris Shkolnik wrote: > > > +1 (non-binding). > > One note. I f

Fwd: [REPORT] Apache Samza

Hey all, I've submitted the Samza October board report filing. Cheers, Chris -- Forwarded message -- From: Chris Riccomini Date: Fri, Oct 9, 2015 at 7:38 AM Subject: [REPORT] Apache Samza To: "bo...@apache.org" Attachment AU: Report from the Apache Samza

Re: [Discuss/Vote] upgrade to Yarn 2.6.0

+1 on my end On Monday, August 17, 2015, Yi Pan wrote: > Hi, Yan, > > Thanks for rolling the ball! > > +1 from me to upgrade the minimum supported version to YARN 2.6, assuming > that we are going to fix SAMZA-750 together. > > On Mon, Aug 17, 2015 at 4:41 PM, Yan Fang > wrote: > > > Hi guys, >

KIP-28 kafka processor

Hey all, I wanted to call attention to KIP-28: https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client This is the result of the last conversation that we had about samza's future direction. It would be good to have the samza community involved in this. Cheers, Chris

Re: Thoughts and obesrvations on Samza

That was meant to be "thread" not "threat". lol. :) On Sun, Jul 12, 2015 at 5:54 PM, Chris Riccomini wrote: > Hey all, > > I want to start by saying that I'm absolutely thrilled to be a part of > this community. The amount of level-headed, thoughtful, educ

Re: Thoughts and obesrvations on Samza

Hey all, I want to start by saying that I'm absolutely thrilled to be a part of this community. The amount of level-headed, thoughtful, educated discussion that's gone on over the past ~10 days is overwhelming. Wonderful. It seems like discussion is waning a bit, and we've reached some conclusion

Re: [VOTE] Apache Samza 0.9.1 RC1

+1 Verified MD5, and asc signature. Build locally, and all tests pass. Cheers, Chris On Tue, Jun 30, 2015 at 1:26 PM, Milinda Pathirage wrote: > +1 (non-binding) > > Verified signature. Tested locally using ./bin/check-all.sh. > > Thanks > Milinda > > On Tue, Jun 30, 2015 at 2:10 AM, Yan Fang

Thoughts and obesrvations on Samza

Hey all, I have had some discussions with Samza engineers at LinkedIn and Confluent and we came up with a few observations and would like to propose some changes. We've observed some things that I want to call out about Samza's design, and I'd like to propose some changes. * Samza is dependent u

Re: Measuring Samza Job Throughput

ghout metrics after all. On Tuesday, June 16, 2015, Chris Riccomini wrote: > Hey Milinda, > > Specifically, for bytes/sec, you might want to look at serde metrics. I > believe the serde manager tracks bytes serialized and deserialized per > second. The consumers and producers also do

Re: Measuring Samza Job Throughput

Hey Milinda, Specifically, for bytes/sec, you might want to look at serde metrics. I believe the serde manager tracks bytes serialized and deserialized per second. The consumers and producers also do this for Kafka, but on a more granular basis. If you want container-level throughput, serde manage

Powered by page update

Hey all, I'm seeing a lot of new faces on the mailing list, which is really awesome. I want to invite you all to add yourselves to our Powered by page: https://cwiki.apache.org/confluence/display/SAMZA/Powered+By The Apache wiki is pretty locked down due to spam. If you'd like to send me a link

Re: improving hello-samza / testing

Hey Tim, This is a really good discussion to have. The testing that I've seen with Samza falls into two categories: 1. Instantiate your StreamTask, and mock all params in the process()/init() methods. 2. A mini-ontegration test that starts ZooKeeper, and Kafka, and feeds messages into a topic, an

Re: [DISCUSS] Samza 0.9.1 release

+1 Here. On Tue, Jun 16, 2015 at 12:01 PM, Guozhang Wang wrote: > Cool. I will start a voting process soon. > > On Tue, Jun 16, 2015 at 11:55 AM, Chinmay Soman > > wrote: > > > +1 > > > > On Tue, Jun 16, 2015 at 11:17 AM, Navina Ramesh < > > nram...@linkedin.com.invalid> wrote: > > > > > +1 for

Re: How to configure the Resource Manager endpoint for YARN?

directory path. Doh! > > Sent from my iPhone > > > On Apr 15, 2015, at 9:41 AM, Roger Hoover > wrote: > > > > I'll try that. Thanks, Chris. > > > >> On Wed, Apr 15, 2015 at 9:37 AM, Chris Riccomini > wrote: > >> Hey Roger, > >

Re: Maximum number of jobs

Hey Jeremy, Samza will be fine, but at this scale you need to start worrying about Kafka and YARN. 1 million jobs will likely start to put pressure on YARN's RM due to memory usage and CPU usage for the scheduler. With 1 million jobs, assuming 1 container each, you'll have over 1 million connectio

Re: How to configure the Resource Manager endpoint for YARN?

Hey Roger, Not sure if this makes a difference, but have you tried using: export YARN_CONF_DIR=... Instead? This is what we use. Cheers, Chris On Wed, Apr 15, 2015 at 9:33 AM, Roger Hoover wrote: > Hi, > > I'm trying to deploy a job to a small YARN cluster. How do tell the > launcher scri

Re: Updating samza-sql branch to Java 1.7

@Yi, are you going to merge master into the samza-sql branch, or should I? On Tue, Apr 14, 2015 at 2:02 PM, Yi Pan wrote: > Hi, Milinda, > > Jacob already committed a change to remove Java 1.6 support in SAMZA-646. I > think that it would be fine to move samza-sql branch to Java 1.7. > > Regards

Re: Review Request 33146: Adding a new KV store contract: BatchingKeyValueStore

/TestKeyValueStores.scala <https://reviews.apache.org/r/33146/#comment129874> Prefer not to introduce a dependency on google common. - Chris Riccomini On April 13, 2015, 9:24 p.m., Mohamed Mahmoud (El-Geish) wrote: > > ---

Re: Review Request 33146: Adding a new KV store contract: BatchingKeyValueStore

eviews.apache.org/r/33146/#comment129787> Do we need a deleteAlls metric as well? - Chris Riccomini On April 13, 2015, 9:24 p.m., Mohamed Mahmoud (El-Geish) wrote: > > --- > This is an automatically generated e

Re: Dealing with partitioning mismatches between bootstrap and input streams

Hey Tommy, Your summary sounds pretty accurate. One other way, which requires no change to Samza, would be to repartition the input topic properly for each task. This is kind of hacky, though. (2) is the ideal solution. It is a bit of work, but it might not be so bad. I think most of the changes

Re: Producer performance in 0.9.0

Hey Gian, Hmm, this is strange. We ran some tests, and found that the new producer to be faster than the old producer default (sync), and almost as fast as the old producer's async producer. Could you paste all of your configs? Cheers, Chris On Tue, Apr 7, 2015 at 10:40 AM, Gian Merlino wrote:

Re: Review Request 32147: SAMZA-465

omment127747> Does it make sense to just have JobCoordinator.apply default to new MetricsRegistryMap? It seems like we had to make a bunch of changes rather than just set the default in the constructor, and leave the code alone everywhere. - Chris Riccomini On Apr

Re: Review Request 32147: SAMZA-465

to parameterize this (the {}). SLF4J's Logger takes an error(String, Throwable) .. http://www.slf4j.org/apidocs/org/slf4j/Logger.html - Chris Riccomini On April 1, 2015, 9:02 p.m., Naveen Somasundaram wrote: > > --- > Th

Re: Store changelog

Hey Dan, I think you might have a misunderstanding in how changelogs work with Samza. Suppose you have a job with two tasks, and a single kv-store is configured with a changelog attached. The changelog, in Kafka, will have two partitions. Each task will use one partition of the changelog topic. Yo

Re: [VOTE] Apache Samza 0.9.0 RC0

SAMZA-624 > > > ** Will publish the website after mirrors pick up the 0.9.0 release > > > ** In terms of the blog, seem not have the access ? > > > > > > Thanks, > > > > > > Fang, Yan > > > yanfang...@gmail.com > > > > > &g

Re: Samza on Yarn

at. > > > > I see that the 2 nodes are showing up on rm. We are testing redundancy > now. > > > > - Shekar > > > > > > > > On Fri, Mar 13, 2015 at 1:20 PM, Chris Riccomini > > wrote: > > > >> Hey Shekar, > >> > >>

Re: [VOTE] Apache Samza 0.9.0 RC0

n terms of the blog, seem not have the access ? > > Thanks, > > Fang, Yan > yanfang...@gmail.com > > On Tue, Mar 31, 2015 at 1:36 PM, Chris Riccomini > wrote: > >> Hey Yan/Jakob, >> >> Awesome, thanks! Yan, feel free to finish up the release. :) Very

Re: Samza closing and re-opening kafka connection rapidly, cannot consume or produce, no useful logs

ed: 0 > Active Nodes: 1 > Decommissioned Nodes: 0 > Lost Nodes: 0 > Unhealthy Nodes: 0 > Rebooted Nodes: 0 > > > Again, much obliged for your response. > > Andrew Sannier > > > > On 3/31/15, 3:54 PM, "Chris Riccomini" wrote: > > >Hey Andrew

Re: Samza closing and re-opening kafka connection rapidly, cannot consume or produce, no useful logs

Hey Andrew, I'm wondering if your YARN cluster doesn't have enough memory to fit both the AM and its containers. The fact that the AM UI shows no running containers is suspicious. Can you check these four settings in your YARN RM's UI: Memory Used Memory Total Memory Reserved VCores Used

Re: [VOTE] Apache Samza 0.9.0 RC0

; distribution, publicity, etc. > -jg > > On 31 March 2015 at 12:44, Chris Riccomini wrote: > > Hey Yan, > > > > Let's confirm with Jakob. I *think* we don't need any intervention from > > Apache. We should be able to move forward with the release. @Jak

Re: [VOTE] Apache Samza 0.9.0 RC0

write 0.8.0 blog post? > > Thanks, > Fang, Yan > yanfang...@gmail.com > > On Tue, Mar 31, 2015 at 9:21 AM, Ash W Matheson > wrote: > >> Of say yes, is been a few days with little traffic on the topic. >> On Mar 31, 2015 9:18 AM, "Chris Riccomini"

Re: Kafka Question

hekar > > On Tue, Mar 31, 2015 at 9:19 AM, Chris Riccomini > wrote: > > > Hey Shekar, > > > > Are you running with 0.8.0 when you run these tests? If so, there are > some > > known issues where a Samza consumer can get stuck when brokers > disappeared. &g

Re: Samza install quick guide

Hey Jordi, Thanks for this! I've copied your pastebin script, and put it on: https://issues.apache.org/jira/browse/SAMZA-189 So that we don't lose track of it. Cheers, Chris On Tue, Mar 31, 2015 at 3:48 AM, Jordi Blasi Uribarri wrote: > Hi, > > I am not sure what is the correct way of doin

Re: Kafka Question

Hey Shekar, Are you running with 0.8.0 when you run these tests? If so, there are some known issues where a Samza consumer can get stuck when brokers disappeared. All known issues have been resolved in the 0.9.0 release, but they exist in the 0.8.0 release. Cheers, Chris On Mon, Mar 30, 2015 at

Re: [VOTE] Apache Samza 0.9.0 RC0

Hey all, Is the vote done? Cheers, Chris On Mon, Mar 30, 2015 at 2:10 PM, Chris Riccomini wrote: > +1 > > 1. Validated hello-samza works with 0.9.0 Maven binaries. > 2. Validated release-0.9.0-rc0 tag exists and has correct checksums. > 3. Validated source release tarball

Re: [VOTE] Apache Samza 0.9.0 RC0

15 at 1:36 AM, Yan Fang > wrote: > > > > > >> Hi Chris and Jakob, > > >> > > >> Sure. Let's do the voting until Monday. Hope more guys have time to > try > > and > > >> validate the 0.9.0 version. > > >> > >

Re: [VOTE] Apache Samza 0.9.0 RC0

all, > >> > > >> > I created a basic job that does a join from local state with Samza > 0.9.0 > >> ( > >> > https://github.com/Quantiply/rico-playground/tree/master/join/samza). > >> So > >> > far so good. I hoping to get s

Re: java.lang.NoClassDefFoundError on Yarn job

Hey Jordi, "No FileSystem for scheme: http" means that your YARN NMs aren't configured with the http filesystem in your core-site.xml: fs.http.impl org.apache.samza.util.hadoop.HttpFileSystem As documented here: http://samza.apache.org/learn/tutorials/0.8/run-in-multi-

Re: [VOTE] Apache Samza 0.9.0 RC0

Opps, signed it with another key. Now updated all files in > http://people.apache.org/~yanfang/samza-0.9.0-rc0/ . Verified. Sorry for > the inconvenience. > > Cheers, > > Fang, Yan > yanfang...@gmail.com > +1 (206) 849-4108 > > On Wed, Mar 25, 2015 at 10:58 AM, Chris

Re: Error running integration tests

Hey Roger, Are you able to run this command? ssh localhost This is effectively what Zopkio is doing. Wondering if you need to enable SSH on your laptop? I have "remote login" enabled on my OSX laptop. Cheers, Chris On Wed, Mar 25, 2015 at 4:29 PM, Roger Hoover wrote: > Do I need to bring u

Re: [VOTE] Apache Samza 0.9.0 RC0

Hey Yan, Were you able to validate the source tarball? I ran: $ gpg --keyserver pgpkeys.mit.edu --recv-key CAC06239EA00BA80 gpg: requesting key EA00BA80 from hkp server pgpkeys.mit.edu gpg: key EA00BA80: public key "Yan Fang (CODE SIGNING KEY) < yanf...@apache.org>" imported gpg: Total number pro

Re: Log error deploying on YARN [Samza 0.8.0]

Cool, website should be fixed when we publish for the 0.9.0 release. On Tue, Mar 24, 2015 at 2:32 PM, Roger Hoover wrote: > Ah, yes. That's it! Thanks, Chris. > > On Tue, Mar 24, 2015 at 2:30 PM, Chris Riccomini > wrote: > > > Hey Roger, > > >

Re: Log error deploying on YARN [Samza 0.8.0]

Hey Roger, You're likely hitting this issue: https://issues.apache.org/jira/browse/SAMZA-456 Can you have a look and see if that's the problem? We missed some JARs that need to be put in to the YARN NM classpath. Cheers, Chris On Tue, Mar 24, 2015 at 2:22 PM, Roger Hoover wrote: > Hi all,

Re: Submitting yarn job with custom properties

FYI- I haven't actually tried this, but I think it should work. If __package doesn't work, try without the __package/ prefix (i.e. fopen('yourfile.txt')). On Tue, Mar 24, 2015 at 1:26 PM, Chris Riccomini wrote: > Cool, that should work. In YARN, the .tgz file is unzippe

Re: Submitting yarn job with custom properties

gt; - Shekar > > On Tue, Mar 24, 2015 at 11:09 AM, Chris Riccomini > wrote: > > > Hey Shekar, > > > > So you have two properties files? One is the Samza job file, and the > other > > is one with the format you described above? > > > > Several

Samza 0.9.0

Hey all, We've cut the 0.9.0 release for Samza. Yan is going to run the release. Expect a [VOTE] email shortly. Cheers, Chris

Re: Submitting yarn job with custom properties

Hey Shekar, So you have two properties files? One is the Samza job file, and the other is one with the format you described above? Several options: 1. Move these properties into the main Samza job file. You'll then have access to them via the Config object. 2. Put the property file into your .tg

Re: Review Request 32407: SAMZA-571: add suppression interface for uncaught exceptions

rg/r/32407/#comment125580> Nit: white space. samza-core/src/main/scala/org/apache/samza/system/SystemConsumers.scala <https://reviews.apache.org/r/32407/#comment125581> Nit: white space - Chris Riccomini On Marc

Re: Message processing stuck after invalid JSON [BUG?!] - Version 0.8.x

Hey all, I've reproduced and documented the bug here: https://issues.apache.org/jira/browse/SAMZA-608 Cheers, Chris On Thu, Mar 19, 2015 at 9:08 AM, Chris Riccomini wrote: > Hey Michael, > > Hmm. I checked the SystemConsumers code, and nothing jumped out at me as > broken

Re: How to pass quoted string in the configuration

Hey Jae, If you're using the PropertiesConfigFactory (the default), then it's just a Java Properties object. You should be able to escape it. I'm actually not even sure if quotes are a problem in Java Properties objects. I would think that they wouldn't be. The second '=' sign might be, though. I

Re: Review Request 32147: SAMZA-465

erformance.properties <https://reviews.apache.org/r/32147/#comment124990> Does this matter? Should coordinator stream always start from the oldest offset? samza-test/src/main/python/samza_failure_testing.py <https://reviews.apache.org/r/32147/#comment124869> Conventio

Re: Message processing stuck after invalid JSON [BUG?!] - Version 0.8.x

Hey Michael, Hmm. I checked the SystemConsumers code, and nothing jumped out at me as broken. Could you paste your logs somewhere (pastebin/gist) with DEBUG-level logging enabled? Cheers, Chris On Thu, Mar 19, 2015 at 6:52 AM, Michael Strobl < michael.str...@gameforge.com> wrote: > Hey there, >

Re: Kafka topic naming conventions

ary to do in case we're > doing some kind of joins. > > > On Wed, Mar 18, 2015 at 5:59 PM, Jakob Homan wrote: > > > On 18 March 2015 at 17:48, Chris Riccomini > wrote: > > > One thing I haven't seen, but might be relevant, is including partition > >

Re: Kafka topic naming conventions

the operation. Agree that it's just best effort, and kind of hacky. Was just a thought. I haven't seen anyone actually do this. Cheers, Chris On Wed, Mar 18, 2015 at 5:59 PM, Jakob Homan wrote: > On 18 March 2015 at 17:48, Chris Riccomini wrote: > > One thing I haven't s

Re: Kafka topic naming conventions

Hey Roger, We haven't thought about this in great detail. People do all kinds of wacky things in practice. We have some that are like, "AdViewsByMemberId". There are various permutations of that. One thing I haven't seen, but might be relevant, is including partition counts in the topic. If you'r

Re: Example Samza job using Confluent Platform

Hey Roger, This is awesome! I've added it to the ecosystem wiki: https://cwiki.apache.org/confluence/display/SAMZA/Ecosystem Cheers, Chris On Wed, Mar 18, 2015 at 10:59 AM, Roger Hoover wrote: > Hi all, > > In case others find this useful I created a simple Samza job that uses Avro > + the

Re: Hello Samza: yarn symlink problem

Hey George, Could you paste the log in plain-text format, rather than image? Apache is stripping out the image. Based on your comment, I wonder if this is the problem: http://superuser.com/questions/584146/why-is-this-archive-failing-to-create-symlinks What file system are you using? Are you u

Re: NoSuchMethodError

--config-path=file://$PWD/config/hello-world.properties > > The way you say it works. Now I have another different problem that I will > have to check before asking. > > Thanks for your help. > > Cheers, > > Jordi > > > -Mensaje original- > De: Chris Ricco

Re: NoSuchMethodError

Hey Jordi, PropertiesConfigFactory is located in this package: org.apache.samza.config.factories You have the package set to samza.config.factories. You'll need to set it to: org.apache.samza.config.factories.PropertiesConfigFactory Curious where you're getting that value from? We haven't had

Review Request 32127: SAMZA-586

4f15002325bc0154991f9a35312e903d15ef81e7 samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemFactory.scala 5f65144146109418b4ec2ac40b3a69cb97f26224 Diff: https://reviews.apache.org/r/32127/diff/ Testing --- Thanks, Chris Riccomini

Re: NoSuchMethodError

Hey Jordi, The stack trace you've pasted suggests that you're missing Scala in the classpath, or have a different version of Scala in the classpath than what Samza was compiled with. You should not be manually assembling the dependencies for your job. Your build system should be doing this for yo

Re: Review Request 32052: SAMZA-592

amza-kafka/src/test/scala/org/apache/samza/utils/TestKafkaUtil.scala PRE-CREATION samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala a8b724bf781003142e455fdf1fed2f13d6c18353 Diff: https://reviews.apache.org/r/32052/diff/ Testing --- Thanks, Chris Riccomini

Re: Review Request 32052: SAMZA-592

samza/utils/TestKafkaUtil.scala PRE-CREATION samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala a8b724bf781003142e455fdf1fed2f13d6c18353 Diff: https://reviews.apache.org/r/32052/diff/ Testing --- Thanks, Chris Riccomini

Re: Review Request 32052: SAMZA-592

wException method will throw an exception, which will be caught in KafkaSystemAdmin.getSystemStreamMetadata. The retryBackoff loop will then catch the exception, and restart from scratch. - Chris Riccomini On March 13, 2015, 7:56 p.m., Chris Riccomini wrote: > >

Re: Review Request 32006: SAMZA-597

samza-log4j/src/test/java/org/apache/samza/logging/log4j/TestStreamAppender.java 3e4ddc9c72868e22f993f60015224cd3a153266c Diff: https://reviews.apache.org/r/32006/diff/ Testing --- Thanks, Chris Riccomini

Re: Review Request 32006: SAMZA-597

f/1/?file=892563#file892563line53> > > > > nit: Serde Done. - Chris --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32006/#review76381 -

Re: Samza on Yarn

it the Samza documentation to reflect any > changes. > > - Shekar > > On Thu, Mar 12, 2015 at 5:19 PM, Chris Riccomini > wrote: > > > Hey Shekar, > > > > Yes, this is definitely a classpath issue. The pastebin you sent does not > > include any of t

Review Request 32052: SAMZA-592

/test/integration/TestStatefulTask.scala a8b724bf781003142e455fdf1fed2f13d6c18353 Diff: https://reviews.apache.org/r/32052/diff/ Testing --- Thanks, Chris Riccomini

Re: Review Request 32006: SAMZA-597

ms.%s.streams.%s.samza.msg.serde (stream-level) serde config. - Chris --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32006/#review76396 ------

SAMZA-576 and SAMZA-597 Reviews

Hey all, I could use reviews for: https://issues.apache.org/jira/browse/SAMZA-576 https://issues.apache.org/jira/browse/SAMZA-597 Thanks! :) Cheers, Chris

Review Request 32006: SAMZA-597

3e4ddc9c72868e22f993f60015224cd3a153266c Diff: https://reviews.apache.org/r/32006/diff/ Testing --- Thanks, Chris Riccomini

Re: Samza on Yarn

ssue with yarn but not sure how to resolve as yet. > > > - Shekar > > On Thu, Mar 12, 2015 at 1:44 PM, Shekar Tippur wrote: > > > Chris - Here it is. > > > > http://pastebin.com/c3e21Hzf > > > > - Shekar > > > > On Thu, Mar 12, 2015 at 10:58

Re: Samza on Yarn

This is the line that I'm interested in: STARTUP_MSG: classpath On Thu, Mar 12, 2015 at 10:55 AM, Chris Riccomini wrote: > Hey Shekar, > > Could you paste the full log on pastebin? It really seems like something's > missing from the classpath. If samza-yarn is th

Re: Samza on Yarn

tarting up, and dumps the full classpath? Cheers, Chris On Thu, Mar 12, 2015 at 10:17 AM, Shekar Tippur wrote: > I think all these jars are in place (Under > $HADOOP_YARN_HOME/share/hadoop/hdfs/lib) > > - Shekar > > On Thu, Mar 12, 2015 at 9:36 AM, Chris Riccomini > wrote

Re: Samza on Yarn

Hey Shekar, You need that samza-yarn file on your RM/NM's classpath, along with scala. We missed this in the docs, and are tracking the issue here: https://issues.apache.org/jira/browse/SAMZA-456 You'll also need samza-core in the classpath, based on the discussion on SAMZA-456. Sorry about th

Re: Review Request 31881: SAMZA-506: Shutdown RunLoop on SIGTERM.

tile. The hook will be executed from another thread than the RunLoop, which is reading the variable. - Chris Riccomini On March 10, 2015, 6:55 a.m., Ewen Cheslack-Postava wrote: > > --- > This is an automatically generated e

Re: Extraneous wiki page

uot;Milinda Pathirage" > wrote: > > > Hi Chris, > > > > There is a another page which is not related Samza. > > > > https://cwiki.apache.org/confluence/display/SAMZA/asdf > > > > Thanks > > Milinda > > > > On Tue, Mar 10, 2015 at 8:16

Re: Review Request 31910: SAMZA-505

za-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala 84cf6db3a1b035c639a6dec30fe9ee997b282a80 samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala PRE-CREATION Diff: https://reviews.apache.org/r/31910/diff/ Testing --- Thanks, Chris Riccomini

Re: Extraneous wiki page

Lol, no worries. Fixed. :) Wonder if we should re-evaluate the perms on the wiki to either allow deletions, or not allow page adds. On Tue, Mar 10, 2015 at 4:04 PM, Sean Mackrory wrote: > All, > > I intended a wiki page I created to be in the Bigtop space, however I > inadvertently created it i

Re: Review Request 31881: SAMZA-506: Shutdown RunLoop on SIGTERM.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31881/#review75985 --- Ship it! Looks good. Thanks! - Chris Riccomini On March 10

Review Request 31910: SAMZA-505

84cf6db3a1b035c639a6dec30fe9ee997b282a80 samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala PRE-CREATION Diff: https://reviews.apache.org/r/31910/diff/ Testing --- Thanks, Chris Riccomini

Review Request 31909: SAMZA-590

scala f768263961d395c5d21a857a7581e0c472bbe547 samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestBrokerProxy.scala d559d8b276007ebb99b37b2ebf42c1b415e3fbce Diff: https://reviews.apache.org/r/31909/diff/ Testing --- Thanks, Chris Riccomini

Re: Review Request 31881: SAMZA-506: Shutdown RunLoop on SIGTERM.

st, we can just override these two methods. Saves us the null checks, and the params in the constructor. - Chris Riccomini On March 10, 2015, 6:01 p.m., Ewen Cheslack-Postava wrote: > > --- > This is an automatically generated e

Re: Java Opts and Max Heap

Hey Jordan, There are two settings you need to care about when changing memory. One is yarn.container.memory.mb. The other is the task.opts setting, which will allow you to specify a custom -Xmx parameter. If you set the yarn.container.memory.mb to 4000 (4GB), as you suggest, the JAVA_OPTS check

Re: Storing sensitive data in the Config

Hey Tommy, Yea, this has come up a few times. We don't currently have an answer for it. The simplest thing to do would be to have a prefix. Any config with the prefix could be stripped from the AM and logs. Another possibility is to store the configs in an encrypted way, and have the code decrypt

Re: Review Request 31392: SAMZA-555

iff: https://reviews.apache.org/r/31392/diff/ Testing --- Thanks, Chris Riccomini

Re: Review Request 31718: Samza 465 [Draft]

t; Nit: white space. samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala <https://reviews.apache.org/r/31718/#comment122592> Idiomatic scala, please. I think something like: val combinedOffsets = containerModel.getTasks.values.flatMap(_.getCheckp

Re: How to get Container ID from configuration

ae Hyeon wrote: > ENV_CONTAINER_ID does not look existing in 0.8.0. I will try > ENV_CONTAINER_NAME. > > On Thu, Mar 5, 2015 at 5:09 PM, Chris Riccomini > wrote: > >> Hey Jae, >> >> Also, this should work: >> >> System.getenv(ShellCommandConfig.ENV_CONTAIN

Re: [DISCUSS] Samza 0.9.0 release

> On Tue, Mar 3, 2015 at 5:03 PM, Yan Fang wrote: > > > Yes, it sounds good. I asked was because I saw the "patch available" > > status. :) > > > > Cheers, > > > > Fang, Yan > > yanfang...@gmail.com > > +1 (206) 849-4108 > > &

Re: How to get Container ID from configuration

Hey Jae, Also, this should work: System.getenv(ShellCommandConfig.ENV_CONTAINER_ID).toInt This is actually how SamzaContainer gets its containerId variable. BTW, I've opened: https://issues.apache.org/jira/browse/SAMZA-586 To track exposing SamzaContainerContext. Cheers, Chris

Re: How to get Container ID from configuration

Hey Jae, We expose this in SamzaContainerContext. Unfortunately, this class never gets exposed to the StreamTask. The plan was to expose this class via TaskContext, but we haven't done that yet. Have you tried using System.getProperty("samza.container.id")? Cheers, Chris On Thu, Mar 5, 2015 at

Re: [DISCUSS] Samza 0.9.0 release

pread the > words. > > Cheers, > > Fang, Yan > yanfang...@gmail.com > +1 (206) 849-4108 > > On Tue, Mar 3, 2015 at 8:01 AM, Chris Riccomini > wrote: > > > Hey all, > > > > We haven't done a release in a couple of months, and I think master is

Re: Stream-Stream joins - restricting the size of the KV store

Hey Kishore, Right now, that has to be done by the application code. Our plan is to support a TTL in RocksDB once that feature is available to us in the Java JNI bindings (should be soon). In the meantime, you'll have to implement the WindowableTask.window() method, and call store.all() to periodi

[DISCUSS] Samza 0.9.0 release

Hey all, We haven't done a release in a couple of months, and I think master is getting to the point where it's got enough work to warrant a release. We've (LinkedIn) begun running a couple of our jobs against `master` last week, and will continue to test for stability. Here are the JIRAs that I

Re: Apache Samza Meetup - March 4 @6PM hosted at LinkedIn's campus in Mountain View CA

ting Started with > Apache Samza > * 6:45 - 7:30PM: Shekar Tippur (Intuit) – Powering Contextual Alerts > for Intuit’s Operations Center with Apache Samza > * 7:30 - 8:00PM: Chris Riccomini (LinkedIn) – Where we’re headed next: > Apache Samza Roadmap > > We plan to provide

Review Request 31649: SAMZA-584

, Chris Riccomini

Re: Stream SQL for Samza Query Language Guide and Design

Hey all, Just closing the loop on this. I've migrated the wiki to: https://cwiki.apache.org/confluence/display/SAMZA/Apache+Samza The website has been updated accordingly. Cheers, Chris On Wed, Feb 18, 2015 at 2:45 PM, Chris Riccomini wrote: > Hey all, > > I've setup a

Re: Review Request 31417: Samza 465 [Draft]

an `rbt post` with just my changes, then a second with yours? This way, I can diff between my changes, and yours, to see just what you've done. I don't want to get things tangled between SAMZA-448 and SAMZA-465. - Chris Riccomini On Feb. 25, 2015, 5:45 p.m., Naveen Somasund

  1   2   3   >