Re: [DISCUSS] Release 0.10.0

2015-07-28 Thread Roger Hoover 
Thanks, Yi.

I propose that we also include SAMZA-741 for Elasticsearch versioning
support with the new ES producer.  I think it's very close to being merged.

Roger


On Tue, Jul 28, 2015 at 10:08 PM, Yi Pan  wrote:

> Hi, all,
>
> I want to start the discussion on the release schedule for 0.10.0. There
> are a few important features that we plan to release in 0.10.0 and I want
> to start this thread s.t. we can agree on what to include in 0.10.0
> release.
>
> There are the following main features added in 0.10.0:
> - RocksDB TTL support
> - Add CoordinatorStream and disable CheckpointManager
> - Elasticsearch Producer
> - Host affinity
> And other 0.10.0 tickets:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(%22In%20Progress%22%2C%20Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.10.0
>
> I propose to cut a 0.10.0 release after we get the following issues
> resolved:
> - SAMZA-615: Migrate checkpoint from checkpoint topic to Coordinator stream
> - SAMZA-617: YARN host affinity in Samza
>
> Thoughts?
>
> Thanks!
>
> -Yi
>

Re: [DISCUSS] Release 0.10.0

2015-07-28 Thread Yan Fang 
Actually, I also want to include a few patch-available features, especially:

1. broadcast stream (SAMZA-676)
- waiting for review

2. graphite support (SAMZA-340)
3. meter and histogram (SAMZA-683)
4. utility (SAMZA-401)
- 2,3,4 belong to Luis, if he does not have time to update, since they
only need some small changes, we can edit it and get +1 from another
committer.

5. hdfs producer (SAMZA-693)
- I am reviewing.

6. upgrade yarn to 2.7.1 (SAMZA-563)
   - though I am reviewing, this ticket is negotiable if we want to put
into the 0.10.0 release. If we do not, I think, when users enable the
worker-persisting and container-persisting features, Samza will not be able
to handle it. (Some classes are only available after yarn 2.5.0 while Samza
currently only support yarn 2.4.0)

7. others: scrooge, class loader isolation, etc.
- those are waiting for reviewing too.

My opinion is that, if we can clean up all the patch-available tickets, it
will be great. Most of them have been already reviewed more than once. So I
think it should not be very time-consuming to have them in the 0.10.0
release.

What do you think?

Of course, another must-have is the bug-fix of the Stream Appender. :)

Thanks,

Fang, Yan
yanfang...@gmail.com

On Tue, Jul 28, 2015 at 10:27 PM, Roger Hoover 
wrote:

> Thanks, Yi.
>
> I propose that we also include SAMZA-741 for Elasticsearch versioning
> support with the new ES producer.  I think it's very close to being merged.
>
> Roger
>
>
> On Tue, Jul 28, 2015 at 10:08 PM, Yi Pan  wrote:
>
> > Hi, all,
> >
> > I want to start the discussion on the release schedule for 0.10.0. There
> > are a few important features that we plan to release in 0.10.0 and I want
> > to start this thread s.t. we can agree on what to include in 0.10.0
> > release.
> >
> > There are the following main features added in 0.10.0:
> > - RocksDB TTL support
> > - Add CoordinatorStream and disable CheckpointManager
> > - Elasticsearch Producer
> > - Host affinity
> > And other 0.10.0 tickets:
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(%22In%20Progress%22%2C%20Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.10.0
> >
> > I propose to cut a 0.10.0 release after we get the following issues
> > resolved:
> > - SAMZA-615: Migrate checkpoint from checkpoint topic to Coordinator
> stream
> > - SAMZA-617: YARN host affinity in Samza
> >
> > Thoughts?
> >
> > Thanks!
> >
> > -Yi
> >
>

Re: [DISCUSS] Release 0.10.0

2015-07-29 Thread Dan 
I agree SAMZA-741 for the ElasticSearch producer should be in too so we've
got a better API as part of that release.

 - Dan


On 29 July 2015 at 07:51, Yan Fang  wrote:

> Actually, I also want to include a few patch-available features,
> especially:
>
> 1. broadcast stream (SAMZA-676)
> - waiting for review
>
> 2. graphite support (SAMZA-340)
> 3. meter and histogram (SAMZA-683)
> 4. utility (SAMZA-401)
> - 2,3,4 belong to Luis, if he does not have time to update, since they
> only need some small changes, we can edit it and get +1 from another
> committer.
>
> 5. hdfs producer (SAMZA-693)
> - I am reviewing.
>
> 6. upgrade yarn to 2.7.1 (SAMZA-563)
>- though I am reviewing, this ticket is negotiable if we want to put
> into the 0.10.0 release. If we do not, I think, when users enable the
> worker-persisting and container-persisting features, Samza will not be able
> to handle it. (Some classes are only available after yarn 2.5.0 while Samza
> currently only support yarn 2.4.0)
>
> 7. others: scrooge, class loader isolation, etc.
> - those are waiting for reviewing too.
>
> My opinion is that, if we can clean up all the patch-available tickets, it
> will be great. Most of them have been already reviewed more than once. So I
> think it should not be very time-consuming to have them in the 0.10.0
> release.
>
> What do you think?
>
> Of course, another must-have is the bug-fix of the Stream Appender. :)
>
> Thanks,
>
> Fang, Yan
> yanfang...@gmail.com
>
> On Tue, Jul 28, 2015 at 10:27 PM, Roger Hoover 
> wrote:
>
> > Thanks, Yi.
> >
> > I propose that we also include SAMZA-741 for Elasticsearch versioning
> > support with the new ES producer.  I think it's very close to being
> merged.
> >
> > Roger
> >
> >
> > On Tue, Jul 28, 2015 at 10:08 PM, Yi Pan  wrote:
> >
> > > Hi, all,
> > >
> > > I want to start the discussion on the release schedule for 0.10.0.
> There
> > > are a few important features that we plan to release in 0.10.0 and I
> want
> > > to start this thread s.t. we can agree on what to include in 0.10.0
> > > release.
> > >
> > > There are the following main features added in 0.10.0:
> > > - RocksDB TTL support
> > > - Add CoordinatorStream and disable CheckpointManager
> > > - Elasticsearch Producer
> > > - Host affinity
> > > And other 0.10.0 tickets:
> > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(%22In%20Progress%22%2C%20Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.10.0
> > >
> > > I propose to cut a 0.10.0 release after we get the following issues
> > > resolved:
> > > - SAMZA-615: Migrate checkpoint from checkpoint topic to Coordinator
> > stream
> > > - SAMZA-617: YARN host affinity in Samza
> > >
> > > Thoughts?
> > >
> > > Thanks!
> > >
> > > -Yi
> > >
> >
>

Re: [DISCUSS] Release 0.10.0

2015-07-29 Thread Chinmay Soman 
I can take care of SAMZA-340, SAMZA-683  and will follow up with Luis for
SAMZA-401,2,3,4

On Wed, Jul 29, 2015 at 12:10 AM, Dan  wrote:

> I agree SAMZA-741 for the ElasticSearch producer should be in too so we've
> got a better API as part of that release.
>
>  - Dan
>
>
> On 29 July 2015 at 07:51, Yan Fang  wrote:
>
> > Actually, I also want to include a few patch-available features,
> > especially:
> >
> > 1. broadcast stream (SAMZA-676)
> > - waiting for review
> >
> > 2. graphite support (SAMZA-340)
> > 3. meter and histogram (SAMZA-683)
> > 4. utility (SAMZA-401)
> > - 2,3,4 belong to Luis, if he does not have time to update, since
> they
> > only need some small changes, we can edit it and get +1 from another
> > committer.
> >
> > 5. hdfs producer (SAMZA-693)
> > - I am reviewing.
> >
> > 6. upgrade yarn to 2.7.1 (SAMZA-563)
> >- though I am reviewing, this ticket is negotiable if we want to put
> > into the 0.10.0 release. If we do not, I think, when users enable the
> > worker-persisting and container-persisting features, Samza will not be
> able
> > to handle it. (Some classes are only available after yarn 2.5.0 while
> Samza
> > currently only support yarn 2.4.0)
> >
> > 7. others: scrooge, class loader isolation, etc.
> > - those are waiting for reviewing too.
> >
> > My opinion is that, if we can clean up all the patch-available tickets,
> it
> > will be great. Most of them have been already reviewed more than once.
> So I
> > think it should not be very time-consuming to have them in the 0.10.0
> > release.
> >
> > What do you think?
> >
> > Of course, another must-have is the bug-fix of the Stream Appender. :)
> >
> > Thanks,
> >
> > Fang, Yan
> > yanfang...@gmail.com
> >
> > On Tue, Jul 28, 2015 at 10:27 PM, Roger Hoover 
> > wrote:
> >
> > > Thanks, Yi.
> > >
> > > I propose that we also include SAMZA-741 for Elasticsearch versioning
> > > support with the new ES producer.  I think it's very close to being
> > merged.
> > >
> > > Roger
> > >
> > >
> > > On Tue, Jul 28, 2015 at 10:08 PM, Yi Pan  wrote:
> > >
> > > > Hi, all,
> > > >
> > > > I want to start the discussion on the release schedule for 0.10.0.
> > There
> > > > are a few important features that we plan to release in 0.10.0 and I
> > want
> > > > to start this thread s.t. we can agree on what to include in 0.10.0
> > > > release.
> > > >
> > > > There are the following main features added in 0.10.0:
> > > > - RocksDB TTL support
> > > > - Add CoordinatorStream and disable CheckpointManager
> > > > - Elasticsearch Producer
> > > > - Host affinity
> > > > And other 0.10.0 tickets:
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(%22In%20Progress%22%2C%20Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.10.0
> > > >
> > > > I propose to cut a 0.10.0 release after we get the following issues
> > > > resolved:
> > > > - SAMZA-615: Migrate checkpoint from checkpoint topic to Coordinator
> > > stream
> > > > - SAMZA-617: YARN host affinity in Samza
> > > >
> > > > Thoughts?
> > > >
> > > > Thanks!
> > > >
> > > > -Yi
> > > >
> > >
> >
>



-- 
Thanks and regards

Chinmay Soman

Re: [DISCUSS] Release 0.10.0

2015-07-29 Thread Navina Ramesh 
+1 for StreamAppender bug fix as a mandatory in 0.10.

@Yan: will review SAMZA-676 today and test it out locally.

Thanks!
Navina

On Wed, Jul 29, 2015 at 8:38 AM, Chinmay Soman 
wrote:

> I can take care of SAMZA-340, SAMZA-683  and will follow up with Luis for
> SAMZA-401,2,3,4
>
> On Wed, Jul 29, 2015 at 12:10 AM, Dan  wrote:
>
> > I agree SAMZA-741 for the ElasticSearch producer should be in too so
> we've
> > got a better API as part of that release.
> >
> >  - Dan
> >
> >
> > On 29 July 2015 at 07:51, Yan Fang  wrote:
> >
> > > Actually, I also want to include a few patch-available features,
> > > especially:
> > >
> > > 1. broadcast stream (SAMZA-676)
> > > - waiting for review
> > >
> > > 2. graphite support (SAMZA-340)
> > > 3. meter and histogram (SAMZA-683)
> > > 4. utility (SAMZA-401)
> > > - 2,3,4 belong to Luis, if he does not have time to update, since
> > they
> > > only need some small changes, we can edit it and get +1 from another
> > > committer.
> > >
> > > 5. hdfs producer (SAMZA-693)
> > > - I am reviewing.
> > >
> > > 6. upgrade yarn to 2.7.1 (SAMZA-563)
> > >- though I am reviewing, this ticket is negotiable if we want to put
> > > into the 0.10.0 release. If we do not, I think, when users enable the
> > > worker-persisting and container-persisting features, Samza will not be
> > able
> > > to handle it. (Some classes are only available after yarn 2.5.0 while
> > Samza
> > > currently only support yarn 2.4.0)
> > >
> > > 7. others: scrooge, class loader isolation, etc.
> > > - those are waiting for reviewing too.
> > >
> > > My opinion is that, if we can clean up all the patch-available tickets,
> > it
> > > will be great. Most of them have been already reviewed more than once.
> > So I
> > > think it should not be very time-consuming to have them in the 0.10.0
> > > release.
> > >
> > > What do you think?
> > >
> > > Of course, another must-have is the bug-fix of the Stream Appender. :)
> > >
> > > Thanks,
> > >
> > > Fang, Yan
> > > yanfang...@gmail.com
> > >
> > > On Tue, Jul 28, 2015 at 10:27 PM, Roger Hoover  >
> > > wrote:
> > >
> > > > Thanks, Yi.
> > > >
> > > > I propose that we also include SAMZA-741 for Elasticsearch versioning
> > > > support with the new ES producer.  I think it's very close to being
> > > merged.
> > > >
> > > > Roger
> > > >
> > > >
> > > > On Tue, Jul 28, 2015 at 10:08 PM, Yi Pan 
> wrote:
> > > >
> > > > > Hi, all,
> > > > >
> > > > > I want to start the discussion on the release schedule for 0.10.0.
> > > There
> > > > > are a few important features that we plan to release in 0.10.0 and
> I
> > > want
> > > > > to start this thread s.t. we can agree on what to include in 0.10.0
> > > > > release.
> > > > >
> > > > > There are the following main features added in 0.10.0:
> > > > > - RocksDB TTL support
> > > > > - Add CoordinatorStream and disable CheckpointManager
> > > > > - Elasticsearch Producer
> > > > > - Host affinity
> > > > > And other 0.10.0 tickets:
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(%22In%20Progress%22%2C%20Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.10.0
> > > > >
> > > > > I propose to cut a 0.10.0 release after we get the following issues
> > > > > resolved:
> > > > > - SAMZA-615: Migrate checkpoint from checkpoint topic to
> Coordinator
> > > > stream
> > > > > - SAMZA-617: YARN host affinity in Samza
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > -Yi
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Thanks and regards
>
> Chinmay Soman
>



-- 
Navina R.

Re: [DISCUSS] Release 0.10.0

2015-07-30 Thread Yi Pan 
Hi, all,

Thanks a lot for helping out to select the features planned in 0.10.0.

Based on the above discussion, I am proposing to move the following tickets
later (i.e. 0.11.0).

30/32 tickets Status=open/reopen (exception for SAMZA-723: stream appender
deadlock issue) :
https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC

Two tickets excluded:
- SAMZA-689, @Chinmay, were you able to re-produce it? I am thinking of
mark it as won't fix since 0.10.0 will remove the checkpoint topic
altogether.
- SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include the fix
that breaks Samza 0.10.0 build on Linux boxes

Anything that I missed?

Thanks!

-Yi


On Wed, Jul 29, 2015 at 9:57 AM, Navina Ramesh  wrote:

> +1 for StreamAppender bug fix as a mandatory in 0.10.
>
> @Yan: will review SAMZA-676 today and test it out locally.
>
> Thanks!
> Navina
>
> On Wed, Jul 29, 2015 at 8:38 AM, Chinmay Soman 
> wrote:
>
> > I can take care of SAMZA-340, SAMZA-683  and will follow up with Luis for
> > SAMZA-401,2,3,4
> >
> > On Wed, Jul 29, 2015 at 12:10 AM, Dan  wrote:
> >
> > > I agree SAMZA-741 for the ElasticSearch producer should be in too so
> > we've
> > > got a better API as part of that release.
> > >
> > >  - Dan
> > >
> > >
> > > On 29 July 2015 at 07:51, Yan Fang  wrote:
> > >
> > > > Actually, I also want to include a few patch-available features,
> > > > especially:
> > > >
> > > > 1. broadcast stream (SAMZA-676)
> > > > - waiting for review
> > > >
> > > > 2. graphite support (SAMZA-340)
> > > > 3. meter and histogram (SAMZA-683)
> > > > 4. utility (SAMZA-401)
> > > > - 2,3,4 belong to Luis, if he does not have time to update, since
> > > they
> > > > only need some small changes, we can edit it and get +1 from another
> > > > committer.
> > > >
> > > > 5. hdfs producer (SAMZA-693)
> > > > - I am reviewing.
> > > >
> > > > 6. upgrade yarn to 2.7.1 (SAMZA-563)
> > > >- though I am reviewing, this ticket is negotiable if we want to
> put
> > > > into the 0.10.0 release. If we do not, I think, when users enable the
> > > > worker-persisting and container-persisting features, Samza will not
> be
> > > able
> > > > to handle it. (Some classes are only available after yarn 2.5.0 while
> > > Samza
> > > > currently only support yarn 2.4.0)
> > > >
> > > > 7. others: scrooge, class loader isolation, etc.
> > > > - those are waiting for reviewing too.
> > > >
> > > > My opinion is that, if we can clean up all the patch-available
> tickets,
> > > it
> > > > will be great. Most of them have been already reviewed more than
> once.
> > > So I
> > > > think it should not be very time-consuming to have them in the 0.10.0
> > > > release.
> > > >
> > > > What do you think?
> > > >
> > > > Of course, another must-have is the bug-fix of the Stream Appender.
> :)
> > > >
> > > > Thanks,
> > > >
> > > > Fang, Yan
> > > > yanfang...@gmail.com
> > > >
> > > > On Tue, Jul 28, 2015 at 10:27 PM, Roger Hoover <
> roger.hoo...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Thanks, Yi.
> > > > >
> > > > > I propose that we also include SAMZA-741 for Elasticsearch
> versioning
> > > > > support with the new ES producer.  I think it's very close to being
> > > > merged.
> > > > >
> > > > > Roger
> > > > >
> > > > >
> > > > > On Tue, Jul 28, 2015 at 10:08 PM, Yi Pan 
> > wrote:
> > > > >
> > > > > > Hi, all,
> > > > > >
> > > > > > I want to start the discussion on the release schedule for
> 0.10.0.
> > > > There
> > > > > > are a few important features that we plan to release in 0.10.0
> and
> > I
> > > > want
> > > > > > to start this thread s.t. we can agree on what to include in
> 0.10.0
> > > > > > release.
> > > > > >
> > > > > > There are the following main features added in 0.10.0:
> > > > > > - RocksDB TTL support
> > > > > > - Add CoordinatorStream and disable CheckpointManager
> > > > > > - Elasticsearch Producer
> > > > > > - Host affinity
> > > > > > And other 0.10.0 tickets:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(%22In%20Progress%22%2C%20Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.10.0
> > > > > >
> > > > > > I propose to cut a 0.10.0 release after we get the following
> issues
> > > > > > resolved:
> > > > > > - SAMZA-615: Migrate checkpoint from checkpoint topic to
> > Coordinator
> > > > > stream
> > > > > > - SAMZA-617: YARN host affinity in Samza
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > -Yi
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Thanks and regards
> >
> > Chinmay Soman
> >
>
>
>
> --
> Navina R.
>

Re: [DISCUSS] Release 0.10.0

2015-07-30 Thread Yi Pan 
Sorry, hit the send button too fast. Let me correct the summary section:

29/32 tickets Status=open/reopen :
https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC

Three tickets excluded:
- SAMZA-689, @Chinmay, were you able to re-produce it? I am thinking of
mark it as won't fix since 0.10.0 will remove the checkpoint topic
altogether.
- SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include the fix
that breaks Samza 0.10.0 build on Linux boxes
- SAMZA-723, stream appender deadlock issue

Thanks!

-Yi


On Thu, Jul 30, 2015 at 4:52 PM, Yi Pan  wrote:

> Hi, all,
>
> Thanks a lot for helping out to select the features planned in 0.10.0.
>
> Based on the above discussion, I am proposing to move the following
> tickets later (i.e. 0.11.0).
>
> 30/32 tickets Status=open/reopen (exception for SAMZA-723: stream appender
> deadlock issue) :
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
>
> Two tickets excluded:
> - SAMZA-689, @Chinmay, were you able to re-produce it? I am thinking of
> mark it as won't fix since 0.10.0 will remove the checkpoint topic
> altogether.
> - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include the fix
> that breaks Samza 0.10.0 build on Linux boxes
>
> Anything that I missed?
>
> Thanks!
>
> -Yi
>
>
> On Wed, Jul 29, 2015 at 9:57 AM, Navina Ramesh <
> nram...@linkedin.com.invalid> wrote:
>
>> +1 for StreamAppender bug fix as a mandatory in 0.10.
>>
>> @Yan: will review SAMZA-676 today and test it out locally.
>>
>> Thanks!
>> Navina
>>
>> On Wed, Jul 29, 2015 at 8:38 AM, Chinmay Soman > >
>> wrote:
>>
>> > I can take care of SAMZA-340, SAMZA-683  and will follow up with Luis
>> for
>> > SAMZA-401,2,3,4
>> >
>> > On Wed, Jul 29, 2015 at 12:10 AM, Dan  wrote:
>> >
>> > > I agree SAMZA-741 for the ElasticSearch producer should be in too so
>> > we've
>> > > got a better API as part of that release.
>> > >
>> > >  - Dan
>> > >
>> > >
>> > > On 29 July 2015 at 07:51, Yan Fang  wrote:
>> > >
>> > > > Actually, I also want to include a few patch-available features,
>> > > > especially:
>> > > >
>> > > > 1. broadcast stream (SAMZA-676)
>> > > > - waiting for review
>> > > >
>> > > > 2. graphite support (SAMZA-340)
>> > > > 3. meter and histogram (SAMZA-683)
>> > > > 4. utility (SAMZA-401)
>> > > > - 2,3,4 belong to Luis, if he does not have time to update,
>> since
>> > > they
>> > > > only need some small changes, we can edit it and get +1 from another
>> > > > committer.
>> > > >
>> > > > 5. hdfs producer (SAMZA-693)
>> > > > - I am reviewing.
>> > > >
>> > > > 6. upgrade yarn to 2.7.1 (SAMZA-563)
>> > > >- though I am reviewing, this ticket is negotiable if we want to
>> put
>> > > > into the 0.10.0 release. If we do not, I think, when users enable
>> the
>> > > > worker-persisting and container-persisting features, Samza will not
>> be
>> > > able
>> > > > to handle it. (Some classes are only available after yarn 2.5.0
>> while
>> > > Samza
>> > > > currently only support yarn 2.4.0)
>> > > >
>> > > > 7. others: scrooge, class loader isolation, etc.
>> > > > - those are waiting for reviewing too.
>> > > >
>> > > > My opinion is that, if we can clean up all the patch-available
>> tickets,
>> > > it
>> > > > will be great. Most of them have been already reviewed more than
>> once.
>> > > So I
>> > > > think it should not be very time-consuming to have them in the
>> 0.10.0
>> > > > release.
>> > > >
>> > > > What do you think?
>> > > >
>> > > > Of course, another must-have is the bug-fix of the Stream Appender.
>> :)
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Fang, Yan
>> > > > yanfang...@gmail.com
>> > > >
>> > > > On Tue, Jul 28, 2015 at 10:27 PM, Roger Hoover <
>> roger.hoo...@gmail.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > Thanks, Yi.
>> > > > >
>> > > > > I propose that we also include SAMZA-741 for Elasticsearch
>> versioning
>> > > > > support with the new ES producer.  I think it's very close to
>> being
>> > > > merged.
>> > > > >
>> > > > > Roger
>> > > > >
>> > > > >
>> > > > > On Tue, Jul 28, 2015 at 10:08 PM, Yi Pan 
>> > wrote:
>> > > > >
>> > > > > > Hi, all,
>> > > > > >
>> > > > > > I want to start the discussion on the release schedule for
>> 0.10.0.
>> > > > There
>> > > > > > are a few important features that we plan to release in 0.10.0
>> and
>> > I
>> > > > want
>> > > > > > to start this thread s.t. we can agree on what to include in
>> 0.10.0
>> > > > > > release.
>> > > > > >
>> > > > > > There are the following main features added in 0.10.0:
>> > > > > > - RocksDB TTL support
>> > > > > > - Add CoordinatorStream and disable CheckpointManager
>> > > > > > - Elasticsearch Producer
>> > > > > > - Host affinity
>

Re: [DISCUSS] Release 0.10.0

2015-07-30 Thread Navina Ramesh 
Hi Yi,
Thanks for summarizing. But why are we excluding SAMZA-723 from the current
release ? Doesn't this break the existing StreamAppender functionality in
0.9?

Thanks!
Navina

On Thu, Jul 30, 2015 at 4:55 PM, Yi Pan  wrote:

> Sorry, hit the send button too fast. Let me correct the summary section:
>
> 29/32 tickets Status=open/reopen :
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
>
> Three tickets excluded:
> - SAMZA-689, @Chinmay, were you able to re-produce it? I am thinking of
> mark it as won't fix since 0.10.0 will remove the checkpoint topic
> altogether.
> - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include the fix
> that breaks Samza 0.10.0 build on Linux boxes
> - SAMZA-723, stream appender deadlock issue
>
> Thanks!
>
> -Yi
>
>
> On Thu, Jul 30, 2015 at 4:52 PM, Yi Pan  wrote:
>
> > Hi, all,
> >
> > Thanks a lot for helping out to select the features planned in 0.10.0.
> >
> > Based on the above discussion, I am proposing to move the following
> > tickets later (i.e. 0.11.0).
> >
> > 30/32 tickets Status=open/reopen (exception for SAMZA-723: stream
> appender
> > deadlock issue) :
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> >
> > Two tickets excluded:
> > - SAMZA-689, @Chinmay, were you able to re-produce it? I am thinking of
> > mark it as won't fix since 0.10.0 will remove the checkpoint topic
> > altogether.
> > - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include the fix
> > that breaks Samza 0.10.0 build on Linux boxes
> >
> > Anything that I missed?
> >
> > Thanks!
> >
> > -Yi
> >
> >
> > On Wed, Jul 29, 2015 at 9:57 AM, Navina Ramesh <
> > nram...@linkedin.com.invalid> wrote:
> >
> >> +1 for StreamAppender bug fix as a mandatory in 0.10.
> >>
> >> @Yan: will review SAMZA-676 today and test it out locally.
> >>
> >> Thanks!
> >> Navina
> >>
> >> On Wed, Jul 29, 2015 at 8:38 AM, Chinmay Soman <
> chinmay.cere...@gmail.com
> >> >
> >> wrote:
> >>
> >> > I can take care of SAMZA-340, SAMZA-683  and will follow up with Luis
> >> for
> >> > SAMZA-401,2,3,4
> >> >
> >> > On Wed, Jul 29, 2015 at 12:10 AM, Dan  wrote:
> >> >
> >> > > I agree SAMZA-741 for the ElasticSearch producer should be in too so
> >> > we've
> >> > > got a better API as part of that release.
> >> > >
> >> > >  - Dan
> >> > >
> >> > >
> >> > > On 29 July 2015 at 07:51, Yan Fang  wrote:
> >> > >
> >> > > > Actually, I also want to include a few patch-available features,
> >> > > > especially:
> >> > > >
> >> > > > 1. broadcast stream (SAMZA-676)
> >> > > > - waiting for review
> >> > > >
> >> > > > 2. graphite support (SAMZA-340)
> >> > > > 3. meter and histogram (SAMZA-683)
> >> > > > 4. utility (SAMZA-401)
> >> > > > - 2,3,4 belong to Luis, if he does not have time to update,
> >> since
> >> > > they
> >> > > > only need some small changes, we can edit it and get +1 from
> another
> >> > > > committer.
> >> > > >
> >> > > > 5. hdfs producer (SAMZA-693)
> >> > > > - I am reviewing.
> >> > > >
> >> > > > 6. upgrade yarn to 2.7.1 (SAMZA-563)
> >> > > >- though I am reviewing, this ticket is negotiable if we want
> to
> >> put
> >> > > > into the 0.10.0 release. If we do not, I think, when users enable
> >> the
> >> > > > worker-persisting and container-persisting features, Samza will
> not
> >> be
> >> > > able
> >> > > > to handle it. (Some classes are only available after yarn 2.5.0
> >> while
> >> > > Samza
> >> > > > currently only support yarn 2.4.0)
> >> > > >
> >> > > > 7. others: scrooge, class loader isolation, etc.
> >> > > > - those are waiting for reviewing too.
> >> > > >
> >> > > > My opinion is that, if we can clean up all the patch-available
> >> tickets,
> >> > > it
> >> > > > will be great. Most of them have been already reviewed more than
> >> once.
> >> > > So I
> >> > > > think it should not be very time-consuming to have them in the
> >> 0.10.0
> >> > > > release.
> >> > > >
> >> > > > What do you think?
> >> > > >
> >> > > > Of course, another must-have is the bug-fix of the Stream
> Appender.
> >> :)
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Fang, Yan
> >> > > > yanfang...@gmail.com
> >> > > >
> >> > > > On Tue, Jul 28, 2015 at 10:27 PM, Roger Hoover <
> >> roger.hoo...@gmail.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > Thanks, Yi.
> >> > > > >
> >> > > > > I propose that we also include SAMZA-741 for Elasticsearch
> >> versioning
> >> > > > > support with the new ES producer.  I think it's very close to
> >> being
> >> > > > merged.
> >> > > > >
> >> > > > > Roger
> >> > > > >
> >> > > > >
> >> > > > > On Tue, Jul 28, 2015 at 10:08 PM, Yi Pan 
> >> > wrote:
> >> > > > >
> >> > > > > > Hi, all,
> >> > > >

Re: [DISCUSS] Release 0.10.0

2015-07-30 Thread Yi Pan 
Hi, Navina,

The 29/30 tickets are to be excluded from 0.10.0 (i.e. moved to 0.11.0),
the three tickets are either to be included in 0.10.0, or won't fix.

-Yi

On Thu, Jul 30, 2015 at 5:10 PM, Navina Ramesh  wrote:

> Hi Yi,
> Thanks for summarizing. But why are we excluding SAMZA-723 from the current
> release ? Doesn't this break the existing StreamAppender functionality in
> 0.9?
>
> Thanks!
> Navina
>
> On Thu, Jul 30, 2015 at 4:55 PM, Yi Pan  wrote:
>
> > Sorry, hit the send button too fast. Let me correct the summary section:
> >
> > 29/32 tickets Status=open/reopen :
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> >
> > Three tickets excluded:
> > - SAMZA-689, @Chinmay, were you able to re-produce it? I am thinking of
> > mark it as won't fix since 0.10.0 will remove the checkpoint topic
> > altogether.
> > - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include the fix
> > that breaks Samza 0.10.0 build on Linux boxes
> > - SAMZA-723, stream appender deadlock issue
> >
> > Thanks!
> >
> > -Yi
> >
> >
> > On Thu, Jul 30, 2015 at 4:52 PM, Yi Pan  wrote:
> >
> > > Hi, all,
> > >
> > > Thanks a lot for helping out to select the features planned in 0.10.0.
> > >
> > > Based on the above discussion, I am proposing to move the following
> > > tickets later (i.e. 0.11.0).
> > >
> > > 30/32 tickets Status=open/reopen (exception for SAMZA-723: stream
> > appender
> > > deadlock issue) :
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> > >
> > > Two tickets excluded:
> > > - SAMZA-689, @Chinmay, were you able to re-produce it? I am thinking of
> > > mark it as won't fix since 0.10.0 will remove the checkpoint topic
> > > altogether.
> > > - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include the
> fix
> > > that breaks Samza 0.10.0 build on Linux boxes
> > >
> > > Anything that I missed?
> > >
> > > Thanks!
> > >
> > > -Yi
> > >
> > >
> > > On Wed, Jul 29, 2015 at 9:57 AM, Navina Ramesh <
> > > nram...@linkedin.com.invalid> wrote:
> > >
> > >> +1 for StreamAppender bug fix as a mandatory in 0.10.
> > >>
> > >> @Yan: will review SAMZA-676 today and test it out locally.
> > >>
> > >> Thanks!
> > >> Navina
> > >>
> > >> On Wed, Jul 29, 2015 at 8:38 AM, Chinmay Soman <
> > chinmay.cere...@gmail.com
> > >> >
> > >> wrote:
> > >>
> > >> > I can take care of SAMZA-340, SAMZA-683  and will follow up with
> Luis
> > >> for
> > >> > SAMZA-401,2,3,4
> > >> >
> > >> > On Wed, Jul 29, 2015 at 12:10 AM, Dan 
> wrote:
> > >> >
> > >> > > I agree SAMZA-741 for the ElasticSearch producer should be in too
> so
> > >> > we've
> > >> > > got a better API as part of that release.
> > >> > >
> > >> > >  - Dan
> > >> > >
> > >> > >
> > >> > > On 29 July 2015 at 07:51, Yan Fang  wrote:
> > >> > >
> > >> > > > Actually, I also want to include a few patch-available features,
> > >> > > > especially:
> > >> > > >
> > >> > > > 1. broadcast stream (SAMZA-676)
> > >> > > > - waiting for review
> > >> > > >
> > >> > > > 2. graphite support (SAMZA-340)
> > >> > > > 3. meter and histogram (SAMZA-683)
> > >> > > > 4. utility (SAMZA-401)
> > >> > > > - 2,3,4 belong to Luis, if he does not have time to update,
> > >> since
> > >> > > they
> > >> > > > only need some small changes, we can edit it and get +1 from
> > another
> > >> > > > committer.
> > >> > > >
> > >> > > > 5. hdfs producer (SAMZA-693)
> > >> > > > - I am reviewing.
> > >> > > >
> > >> > > > 6. upgrade yarn to 2.7.1 (SAMZA-563)
> > >> > > >- though I am reviewing, this ticket is negotiable if we want
> > to
> > >> put
> > >> > > > into the 0.10.0 release. If we do not, I think, when users
> enable
> > >> the
> > >> > > > worker-persisting and container-persisting features, Samza will
> > not
> > >> be
> > >> > > able
> > >> > > > to handle it. (Some classes are only available after yarn 2.5.0
> > >> while
> > >> > > Samza
> > >> > > > currently only support yarn 2.4.0)
> > >> > > >
> > >> > > > 7. others: scrooge, class loader isolation, etc.
> > >> > > > - those are waiting for reviewing too.
> > >> > > >
> > >> > > > My opinion is that, if we can clean up all the patch-available
> > >> tickets,
> > >> > > it
> > >> > > > will be great. Most of them have been already reviewed more than
> > >> once.
> > >> > > So I
> > >> > > > think it should not be very time-consuming to have them in the
> > >> 0.10.0
> > >> > > > release.
> > >> > > >
> > >> > > > What do you think?
> > >> > > >
> > >> > > > Of course, another must-have is the bug-fix of the Stream
> > Appender.
> > >> :)
> > >> > > >
> > >> > > > Thanks,
> > >> > > >
> > >> > > > Fang, Yan
> > >> > > > yanfang...@gmail.com
> > >> > > >
> > >> > >

Re: [DISCUSS] Release 0.10.0

2015-07-30 Thread Yi Pan 
Uh... wrong math all the day today... :(

Let me re-try:

29/32 tickets are to be moved to later (i.e. excluded from 0.10.0).

2/32 tickets (including SAMZA-723) will be included

1/32 ticket (SAMZA-689) is up for discussion and I am leaning toward mark
it as won't fix.

-Yi

On Thu, Jul 30, 2015 at 5:24 PM, Yi Pan  wrote:

> Hi, Navina,
>
> The 29/30 tickets are to be excluded from 0.10.0 (i.e. moved to 0.11.0),
> the three tickets are either to be included in 0.10.0, or won't fix.
>
> -Yi
>
> On Thu, Jul 30, 2015 at 5:10 PM, Navina Ramesh <
> nram...@linkedin.com.invalid> wrote:
>
>> Hi Yi,
>> Thanks for summarizing. But why are we excluding SAMZA-723 from the
>> current
>> release ? Doesn't this break the existing StreamAppender functionality in
>> 0.9?
>>
>> Thanks!
>> Navina
>>
>> On Thu, Jul 30, 2015 at 4:55 PM, Yi Pan  wrote:
>>
>> > Sorry, hit the send button too fast. Let me correct the summary section:
>> >
>> > 29/32 tickets Status=open/reopen :
>> >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
>> >
>> > Three tickets excluded:
>> > - SAMZA-689, @Chinmay, were you able to re-produce it? I am thinking of
>> > mark it as won't fix since 0.10.0 will remove the checkpoint topic
>> > altogether.
>> > - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include the
>> fix
>> > that breaks Samza 0.10.0 build on Linux boxes
>> > - SAMZA-723, stream appender deadlock issue
>> >
>> > Thanks!
>> >
>> > -Yi
>> >
>> >
>> > On Thu, Jul 30, 2015 at 4:52 PM, Yi Pan  wrote:
>> >
>> > > Hi, all,
>> > >
>> > > Thanks a lot for helping out to select the features planned in 0.10.0.
>> > >
>> > > Based on the above discussion, I am proposing to move the following
>> > > tickets later (i.e. 0.11.0).
>> > >
>> > > 30/32 tickets Status=open/reopen (exception for SAMZA-723: stream
>> > appender
>> > > deadlock issue) :
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
>> > >
>> > > Two tickets excluded:
>> > > - SAMZA-689, @Chinmay, were you able to re-produce it? I am thinking
>> of
>> > > mark it as won't fix since 0.10.0 will remove the checkpoint topic
>> > > altogether.
>> > > - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include the
>> fix
>> > > that breaks Samza 0.10.0 build on Linux boxes
>> > >
>> > > Anything that I missed?
>> > >
>> > > Thanks!
>> > >
>> > > -Yi
>> > >
>> > >
>> > > On Wed, Jul 29, 2015 at 9:57 AM, Navina Ramesh <
>> > > nram...@linkedin.com.invalid> wrote:
>> > >
>> > >> +1 for StreamAppender bug fix as a mandatory in 0.10.
>> > >>
>> > >> @Yan: will review SAMZA-676 today and test it out locally.
>> > >>
>> > >> Thanks!
>> > >> Navina
>> > >>
>> > >> On Wed, Jul 29, 2015 at 8:38 AM, Chinmay Soman <
>> > chinmay.cere...@gmail.com
>> > >> >
>> > >> wrote:
>> > >>
>> > >> > I can take care of SAMZA-340, SAMZA-683  and will follow up with
>> Luis
>> > >> for
>> > >> > SAMZA-401,2,3,4
>> > >> >
>> > >> > On Wed, Jul 29, 2015 at 12:10 AM, Dan 
>> wrote:
>> > >> >
>> > >> > > I agree SAMZA-741 for the ElasticSearch producer should be in
>> too so
>> > >> > we've
>> > >> > > got a better API as part of that release.
>> > >> > >
>> > >> > >  - Dan
>> > >> > >
>> > >> > >
>> > >> > > On 29 July 2015 at 07:51, Yan Fang  wrote:
>> > >> > >
>> > >> > > > Actually, I also want to include a few patch-available
>> features,
>> > >> > > > especially:
>> > >> > > >
>> > >> > > > 1. broadcast stream (SAMZA-676)
>> > >> > > > - waiting for review
>> > >> > > >
>> > >> > > > 2. graphite support (SAMZA-340)
>> > >> > > > 3. meter and histogram (SAMZA-683)
>> > >> > > > 4. utility (SAMZA-401)
>> > >> > > > - 2,3,4 belong to Luis, if he does not have time to update,
>> > >> since
>> > >> > > they
>> > >> > > > only need some small changes, we can edit it and get +1 from
>> > another
>> > >> > > > committer.
>> > >> > > >
>> > >> > > > 5. hdfs producer (SAMZA-693)
>> > >> > > > - I am reviewing.
>> > >> > > >
>> > >> > > > 6. upgrade yarn to 2.7.1 (SAMZA-563)
>> > >> > > >- though I am reviewing, this ticket is negotiable if we
>> want
>> > to
>> > >> put
>> > >> > > > into the 0.10.0 release. If we do not, I think, when users
>> enable
>> > >> the
>> > >> > > > worker-persisting and container-persisting features, Samza will
>> > not
>> > >> be
>> > >> > > able
>> > >> > > > to handle it. (Some classes are only available after yarn 2.5.0
>> > >> while
>> > >> > > Samza
>> > >> > > > currently only support yarn 2.4.0)
>> > >> > > >
>> > >> > > > 7. others: scrooge, class loader isolation, etc.
>> > >> > > > - those are waiting for reviewing too.
>> > >> > > >
>> > >> > > > My opinion is that, if we can clean up all the patch-available
>> > >> t

Re: [DISCUSS] Release 0.10.0

2015-07-30 Thread Navina Ramesh 
Ok. I think it got confusing because you are talking about tickets NOT to
be included in the release :)

Got it, now. +1 for this list of exclusions!
SAMZA-723 (StreamAppender bug) and SAMZA-747 (rocksdb) should be in 0.10.0.
I think we don't have an ETA on the fix for SAMZA-747?

Thanks!
Navina

On Thu, Jul 30, 2015 at 5:26 PM, Yi Pan  wrote:

> Uh... wrong math all the day today... :(
>
> Let me re-try:
>
> 29/32 tickets are to be moved to later (i.e. excluded from 0.10.0).
>
> 2/32 tickets (including SAMZA-723) will be included
>
> 1/32 ticket (SAMZA-689) is up for discussion and I am leaning toward mark
> it as won't fix.
>
> -Yi
>
> On Thu, Jul 30, 2015 at 5:24 PM, Yi Pan  wrote:
>
> > Hi, Navina,
> >
> > The 29/30 tickets are to be excluded from 0.10.0 (i.e. moved to 0.11.0),
> > the three tickets are either to be included in 0.10.0, or won't fix.
> >
> > -Yi
> >
> > On Thu, Jul 30, 2015 at 5:10 PM, Navina Ramesh <
> > nram...@linkedin.com.invalid> wrote:
> >
> >> Hi Yi,
> >> Thanks for summarizing. But why are we excluding SAMZA-723 from the
> >> current
> >> release ? Doesn't this break the existing StreamAppender functionality
> in
> >> 0.9?
> >>
> >> Thanks!
> >> Navina
> >>
> >> On Thu, Jul 30, 2015 at 4:55 PM, Yi Pan  wrote:
> >>
> >> > Sorry, hit the send button too fast. Let me correct the summary
> section:
> >> >
> >> > 29/32 tickets Status=open/reopen :
> >> >
> >> >
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> >> >
> >> > Three tickets excluded:
> >> > - SAMZA-689, @Chinmay, were you able to re-produce it? I am thinking
> of
> >> > mark it as won't fix since 0.10.0 will remove the checkpoint topic
> >> > altogether.
> >> > - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include the
> >> fix
> >> > that breaks Samza 0.10.0 build on Linux boxes
> >> > - SAMZA-723, stream appender deadlock issue
> >> >
> >> > Thanks!
> >> >
> >> > -Yi
> >> >
> >> >
> >> > On Thu, Jul 30, 2015 at 4:52 PM, Yi Pan  wrote:
> >> >
> >> > > Hi, all,
> >> > >
> >> > > Thanks a lot for helping out to select the features planned in
> 0.10.0.
> >> > >
> >> > > Based on the above discussion, I am proposing to move the following
> >> > > tickets later (i.e. 0.11.0).
> >> > >
> >> > > 30/32 tickets Status=open/reopen (exception for SAMZA-723: stream
> >> > appender
> >> > > deadlock issue) :
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> >> > >
> >> > > Two tickets excluded:
> >> > > - SAMZA-689, @Chinmay, were you able to re-produce it? I am thinking
> >> of
> >> > > mark it as won't fix since 0.10.0 will remove the checkpoint topic
> >> > > altogether.
> >> > > - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include
> the
> >> fix
> >> > > that breaks Samza 0.10.0 build on Linux boxes
> >> > >
> >> > > Anything that I missed?
> >> > >
> >> > > Thanks!
> >> > >
> >> > > -Yi
> >> > >
> >> > >
> >> > > On Wed, Jul 29, 2015 at 9:57 AM, Navina Ramesh <
> >> > > nram...@linkedin.com.invalid> wrote:
> >> > >
> >> > >> +1 for StreamAppender bug fix as a mandatory in 0.10.
> >> > >>
> >> > >> @Yan: will review SAMZA-676 today and test it out locally.
> >> > >>
> >> > >> Thanks!
> >> > >> Navina
> >> > >>
> >> > >> On Wed, Jul 29, 2015 at 8:38 AM, Chinmay Soman <
> >> > chinmay.cere...@gmail.com
> >> > >> >
> >> > >> wrote:
> >> > >>
> >> > >> > I can take care of SAMZA-340, SAMZA-683  and will follow up with
> >> Luis
> >> > >> for
> >> > >> > SAMZA-401,2,3,4
> >> > >> >
> >> > >> > On Wed, Jul 29, 2015 at 12:10 AM, Dan 
> >> wrote:
> >> > >> >
> >> > >> > > I agree SAMZA-741 for the ElasticSearch producer should be in
> >> too so
> >> > >> > we've
> >> > >> > > got a better API as part of that release.
> >> > >> > >
> >> > >> > >  - Dan
> >> > >> > >
> >> > >> > >
> >> > >> > > On 29 July 2015 at 07:51, Yan Fang 
> wrote:
> >> > >> > >
> >> > >> > > > Actually, I also want to include a few patch-available
> >> features,
> >> > >> > > > especially:
> >> > >> > > >
> >> > >> > > > 1. broadcast stream (SAMZA-676)
> >> > >> > > > - waiting for review
> >> > >> > > >
> >> > >> > > > 2. graphite support (SAMZA-340)
> >> > >> > > > 3. meter and histogram (SAMZA-683)
> >> > >> > > > 4. utility (SAMZA-401)
> >> > >> > > > - 2,3,4 belong to Luis, if he does not have time to
> update,
> >> > >> since
> >> > >> > > they
> >> > >> > > > only need some small changes, we can edit it and get +1 from
> >> > another
> >> > >> > > > committer.
> >> > >> > > >
> >> > >> > > > 5. hdfs producer (SAMZA-693)
> >> > >> > > > - I am reviewing.
> >> > >> > > >
> >> > >> > > > 6. upgrade yarn to 2.7.1 (SAMZA-563)
> >> > >> > > >- though I am reviewing, this ticket is ne

Re: [DISCUSS] Release 0.10.0

2015-07-30 Thread Yan Fang 
For SAMZA-747, may ping Naveen or Chris. :)

They have the permission to publish to maven. From the discuss
, they seem ready for the
release.

Thanks,

Fang, Yan
yanfang...@gmail.com

On Thu, Jul 30, 2015 at 5:35 PM, Navina Ramesh  wrote:

> Ok. I think it got confusing because you are talking about tickets NOT to
> be included in the release :)
>
> Got it, now. +1 for this list of exclusions!
> SAMZA-723 (StreamAppender bug) and SAMZA-747 (rocksdb) should be in 0.10.0.
> I think we don't have an ETA on the fix for SAMZA-747?
>
> Thanks!
> Navina
>
> On Thu, Jul 30, 2015 at 5:26 PM, Yi Pan  wrote:
>
> > Uh... wrong math all the day today... :(
> >
> > Let me re-try:
> >
> > 29/32 tickets are to be moved to later (i.e. excluded from 0.10.0).
> >
> > 2/32 tickets (including SAMZA-723) will be included
> >
> > 1/32 ticket (SAMZA-689) is up for discussion and I am leaning toward mark
> > it as won't fix.
> >
> > -Yi
> >
> > On Thu, Jul 30, 2015 at 5:24 PM, Yi Pan  wrote:
> >
> > > Hi, Navina,
> > >
> > > The 29/30 tickets are to be excluded from 0.10.0 (i.e. moved to
> 0.11.0),
> > > the three tickets are either to be included in 0.10.0, or won't fix.
> > >
> > > -Yi
> > >
> > > On Thu, Jul 30, 2015 at 5:10 PM, Navina Ramesh <
> > > nram...@linkedin.com.invalid> wrote:
> > >
> > >> Hi Yi,
> > >> Thanks for summarizing. But why are we excluding SAMZA-723 from the
> > >> current
> > >> release ? Doesn't this break the existing StreamAppender functionality
> > in
> > >> 0.9?
> > >>
> > >> Thanks!
> > >> Navina
> > >>
> > >> On Thu, Jul 30, 2015 at 4:55 PM, Yi Pan  wrote:
> > >>
> > >> > Sorry, hit the send button too fast. Let me correct the summary
> > section:
> > >> >
> > >> > 29/32 tickets Status=open/reopen :
> > >> >
> > >> >
> > >>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> > >> >
> > >> > Three tickets excluded:
> > >> > - SAMZA-689, @Chinmay, were you able to re-produce it? I am thinking
> > of
> > >> > mark it as won't fix since 0.10.0 will remove the checkpoint topic
> > >> > altogether.
> > >> > - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include
> the
> > >> fix
> > >> > that breaks Samza 0.10.0 build on Linux boxes
> > >> > - SAMZA-723, stream appender deadlock issue
> > >> >
> > >> > Thanks!
> > >> >
> > >> > -Yi
> > >> >
> > >> >
> > >> > On Thu, Jul 30, 2015 at 4:52 PM, Yi Pan 
> wrote:
> > >> >
> > >> > > Hi, all,
> > >> > >
> > >> > > Thanks a lot for helping out to select the features planned in
> > 0.10.0.
> > >> > >
> > >> > > Based on the above discussion, I am proposing to move the
> following
> > >> > > tickets later (i.e. 0.11.0).
> > >> > >
> > >> > > 30/32 tickets Status=open/reopen (exception for SAMZA-723: stream
> > >> > appender
> > >> > > deadlock issue) :
> > >> > >
> > >> >
> > >>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> > >> > >
> > >> > > Two tickets excluded:
> > >> > > - SAMZA-689, @Chinmay, were you able to re-produce it? I am
> thinking
> > >> of
> > >> > > mark it as won't fix since 0.10.0 will remove the checkpoint topic
> > >> > > altogether.
> > >> > > - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include
> > the
> > >> fix
> > >> > > that breaks Samza 0.10.0 build on Linux boxes
> > >> > >
> > >> > > Anything that I missed?
> > >> > >
> > >> > > Thanks!
> > >> > >
> > >> > > -Yi
> > >> > >
> > >> > >
> > >> > > On Wed, Jul 29, 2015 at 9:57 AM, Navina Ramesh <
> > >> > > nram...@linkedin.com.invalid> wrote:
> > >> > >
> > >> > >> +1 for StreamAppender bug fix as a mandatory in 0.10.
> > >> > >>
> > >> > >> @Yan: will review SAMZA-676 today and test it out locally.
> > >> > >>
> > >> > >> Thanks!
> > >> > >> Navina
> > >> > >>
> > >> > >> On Wed, Jul 29, 2015 at 8:38 AM, Chinmay Soman <
> > >> > chinmay.cere...@gmail.com
> > >> > >> >
> > >> > >> wrote:
> > >> > >>
> > >> > >> > I can take care of SAMZA-340, SAMZA-683  and will follow up
> with
> > >> Luis
> > >> > >> for
> > >> > >> > SAMZA-401,2,3,4
> > >> > >> >
> > >> > >> > On Wed, Jul 29, 2015 at 12:10 AM, Dan 
> > >> wrote:
> > >> > >> >
> > >> > >> > > I agree SAMZA-741 for the ElasticSearch producer should be in
> > >> too so
> > >> > >> > we've
> > >> > >> > > got a better API as part of that release.
> > >> > >> > >
> > >> > >> > >  - Dan
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > On 29 July 2015 at 07:51, Yan Fang 
> > wrote:
> > >> > >> > >
> > >> > >> > > > Actually, I also want to include a few patch-available
> > >> features,
> > >> > >> > > > especially:
> > >> > >> > > >
> > >> > >> > > > 1. broadcast stream (SAMZA-676)
> > >> > >> > > > - waiting for review
> > >> > >> > > >
> >

Re: [DISCUSS] Release 0.10.0

2015-07-31 Thread Guozhang Wang 
Would SAMZA-697 worth adding to 0.10.0 as well?

Guozhang

On Thu, Jul 30, 2015 at 6:45 PM, Yan Fang  wrote:

> For SAMZA-747, may ping Naveen or Chris. :)
>
> They have the permission to publish to maven. From the discuss
> , they seem ready for the
> release.
>
> Thanks,
>
> Fang, Yan
> yanfang...@gmail.com
>
> On Thu, Jul 30, 2015 at 5:35 PM, Navina Ramesh
>  > wrote:
>
> > Ok. I think it got confusing because you are talking about tickets NOT to
> > be included in the release :)
> >
> > Got it, now. +1 for this list of exclusions!
> > SAMZA-723 (StreamAppender bug) and SAMZA-747 (rocksdb) should be in
> 0.10.0.
> > I think we don't have an ETA on the fix for SAMZA-747?
> >
> > Thanks!
> > Navina
> >
> > On Thu, Jul 30, 2015 at 5:26 PM, Yi Pan  wrote:
> >
> > > Uh... wrong math all the day today... :(
> > >
> > > Let me re-try:
> > >
> > > 29/32 tickets are to be moved to later (i.e. excluded from 0.10.0).
> > >
> > > 2/32 tickets (including SAMZA-723) will be included
> > >
> > > 1/32 ticket (SAMZA-689) is up for discussion and I am leaning toward
> mark
> > > it as won't fix.
> > >
> > > -Yi
> > >
> > > On Thu, Jul 30, 2015 at 5:24 PM, Yi Pan  wrote:
> > >
> > > > Hi, Navina,
> > > >
> > > > The 29/30 tickets are to be excluded from 0.10.0 (i.e. moved to
> > 0.11.0),
> > > > the three tickets are either to be included in 0.10.0, or won't fix.
> > > >
> > > > -Yi
> > > >
> > > > On Thu, Jul 30, 2015 at 5:10 PM, Navina Ramesh <
> > > > nram...@linkedin.com.invalid> wrote:
> > > >
> > > >> Hi Yi,
> > > >> Thanks for summarizing. But why are we excluding SAMZA-723 from the
> > > >> current
> > > >> release ? Doesn't this break the existing StreamAppender
> functionality
> > > in
> > > >> 0.9?
> > > >>
> > > >> Thanks!
> > > >> Navina
> > > >>
> > > >> On Thu, Jul 30, 2015 at 4:55 PM, Yi Pan 
> wrote:
> > > >>
> > > >> > Sorry, hit the send button too fast. Let me correct the summary
> > > section:
> > > >> >
> > > >> > 29/32 tickets Status=open/reopen :
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> > > >> >
> > > >> > Three tickets excluded:
> > > >> > - SAMZA-689, @Chinmay, were you able to re-produce it? I am
> thinking
> > > of
> > > >> > mark it as won't fix since 0.10.0 will remove the checkpoint topic
> > > >> > altogether.
> > > >> > - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to include
> > the
> > > >> fix
> > > >> > that breaks Samza 0.10.0 build on Linux boxes
> > > >> > - SAMZA-723, stream appender deadlock issue
> > > >> >
> > > >> > Thanks!
> > > >> >
> > > >> > -Yi
> > > >> >
> > > >> >
> > > >> > On Thu, Jul 30, 2015 at 4:52 PM, Yi Pan 
> > wrote:
> > > >> >
> > > >> > > Hi, all,
> > > >> > >
> > > >> > > Thanks a lot for helping out to select the features planned in
> > > 0.10.0.
> > > >> > >
> > > >> > > Based on the above discussion, I am proposing to move the
> > following
> > > >> > > tickets later (i.e. 0.11.0).
> > > >> > >
> > > >> > > 30/32 tickets Status=open/reopen (exception for SAMZA-723:
> stream
> > > >> > appender
> > > >> > > deadlock issue) :
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> > > >> > >
> > > >> > > Two tickets excluded:
> > > >> > > - SAMZA-689, @Chinmay, were you able to re-produce it? I am
> > thinking
> > > >> of
> > > >> > > mark it as won't fix since 0.10.0 will remove the checkpoint
> topic
> > > >> > > altogether.
> > > >> > > - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to
> include
> > > the
> > > >> fix
> > > >> > > that breaks Samza 0.10.0 build on Linux boxes
> > > >> > >
> > > >> > > Anything that I missed?
> > > >> > >
> > > >> > > Thanks!
> > > >> > >
> > > >> > > -Yi
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Jul 29, 2015 at 9:57 AM, Navina Ramesh <
> > > >> > > nram...@linkedin.com.invalid> wrote:
> > > >> > >
> > > >> > >> +1 for StreamAppender bug fix as a mandatory in 0.10.
> > > >> > >>
> > > >> > >> @Yan: will review SAMZA-676 today and test it out locally.
> > > >> > >>
> > > >> > >> Thanks!
> > > >> > >> Navina
> > > >> > >>
> > > >> > >> On Wed, Jul 29, 2015 at 8:38 AM, Chinmay Soman <
> > > >> > chinmay.cere...@gmail.com
> > > >> > >> >
> > > >> > >> wrote:
> > > >> > >>
> > > >> > >> > I can take care of SAMZA-340, SAMZA-683  and will follow up
> > with
> > > >> Luis
> > > >> > >> for
> > > >> > >> > SAMZA-401,2,3,4
> > > >> > >> >
> > > >> > >> > On Wed, Jul 29, 2015 at 12:10 AM, Dan  >
> > > >> wrote:
> > > >> > >> >
> > > >> > >> > > I agree SAMZA-741 for the ElasticSearch producer should be
> in
> > > >> too so
> > > >> > >> > we've
> > > >> > >> > >

Re: [DISCUSS] Release 0.10.0

2015-07-31 Thread Yi Pan 
Hi, Guozhang,

Good call. I moved SAMZA-697 out of 0.10.0. Boris is still working on that
and it is not clear yet whether this separation of class loaders will be
ready in 0.10.0.

Thanks!

-Yi

On Fri, Jul 31, 2015 at 10:12 AM, Guozhang Wang  wrote:

> Would SAMZA-697 worth adding to 0.10.0 as well?
>
> Guozhang
>
> On Thu, Jul 30, 2015 at 6:45 PM, Yan Fang  wrote:
>
> > For SAMZA-747, may ping Naveen or Chris. :)
> >
> > They have the permission to publish to maven. From the discuss
> > , they seem ready for
> the
> > release.
> >
> > Thanks,
> >
> > Fang, Yan
> > yanfang...@gmail.com
> >
> > On Thu, Jul 30, 2015 at 5:35 PM, Navina Ramesh
> >  > > wrote:
> >
> > > Ok. I think it got confusing because you are talking about tickets NOT
> to
> > > be included in the release :)
> > >
> > > Got it, now. +1 for this list of exclusions!
> > > SAMZA-723 (StreamAppender bug) and SAMZA-747 (rocksdb) should be in
> > 0.10.0.
> > > I think we don't have an ETA on the fix for SAMZA-747?
> > >
> > > Thanks!
> > > Navina
> > >
> > > On Thu, Jul 30, 2015 at 5:26 PM, Yi Pan  wrote:
> > >
> > > > Uh... wrong math all the day today... :(
> > > >
> > > > Let me re-try:
> > > >
> > > > 29/32 tickets are to be moved to later (i.e. excluded from 0.10.0).
> > > >
> > > > 2/32 tickets (including SAMZA-723) will be included
> > > >
> > > > 1/32 ticket (SAMZA-689) is up for discussion and I am leaning toward
> > mark
> > > > it as won't fix.
> > > >
> > > > -Yi
> > > >
> > > > On Thu, Jul 30, 2015 at 5:24 PM, Yi Pan  wrote:
> > > >
> > > > > Hi, Navina,
> > > > >
> > > > > The 29/30 tickets are to be excluded from 0.10.0 (i.e. moved to
> > > 0.11.0),
> > > > > the three tickets are either to be included in 0.10.0, or won't
> fix.
> > > > >
> > > > > -Yi
> > > > >
> > > > > On Thu, Jul 30, 2015 at 5:10 PM, Navina Ramesh <
> > > > > nram...@linkedin.com.invalid> wrote:
> > > > >
> > > > >> Hi Yi,
> > > > >> Thanks for summarizing. But why are we excluding SAMZA-723 from
> the
> > > > >> current
> > > > >> release ? Doesn't this break the existing StreamAppender
> > functionality
> > > > in
> > > > >> 0.9?
> > > > >>
> > > > >> Thanks!
> > > > >> Navina
> > > > >>
> > > > >> On Thu, Jul 30, 2015 at 4:55 PM, Yi Pan 
> > wrote:
> > > > >>
> > > > >> > Sorry, hit the send button too fast. Let me correct the summary
> > > > section:
> > > > >> >
> > > > >> > 29/32 tickets Status=open/reopen :
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> > > > >> >
> > > > >> > Three tickets excluded:
> > > > >> > - SAMZA-689, @Chinmay, were you able to re-produce it? I am
> > thinking
> > > > of
> > > > >> > mark it as won't fix since 0.10.0 will remove the checkpoint
> topic
> > > > >> > altogether.
> > > > >> > - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to
> include
> > > the
> > > > >> fix
> > > > >> > that breaks Samza 0.10.0 build on Linux boxes
> > > > >> > - SAMZA-723, stream appender deadlock issue
> > > > >> >
> > > > >> > Thanks!
> > > > >> >
> > > > >> > -Yi
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Jul 30, 2015 at 4:52 PM, Yi Pan 
> > > wrote:
> > > > >> >
> > > > >> > > Hi, all,
> > > > >> > >
> > > > >> > > Thanks a lot for helping out to select the features planned in
> > > > 0.10.0.
> > > > >> > >
> > > > >> > > Based on the above discussion, I am proposing to move the
> > > following
> > > > >> > > tickets later (i.e. 0.11.0).
> > > > >> > >
> > > > >> > > 30/32 tickets Status=open/reopen (exception for SAMZA-723:
> > stream
> > > > >> > appender
> > > > >> > > deadlock issue) :
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20fixVersion%20%3D%200.10.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> > > > >> > >
> > > > >> > > Two tickets excluded:
> > > > >> > > - SAMZA-689, @Chinmay, were you able to re-produce it? I am
> > > thinking
> > > > >> of
> > > > >> > > mark it as won't fix since 0.10.0 will remove the checkpoint
> > topic
> > > > >> > > altogether.
> > > > >> > > - SAMZA-747, we will have to upgrade to RocksDB 3.11.1 to
> > include
> > > > the
> > > > >> fix
> > > > >> > > that breaks Samza 0.10.0 build on Linux boxes
> > > > >> > >
> > > > >> > > Anything that I missed?
> > > > >> > >
> > > > >> > > Thanks!
> > > > >> > >
> > > > >> > > -Yi
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, Jul 29, 2015 at 9:57 AM, Navina Ramesh <
> > > > >> > > nram...@linkedin.com.invalid> wrote:
> > > > >> > >
> > > > >> > >> +1 for StreamAppender bug fix as a mandatory in 0.10.
> > > > >> > >>
> > > > >> > >> @Yan: will review SAMZA-676 today and test it out locally.
> > > > >> > >>
> > > > >> > >> Thanks!
> > >