Re: Beam Interview

2016-07-12 Thread Jesse Anderson 
Last call. If you want your words of wisdom forever kept in the annals of
Apache Beam lore, I'm publishing tomorrow (7-13) at 9 AM PT.

On Mon, Jul 11, 2016 at 11:13 PM Tyler Akidau  wrote:

> +1. Thanks a lot for putting this together. :-)
>
> On Mon, Jul 11, 2016 at 9:33 PM Frances Perry 
> wrote:
>
> > Love this, Jesse! And pretty inspired reading the answers so far ;-)
> >
> > On Mon, Jul 11, 2016 at 1:42 PM, Jesse Anderson 
> > wrote:
> >
> > > Thanks!
> > >
> > > On Mon, Jul 11, 2016 at 1:02 PM Ismaël Mejía 
> wrote:
> > >
> > > > Great Idea, I just added my answers, English is not my native
> language,
> > > so
> > > > feel free to edit if you find any grammatical mistakes, sorry.
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Jul 11, 2016 at 7:12 PM, Jesse Anderson <
> je...@smokinghand.com
> > >
> > > > wrote:
> > > >
> > > > > I really appreciate the turnout. I'm pleasantly surprised with the
> > > varied
> > > > > responses I've received.
> > > > >
> > > > > I plan to publish this post on July 13 at 9 AM PT. If you'd like to
> > add
> > > > > your input, please do it before that time.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jesse
> > > > >
> > > > > On Fri, Jul 8, 2016 at 1:30 PM Amit Sela 
> > wrote:
> > > > >
> > > > > > That's great Jesse!
> > > > > >
> > > > > > Added my comments.
> > > > > >
> > > > > > Thanks,
> > > > > > Amit
> > > > > >
> > > > > > On Fri, Jul 8, 2016 at 8:56 PM Shiv Shankar <
> > > > shiv.shivshan...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > I am a User and learner. I just added my view points.
> > > > > > >
> > > > > > > Thanks
> > > > > > > SV
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 8, 2016 at 1:51 AM, Sergio Fernández <
> > > wik...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Great idea!
> > > > > > > >
> > > > > > > > On Fri, Jul 8, 2016 at 7:44 AM, Jean-Baptiste Onofré <
> > > > > j...@nanthrax.net>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jesse,
> > > > > > > > >
> > > > > > > > > good idea. Just complete the doc.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > JB
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 07/08/2016 02:18 AM, Jesse Anderson wrote:
> > > > > > > > >
> > > > > > > > >> I've been thinking about ways to get more Beam information
> > out
> > > > > there
> > > > > > > > >> without too much fuss over getting everything right. I
> came
> > up
> > > > > with
> > > > > > a
> > > > > > > > >> written Q and A that represents the most common questions
> I
> > > get.
> > > > > > > > >>
> > > > > > > > >> Answering the questions should take 5-10 minutes. I think
> it
> > > > will
> > > > > > go a
> > > > > > > > >> long
> > > > > > > > >> ways towards getting more Beam users.
> > > > > > > > >>
> > > > > > > > >> 1. Here is the Google Doc link:
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1IQt6FfQI7W4d2QxZm6WwGnZFdA8JmaseKZrMGPu8zgY/edit#
> > > > > > > > >> 2. Add your name and initials.
> > > > > > > > >> 3. When you answer a question, just prefix it with
> your
> > > > > > initials.
> > > > > > > > >>
> > > > > > > > >> I really appreciate you taking the time to answer things.
> > I'll
> > > > > > publish
> > > > > > > > the
> > > > > > > > >> results of the Q and A on my blog and email out the link
> > once
> > > > it's
> > > > > > up
> > > > > > > > >> there.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >>
> > > > > > > > >> Jesse
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > > --
> > > > > > > > > Jean-Baptiste Onofré
> > > > > > > > > jbono...@apache.org
> > > > > > > > > http://blog.nanthrax.net
> > > > > > > > > Talend - http://www.talend.com
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Sergio Fernández
> > > > > > > > Partner Technology Manager
> > > > > > > > Redlink GmbH
> > > > > > > > m: +43 6602747925
> > > > > > > > e: sergio.fernan...@redlink.co
> > > > > > > > w: http://redlink.co
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: New apache_beam branch ?

2016-07-12 Thread Dan Halperin 
Hi JB,

Actually, this is a good time for a process question. How do we clean this
up -- do we have to file an INFRA ticket?

Presumably we can't/shouldn't be able to just push a deletion of the branch.

Thanks,
Dan

On Tue, Jul 12, 2016 at 9:39 AM, Jean-Baptiste Onofré 
wrote:

> Hi Silviu,
>
> thanks to the update !
>
> And no worries ;)
>
> Regards
> JB
>
>
> On 07/12/2016 05:30 PM, Silviu Calinoiu wrote:
>
>> This change was supposed to go into python-sdk branch. We will clean up
>> today.
>> I just noticed that the python-sdk branch did not have the change and was
>> scratching my head where did it go :-)
>> Thanks for noticing Jean-Baptiste!
>>
>> On Tue, Jul 12, 2016 at 7:51 AM, Jean-Baptiste Onofré 
>> wrote:
>>
>> Hi guys,
>>>
>>> I can see a new branch on the git repo:
>>>
>>> apache_beam
>>>
>>> The last commit on this branch is:
>>>
>>> commit cd3bcd51fa83e19b7d284d6394da98d1462c5732
>>> Merge: 73168b2 2809edc
>>> Author: Robert Bradshaw 
>>> Date:   Mon Jul 11 15:59:15 2016 -0700
>>>
>>>  Closes #629
>>>
>>>
>>> I wonder what's the purpose of this branch ? Is it a mistake ? If yes,
>>> then we can delete it.
>>>
>>> @Robert ?
>>>
>>> Thanks !
>>> Regards
>>> JB
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>>
>>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: New apache_beam branch ?

2016-07-12 Thread Jean-Baptiste Onofré 

Hi Silviu,

thanks to the update !

And no worries ;)

Regards
JB

On 07/12/2016 05:30 PM, Silviu Calinoiu wrote:

This change was supposed to go into python-sdk branch. We will clean up
today.
I just noticed that the python-sdk branch did not have the change and was
scratching my head where did it go :-)
Thanks for noticing Jean-Baptiste!

On Tue, Jul 12, 2016 at 7:51 AM, Jean-Baptiste Onofré 
wrote:


Hi guys,

I can see a new branch on the git repo:

apache_beam

The last commit on this branch is:

commit cd3bcd51fa83e19b7d284d6394da98d1462c5732
Merge: 73168b2 2809edc
Author: Robert Bradshaw 
Date:   Mon Jul 11 15:59:15 2016 -0700

 Closes #629


I wonder what's the purpose of this branch ? Is it a mistake ? If yes,
then we can delete it.

@Robert ?

Thanks !
Regards
JB
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com





--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: New apache_beam branch ?

2016-07-12 Thread Silviu Calinoiu 
This change was supposed to go into python-sdk branch. We will clean up
today.
I just noticed that the python-sdk branch did not have the change and was
scratching my head where did it go :-)
Thanks for noticing Jean-Baptiste!

On Tue, Jul 12, 2016 at 7:51 AM, Jean-Baptiste Onofré 
wrote:

> Hi guys,
>
> I can see a new branch on the git repo:
>
> apache_beam
>
> The last commit on this branch is:
>
> commit cd3bcd51fa83e19b7d284d6394da98d1462c5732
> Merge: 73168b2 2809edc
> Author: Robert Bradshaw 
> Date:   Mon Jul 11 15:59:15 2016 -0700
>
> Closes #629
>
>
> I wonder what's the purpose of this branch ? Is it a mistake ? If yes,
> then we can delete it.
>
> @Robert ?
>
> Thanks !
> Regards
> JB
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [jira] [Commented] (BEAM-434) When examples write output to file it creates many output files instead of one

2016-07-12 Thread Lukasz Cwik 
If we go with any option that restricts the number of outputs then in the
example we should discuss what it does and why it is not considered a good
thing.

On Tue, Jul 12, 2016 at 2:11 AM, Amit Sela (JIRA)  wrote:

>
> [
> https://issues.apache.org/jira/browse/BEAM-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372225#comment-15372225
> ]
>
> Amit Sela commented on BEAM-434:
> 
>
> I sort of prefer 2, but by letting the user pass the numShards
> configuration (which may need a better name)
> Like I mentioned in the PR, if we want to give a simple example result on
> one hand, while keeping in the user's mind the fact that multiple shards
> are a thing to consider, we could add a --numShards option and add it to
> the examples code with a default of 1 (or 3).
> If we want the users to know about multiple output shards, why should we
> keep the examples "pure" ?
>
> How about adding an option named "--numOutputShards" with default value 1
> (or 3, I could live with 3 :) ) and adding this to the examples README,
> thus giving a better experience in terms of "seeing" the output, while
> keeping the multiple-shards "on the table" and as a bonus, the Travis CI
> tests could still run with as many shards as we want (while I wanted
> examples to be easy enough, I definitely didn't want that for Travis!)
>
> WDYT ?
>
>
> > When examples write output to file it creates many output files instead
> of one
> >
> --
> >
> > Key: BEAM-434
> > URL: https://issues.apache.org/jira/browse/BEAM-434
> > Project: Beam
> >  Issue Type: Bug
> >  Components: examples-java
> >Reporter: Amit Sela
> >Assignee: Amit Sela
> >Priority: Minor
> >
> > When using `TextIO.Write.to("/path/to/output")` without any
> restrictions on the number of shards, it might generate many output files
> (depending on your input), for WordCount for example, you'll get as many
> output files as unique words in your input.
> > Since I think examples are expected to execute in a friendly manner to
> "see" what it does and not optimize for performance in some way, I suggest
> to use `withoutSharding()` when writing the example output to an output
> file.
> > Examples I could find that behave this way:
> > org.apache.beam.examples.WordCount
> > org.apache.beam.examples.complete.TfIdf
> > org.apache.beam.examples.cookbook.DeDupExample
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>