Re: Apache Beam UI job creator

[Karan Kumar]

HI Juan and Jean
Thanks for the reply. We were looking to adopt an open source codebase. Any
pointers in that direction?


On Mon, Oct 8, 2018 at 9:05 PM Jean-Baptiste Onofré  wrote:

> Hi
>
> We have such tool at Talend (named datastreams), already available (beta)
> as Amazon ami.
>
> Regards
> JB
> Le 8 oct. 2018, à 12:24, Karan Kumar  a écrit:
>>
>> Hello
>>
>> We want to expose a GUI for our engineers/business analysts to create
>> real time pipelines using drag and drop constructs. Projects such as
>> https://github.com/TouK/nussknacker for flink and
>> https://github.com/hortonworks/streamline for storm match our
>> requirements.
>>
>> We wanted to understand if a UI job creator is on the road map for the
>> beam community or
>> if there are any projects which have taken a stab at solving this problem.
>>
>> --
>> Thanks
>> Karan
>>
>

-- 
Thanks
Karan

[Call for items] October Beam Newsletter

[Rose Nguyen]

Hi Beamers:

So much has been going on that it's time to sync up again in the October
Beam Newsletter [1]! :)

*Add the highlights from September to now (or planned events and talks)
that you want to share with the community by 10/14 11:59 p.m. **PDT.*

We will collect the notes via Google docs but send out the final version
directly to the user mailing list. If you do not know how to format
something, it is OK to just put down the info and I will edit. I'll ship
out the newsletter on 10/15.

[1]
https://docs.google.com/document/d/1KWk-pgq0_UR8PrJFstuRPb-dYtW4WspBwMJhfEPGYIM

-- 
Rose Thị Nguyễn

Re: SF Meetup(s)

[Austin Bennett]

Great!  Given the responses, seems a wealth of suitable locations.  It
sounds like this would be great to be a roaming meet up, to not be too tied
to SF or peninsula location nor a specific company/office.  I'll be in
touch with the individuals who responded to me (both on-list and off), with
aims get this off the ground in the not too distant future.

Especially @users: who is willing to speak and share what fantastic things
are getting accomplished with Beam!?

On Fri, Oct 5, 2018, 1:43 PM Ahmet Altay  wrote:

> I checked. We can host a meetup in Google's San Francisco or Sunnyvale
> offices. I can help with planning while Gris is out.
>
> On Fri, Oct 5, 2018 at 1:07 PM, Ahmet Altay  wrote:
>
>> Gris is out of office for a week I believe. We should be able to help
>> with hosting a meetup in Sunnyvale, not sure about the city. I will check
>> and update here.
>>
>> On Fri, Oct 5, 2018 at 1:02 PM, Thomas Weise  wrote:
>>
>>> Thanks for the initiative.
>>>
>>> Lyft may be able to help with hosting and I can help with talks. I will
>>> check and circle back.
>>>
>>> Thomas
>>>
>>>
>>> On Fri, Oct 5, 2018 at 8:48 AM Austin Bennett <
>>> whatwouldausti...@gmail.com> wrote:
>>>
 Hi All,

 Looking to start organizing events for Beam around San Francisco.  On
 the lookout for space -- anyone work for a company that could offer space
 around the city (my company offices are in Foster City, a wholly
 undesirable Meetup location)?

 Also, anyone Beam using that would be happy to share use cases?  And
 devs that could talk.about experiences building things, etc?

 I imagine will prompt more and more, to get off the ground.  This is
 just the beginning!

 Thanks,
 Austin



>>
>

Re: Apache Beam UI job creator

[Jean-Baptiste Onofré]

Hi

We have such tool at Talend (named datastreams), already available (beta) as 
Amazon ami.

Regards
JB

Le 8 oct. 2018 à 12:24, à 12:24, Karan Kumar  a écrit:
>Hello
>
>We want to expose a GUI for our engineers/business analysts to create
>real
>time pipelines using drag and drop constructs. Projects such as
>https://github.com/TouK/nussknacker for flink and
>https://github.com/hortonworks/streamline for storm match our
>requirements.
>
>We wanted to understand if a UI job creator is on the road map for the
>beam
>community or
>if there are any projects which have taken a stab at solving this
>problem.
>
>-- 
>Thanks
>Karan

Re: mapping 1k + columns to BQ row

[Satya Sheel]

Thank you for the response and the pointer, That make sense.  

Thanks,
Satyasheel 
 

> On 8 Oct 2018, at 16:04, Kenneth Knowles  wrote:
> 
> I think you are correct that your question is more related to Java. If you 
> don't want to have to author all of those POJO classes, then you may want to 
> simply not have such a layer, but do your processing on some JSON 
> representation directly (so you can just loop over the fields, etc). If you 
> do want the classes, then you can still loop over the fields somewhat by 
> using Java reflection. I think it is too much for me to say more about these 
> approaches here, so I hope you this helps you find what you want.
> 
> Kenn
> 
> On Mon, Oct 8, 2018 at 7:48 AM Satya Sheel  > wrote:
> Hi Kenn, 
> 
> Thank you for quick reply. What we are doing is first changing the XML to 
> Json and doing the processing. So, I am attaching a zip file which contains 
> small sample JSON, POJO for the JSON and a BQ row for the same. 
> 
> 
> Thanks, 
> Satyasheel 
> 
> 
> 
>> On 8 Oct 2018, at 15:08, Kenneth Knowles > > wrote:
>> 
>> Hi Satyasheel,
>> 
>> It may help if you provide a small piece of example XML, what the POJO looks 
>> like, and the row that you want to write to BigQuery.
>> 
>> Kenn
>> 
>> On Mon, Oct 8, 2018 at 6:39 AM Satya Sheel > > wrote:
>> Hi All, 
>> 
>> I am satyasheel working on a project which includes beam (dataflow as 
>> runner) to process streaming data from pubsub.
>> 
>> My question might sound noob here, so please bare with me. I am parsing a 
>> XML with more than 1000 unique tags and after some usual processing we are 
>> sinking it to BQ. The schema we have in BQ is nested and and some are 
>> repeated as well. The question I asking might be more related to JAVA. 
>> 
>> What I am doing write now is reading XML --> to POJO --> Some processing --> 
>> BQ table row. While converting PCollections to BQ table row I am writing 
>> huge no of setters which is quite manual and not efficient. I was wondering 
>> if the community has some trick for this. I know there is a method called 
>> JsonToRow but I am struggling to get some working example. 
>> 
>> Any help is appreciated. 
>> 
>> Regards,
>> Satyasheel
>> 
> 

Re: mapping 1k + columns to BQ row

[Kenneth Knowles]

I think you are correct that your question is more related to Java. If you
don't want to have to author all of those POJO classes, then you may want
to simply not have such a layer, but do your processing on some JSON
representation directly (so you can just loop over the fields, etc). If you
do want the classes, then you can still loop over the fields somewhat by
using Java reflection. I think it is too much for me to say more about
these approaches here, so I hope you this helps you find what you want.

Kenn

On Mon, Oct 8, 2018 at 7:48 AM Satya Sheel  wrote:

> Hi Kenn,
>
> Thank you for quick reply. What we are doing is first changing the XML to
> Json and doing the processing. So, I am attaching a zip file which contains
> small sample JSON, POJO for the JSON and a BQ row for the same.
>
>
> Thanks,
> Satyasheel
>
>
>
> On 8 Oct 2018, at 15:08, Kenneth Knowles  wrote:
>
> Hi Satyasheel,
>
> It may help if you provide a small piece of example XML, what the POJO
> looks like, and the row that you want to write to BigQuery.
>
> Kenn
>
> On Mon, Oct 8, 2018 at 6:39 AM Satya Sheel  wrote:
>
>> Hi All,
>>
>> I am satyasheel working on a project which includes beam (dataflow as
>> runner) to process streaming data from pubsub.
>>
>> My question might sound noob here, so please bare with me. I am parsing a
>> XML with more than 1000 unique tags and after some usual processing we are
>> sinking it to BQ. The schema we have in BQ is nested and and some are
>> repeated as well. The question I asking might be more related to JAVA.
>>
>> What I am doing write now is reading XML --> to POJO --> Some processing
>> --> BQ table row. While converting PCollections to BQ table row I am
>> writing huge no of *setters *which is quite manual and not efficient. I
>> was wondering if the community has some trick for this. I know there is a
>> method called *JsonToRow *but I am struggling to get some working
>> example.
>>
>> Any help is appreciated.
>>
>> Regards,
>> Satyasheel
>>
>>
>

Re: mapping 1k + columns to BQ row

[Satya Sheel]

Hi Kenn, Thank you for quick reply. What we are doing is first changing the XML to Json and doing the processing. So, I am attaching a zip file which contains small sample JSON, POJO for the JSON and a BQ row for the same. Thanks, Satyasheel <>
On 8 Oct 2018, at 15:08, Kenneth Knowles  wrote:Hi Satyasheel,It may help if you provide a small piece of example XML, what the POJO looks like, and the row that you want to write to BigQuery.KennOn Mon, Oct 8, 2018 at 6:39 AM Satya Sheel  wrote:Hi All, I am satyasheel working on a project which includes beam (dataflow as runner) to process streaming data from pubsub.My question might sound noob here, so please bare with me. I am parsing a XML with more than 1000 unique tags and after some usual processing we are sinking it to BQ. The schema we have in BQ is nested and and some are repeated as well. The question I asking might be more related to JAVA. What I am doing write now is reading XML --> to POJO --> Some processing --> BQ table row. While converting PCollections to BQ table row I am writing huge no of setters which is quite manual and not efficient. I was wondering if the community has some trick for this. I know there is a method called JsonToRow but I am struggling to get some working example. Any help is appreciated. Regards,Satyasheel

Re: FYI on Slack Channels

[Filip Popić]

filip.po...@gmail.com

On Mon, 8 Oct 2018 at 16:44, Jean-Baptiste Onofré  wrote:

> Hi
>
> If you don't have a apache.org e-mail address, you have to be invited.
>
> What's your email (to use for slack) ?
>
> Regards
> JB
> Le 8 oct. 2018, à 17:42, "Filip Popić"  a écrit:
>>
>> How can we join the slack workspace?
>>
>> https://the-asf.slack.com/signup  is accepting only apache emails and
>> the join link on
>>
>> https://beam.apache.org/community/contact-us/
>>
>> is not working anymore.
>>
>> On Wed, 27 Jun 2018 at 01:01, Jean-Baptiste Onofré 
>> wrote:
>>
>>> Great idea. Thanks !
>>>
>>> Regards
>>> JB
>>> Le 27 juin 2018, à 06:51, Griselda Cuevas  a écrit:

 This is awesome, thanks Rafael!




 On Tue, 26 Jun 2018 at 13:46, Scott Wegner  wrote:

> This is great, thanks Rafael!
>
> On Tue, Jun 26, 2018 at 6:45 AM Rafael Fernandez 
> wrote:
>
>> Ah! Didn't know -- thanks Romain!
>>
>> Done for all channels I could find. Also, here is a list of channels:
>>
>> #beam
>> #beam-events-meetups
>> #beam-go
>> #beam-java
>> #beam-portability
>> #beam-python
>> #beam-sql
>> #beam-testing
>>
>>
>> On Tue, Jun 26, 2018 at 1:18 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> +1 sounds very good
>>>
>>> side note: any channel must invite @asfarchivebot, I did it for the
>>> ones before "etc" but if you add others please ensure it is done
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau  |  Blog
>>>  | Old Blog
>>>  | Github
>>>  | LinkedIn
>>>  | Book
>>> 
>>>
>>>
>>> Le mar. 26 juin 2018 à 01:05, Lukasz Cwik  a
>>> écrit :
>>>
 +user@beam.apache.org 

 On Mon, Jun 25, 2018 at 4:04 PM Rafael Fernandez <
 rfern...@google.com> wrote:

> Hello!
>
> I took the liberty to create area-specific channels (such as
> #beam-java, #beam-python, #beam-go, etc.) As our project and community
> grows, I am seeing more and more "organic" interest groups forming -- 
> this
> may help us chat more online. If they don't, we can delete later.
>
> Any thoughts? (I am having second thoughts... #beam-go should
> probably be #beam-burrow ;p )
>
> Cheers,
> r
>

Re: FYI on Slack Channels

[Jean-Baptiste Onofré]

Hi

If you don't have a apache.org e-mail address, you have to be invited.

What's your email (to use for slack) ?

Regards
JB

Le 8 oct. 2018 à 17:42, à 17:42, "Filip Popić"  a écrit:
>How can we join the slack workspace?
>
>https://the-asf.slack.com/signup  is accepting only apache emails and
>the
>join link on
>
>https://beam.apache.org/community/contact-us/
>
>is not working anymore.
>
>On Wed, 27 Jun 2018 at 01:01, Jean-Baptiste Onofré 
>wrote:
>
>> Great idea. Thanks !
>>
>> Regards
>> JB
>> Le 27 juin 2018, à 06:51, Griselda Cuevas  a écrit:
>>>
>>> This is awesome, thanks Rafael!
>>>
>>>
>>>
>>>
>>> On Tue, 26 Jun 2018 at 13:46, Scott Wegner 
>wrote:
>>>
 This is great, thanks Rafael!

 On Tue, Jun 26, 2018 at 6:45 AM Rafael Fernandez
>
 wrote:

> Ah! Didn't know -- thanks Romain!
>
> Done for all channels I could find. Also, here is a list of
>channels:
>
> #beam
> #beam-events-meetups
> #beam-go
> #beam-java
> #beam-portability
> #beam-python
> #beam-sql
> #beam-testing
>
>
> On Tue, Jun 26, 2018 at 1:18 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> +1 sounds very good
>>
>> side note: any channel must invite @asfarchivebot, I did it for
>the
>> ones before "etc" but if you add others please ensure it is done
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>>
>
>>
>>
>> Le mar. 26 juin 2018 à 01:05, Lukasz Cwik  a
>écrit :
>>
>>> +user@beam.apache.org 
>>>
>>> On Mon, Jun 25, 2018 at 4:04 PM Rafael Fernandez
>
>>> wrote:
>>>
 Hello!

 I took the liberty to create area-specific channels (such as
 #beam-java, #beam-python, #beam-go, etc.) As our project and
>community
 grows, I am seeing more and more "organic" interest groups
>forming -- this
 may help us chat more online. If they don't, we can delete
>later.

 Any thoughts? (I am having second thoughts... #beam-go should
 probably be #beam-burrow ;p )

 Cheers,
 r

>>>

Re: FYI on Slack Channels

[Filip Popić]

How can we join the slack workspace?

https://the-asf.slack.com/signup  is accepting only apache emails and the
join link on

https://beam.apache.org/community/contact-us/

is not working anymore.

On Wed, 27 Jun 2018 at 01:01, Jean-Baptiste Onofré  wrote:

> Great idea. Thanks !
>
> Regards
> JB
> Le 27 juin 2018, à 06:51, Griselda Cuevas  a écrit:
>>
>> This is awesome, thanks Rafael!
>>
>>
>>
>>
>> On Tue, 26 Jun 2018 at 13:46, Scott Wegner  wrote:
>>
>>> This is great, thanks Rafael!
>>>
>>> On Tue, Jun 26, 2018 at 6:45 AM Rafael Fernandez 
>>> wrote:
>>>
 Ah! Didn't know -- thanks Romain!

 Done for all channels I could find. Also, here is a list of channels:

 #beam
 #beam-events-meetups
 #beam-go
 #beam-java
 #beam-portability
 #beam-python
 #beam-sql
 #beam-testing


 On Tue, Jun 26, 2018 at 1:18 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> +1 sounds very good
>
> side note: any channel must invite @asfarchivebot, I did it for the
> ones before "etc" but if you add others please ensure it is done
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
>
> Le mar. 26 juin 2018 à 01:05, Lukasz Cwik  a écrit :
>
>> +user@beam.apache.org 
>>
>> On Mon, Jun 25, 2018 at 4:04 PM Rafael Fernandez 
>> wrote:
>>
>>> Hello!
>>>
>>> I took the liberty to create area-specific channels (such as
>>> #beam-java, #beam-python, #beam-go, etc.) As our project and community
>>> grows, I am seeing more and more "organic" interest groups forming -- 
>>> this
>>> may help us chat more online. If they don't, we can delete later.
>>>
>>> Any thoughts? (I am having second thoughts... #beam-go should
>>> probably be #beam-burrow ;p )
>>>
>>> Cheers,
>>> r
>>>
>>

Re: Apache Beam UI job creator

[Juan Carlos Garcia]

I think (maybe i am wrong) but there is already a project within the Google
products that aim for this and still is on beta /alpha ... (i can
recall the name)

And personally i would definitely like to see something like this.

Karan Kumar  schrieb am Mo., 8. Okt. 2018, 11:24:

> Hello
>
> We want to expose a GUI for our engineers/business analysts to create real
> time pipelines using drag and drop constructs. Projects such as
> https://github.com/TouK/nussknacker for flink and
> https://github.com/hortonworks/streamline for storm match our
> requirements.
>
> We wanted to understand if a UI job creator is on the road map for the
> beam community or
> if there are any projects which have taken a stab at solving this problem.
>
> --
> Thanks
> Karan
>

Re: mapping 1k + columns to BQ row

[Kenneth Knowles]

Hi Satyasheel,

It may help if you provide a small piece of example XML, what the POJO
looks like, and the row that you want to write to BigQuery.

Kenn

On Mon, Oct 8, 2018 at 6:39 AM Satya Sheel  wrote:

> Hi All,
>
> I am satyasheel working on a project which includes beam (dataflow as
> runner) to process streaming data from pubsub.
>
> My question might sound noob here, so please bare with me. I am parsing a
> XML with more than 1000 unique tags and after some usual processing we are
> sinking it to BQ. The schema we have in BQ is nested and and some are
> repeated as well. The question I asking might be more related to JAVA.
>
> What I am doing write now is reading XML --> to POJO --> Some processing
> --> BQ table row. While converting PCollections to BQ table row I am
> writing huge no of *setters *which is quite manual and not efficient. I
> was wondering if the community has some trick for this. I know there is a
> method called *JsonToRow *but I am struggling to get some working
> example.
>
> Any help is appreciated.
>
> Regards,
> Satyasheel
>
>

mapping 1k + columns to BQ row

[Satya Sheel]

Hi All, 

I am satyasheel working on a project which includes beam (dataflow as runner) 
to process streaming data from pubsub.

My question might sound noob here, so please bare with me. I am parsing a XML 
with more than 1000 unique tags and after some usual processing we are sinking 
it to BQ. The schema we have in BQ is nested and and some are repeated as well. 
The question I asking might be more related to JAVA. 

What I am doing write now is reading XML --> to POJO --> Some processing --> BQ 
table row. While converting PCollections to BQ table row I am writing huge no 
of setters which is quite manual and not efficient. I was wondering if the 
community has some trick for this. I know there is a method called JsonToRow 
but I am struggling to get some working example. 

Any help is appreciated. 

Regards,
Satyasheel

Apache Beam UI job creator

[Karan Kumar]

Hello

We want to expose a GUI for our engineers/business analysts to create real
time pipelines using drag and drop constructs. Projects such as
https://github.com/TouK/nussknacker for flink and
https://github.com/hortonworks/streamline for storm match our requirements.

We wanted to understand if a UI job creator is on the road map for the beam
community or
if there are any projects which have taken a stab at solving this problem.

-- 
Thanks
Karan