Re: BigQuery table backed by Google Sheet

This sounds like a Dataflow-specific question, so it might be better addressed by a StackOverflow question tagged with google-cloud-dataflow [1]. I'd suggest checking the Dataflow documentation for more information about its security model [2]. You'd likely have to use a service account, as descri

Re: Slack channel invite pls

sent On Thu, May 4, 2017 at 1:47 PM, Parker Coleman wrote: > Can I also get a slack invite? adinsx...@gmail.com > > On Thu, May 4, 2017 at 9:36 AM, Lukasz Cwik wrote: > >> Sent >> >> On Thu, May 4, 2017 at 9:32 AM, Seshadri Raghunathan >> wrote: >> >>> Hi , >>> >>> Please add me to the Slack

Re: Slack channel invite pls

Can I also get a slack invite? adinsx...@gmail.com On Thu, May 4, 2017 at 9:36 AM, Lukasz Cwik wrote: > Sent > > On Thu, May 4, 2017 at 9:32 AM, Seshadri Raghunathan > wrote: > >> Hi , >> >> Please add me to the Slack channel in the next possible cycle. >> >> sesh...@gmail.com >> >> Thanks, >>

Re: [HEADS UP] Using "new" filesystem layer

Thanks Luke, Gonna try and I will let you know. Regards JB On 05/04/2017 06:07 PM, Lukasz Cwik wrote: JB, for your second point it seems as though you may not be setting the Hadoop configuration on HadoopFileSystemOptions. Also, I just merged https://github.com/apache/beam/pull/2890 which will

Re: Http sink - Does it make sense?

Hi Seshadri, Yes RestIO sounds like what I'm looking for. Thanks for the link! Josh > On 4 May 2017, at 17:27, Seshadri Raghunathan wrote: > > Hi Josh, > > If you are looking for RestIO, I saw some discussions earlier in the group. > This JIRA has more details on it - > https://issues.apa

Re: KafkaIO nothing received?

Conrad, It does not look like there is a version incompatibility. You would see errors during during initialization otherwise. Log line "INFO: Reader-0: reading from test-http-logs-json-0 starting at offset 0" says it Kafka consumer was able to connect to servers. Does the the job just wait insid

Re: Slack invite

Sent On Thu, May 4, 2017 at 10:06 AM, Vinoth Chandar wrote: > Can you please add me to the slack group? > > vinoth at uber dot com > > Thanks, > Vinoth >

Slack invite

Can you please add me to the slack group? vinoth at uber dot com Thanks, Vinoth

Re: KafkaIO nothing received?

Hi, Ok, good to know I’m not going totally mad. I think I may have been running around in circles unnecessarily I am using kafka as part of an HDP installation (with Ambari). The Ambari interface is reporting my kafka version as 0.9.0.2.3 and indeed the output given previously give INFO: Kafka v

Re: KafkaIO nothing received?

@Conrad, Your code should be good to go, I can run it in my local env. There're two points you may have a check: 1). does the topic have data there, you can confirm with kafka cli ' *bin/kafka-console-consumer.sh*'; 2). is the port in bootstrapServers right? By default it's 9092. On Thu, May 4,

Re: Slack channel invite pls

Sent On Thu, May 4, 2017 at 9:32 AM, Seshadri Raghunathan wrote: > Hi , > > Please add me to the Slack channel in the next possible cycle. > > sesh...@gmail.com > > Thanks, > Seshadri >

Slack channel invite pls

Hi , Please add me to the Slack channel in the next possible cycle. sesh...@gmail.com Thanks, Seshadri

Re: Http sink - Does it make sense?

Hi Josh, If you are looking for RestIO, I saw some discussions earlier in the group. This JIRA has more details on it - https://issues.apache.org/jira/browse/BEAM-1946 Hope that helps ! Regards, Seshadri On Thu, May 4, 2017 at 9:19 AM, Josh wrote: > Hi all, > > I'm computing some counts in a

Http sink - Does it make sense?

Hi all, I'm computing some counts in a Beam job and want to sink the results to a REST API via HTTP. For example, group incoming elements by a key, count by key, and then every minute write the count to an API which supports incremental updates. Is this something that other people have done with

Re: Slack access?

invite sent.Sent from your iPhone-- Original --From: Conrad Crampton Date: 周五,5月 5,2017 00:06To: user@beam.apache.org Subject: Re: Slack access? Hi, Can I be added to the Slack channel please when next updated?   conrad.cramp...@secdata.com   Many thanks C

Re: [HEADS UP] Using "new" filesystem layer

JB, for your second point it seems as though you may not be setting the Hadoop configuration on HadoopFileSystemOptions. Also, I just merged https://github.com/apache/beam/pull/2890 which will auto detect Hadoop configuration based upon your HADOOP_CONF_DIR and YARN_CONF_DIR environment variables.

Slack access?

Hi, Can I be added to the Slack channel please when next updated? conrad.cramp...@secdata.com Many thanks Conrad SecureData, combating cyber threats __ The information contained in this mess

KafkaIO nothing received?

Hi, New to the group – ‘hello’! Just starting to look into Beam and I very much like the concepts, but have rather fallen at the first hurdle – that being trying to subscribe to a kafka topic and process results. Very simply the following code doesn’t get receive any records (the data is going

[HEADS UP] Using "new" filesystem layer

Hi guys, One of key refactoring/new feature we bring in the first stable release is the "new" Beam filesystems. I started to play with it on couple of use cases I have in beam-samples. 1/ TextIO.write() with unbounded PCollection (stream) The first use case is the TextIO write with unbounded