Re: Google Scheduler with Python Dataflow Pipelines

Hi, Fernando, I have never tried with Python. For Java, we do the following in a scheduled Jenkins Job: export GOOGLE_APPLICATION_CREDENTIALS=... java -cp $artifact_location \ $full_qualified_class_name

Re: Encoding Problem: Kafka - DataFlow

Thanks, Eddie. Just to add to the discussion, I logged the following information: Charset.defaultCharset(): US-ASCII System.getProperty("file.encoding"): ANSI_X3.4-1968 OutputStreamWriter writer = new OutputStreamWriter(new ByteArrayOutputStream()); writer..getEncoding(): ASCII In our case, a

Re: Help in reading hadoop HDFS file into PCollection

Hi, Maybe you could share some code, so we could have a better picture of what is going on. Last time I had to read from HDFS (normally in our pipelines HDFS is just a sink), we used FileIO: https://beam.apache.org/documentation/sdks/javadoc/2.3.0/index.html?org/apache/beam/sdk/io/FileIO.htm