Flink job server does not have artifacts-dir option yet.
We have a PR to add it https://github.com/apache/beam/pull/9648
However, for now you can do a few manual steps to achieve this.
Start Job server.
1. Download
https://repo.maven.apache.org/maven2/org/apache/beam/beam-runners-flink-1.8-job-s
Kyle.
Thank you for the assistance.
Looks like "--artifacts-dir" is not parsed. Below is the DEBUG log of the
pipline.
WARNING:root:Discarding unparseable args: ['--flink_version=1.8',
'--flink_master_url=ip-172-31-12-113.ap-nor
The relevant configuration flag for the job server is `--artifacts-dir`.
@Robert Bradshaw I added this info to the log
message: https://github.com/apache/beam/pull/9646
Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com
On Mon, Sep 23, 2019 at 11:36 AM Robert Bradshaw
wr
Nice diagnosis, Kyle.
…was not going to think of that, thank you.
sudo lsof -i -n -P | grep 8081
firefox2018 mpatterson 115u IPv4 0xc79caf16386 0t0TCP
192.168.2.107:55582->18.209.126.59:8081 (ESTABLISHED)
firefox2018 mpatterson 234u IPv4 0xc79caf164d6
If there are no jobs or logs appearing in the Flink UI, it is likely that
the pipeline is not being submitted to the Flink cluster successfully.
(Right now, there is no timeout on pipeline submission, but I would like to
see that changed; see https://issues.apache.org/jira/browse/BEAM-7933.)
I wou
Kyle, right: saw that. Grabbing at anything.
The process on the client seems to submit, or at least not bomb, but the
cluster does not acknowledge a running job. It is possible I have not waited
long enough yet, and images are still being downloaded & launched.
2.16 section on FlinkRunner doe
"WARNING:root:No unique name set for transform ..." should not affect
the pipeline's ability to complete successfully. Is the pipeline
failing? If so, could you share more logs?
Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com
On Mon, Sep 23, 2019 at 11:43 AM Matthew Pat
Hey all,
we got yet another Apache Beam meetup group... but now in Belgium. We're
starting of with an introduction to Beam and some use-cases:
https://www.meetup.com/Belgium-Apache-Beam-Meetup/events/264933301/
If you're in the neighbourhood , join us for the first Belgian edition!
_/
_/ Alex
After setting runner, adding '--environment_type=LOOPBACK', I now get
“
…
WARNING:root:No unique name set for transform
fn/read/ref_PCollection_PCollection_17:0
WARNING:root:No unique name set for transform
fn/read/ref_PCollection_PCollection_17:0
WARNING:root:No unique name set for trans
You need to set your artifact directory to point to a
distributed filesystem that's also accessible to the workers (when starting
up the job server).
On Mon, Sep 23, 2019 at 5:43 AM Yu Watanabe wrote:
> Hello.
>
> I am working on flink runner (2.15.0) and would like to ask question about
> how t
Sorry: needed to add LOOPBACK
From: Matthew Patterson
Reply-To: "user@beam.apache.org"
Date: Monday, September 23, 2019 at 2:18 PM
To: "user@beam.apache.org"
Subject: Re: Word-count example
OK, now using release-2.15.0, calling word-count example with runner, after
building & calling
./gradl
OK, now using release-2.15.0, calling word-count example with runner, after
building & calling
./gradlew :runners:flink:1.5:job-server:runShadow (tried 1.8 as well)
, as
“f'--runner=PortableRunner', f'--job_endpoint=localhost:8099'
”
…and getting
File
"/Users/mpatterson/miniconda/envs/aws
Hello everyone!
If you are in the Seattle area please come to Beam meetup this Thursday,
September 26th - at 6pm in the Google office in Fremont. There will be
interesting talks, and there should be a number of Beam contributors and
users around. Also pizza and drinks.
The page with al the info:
There's also https://beam.apache.org/contribute/ptransform-style-guide/
(disclaimer:
I'm the main author) which is currently under "Technical docs" but probably
should be moved under "Patterns" instead.
On Mon, Sep 23, 2019 at 8:27 AM Chad Dombrova wrote:
> There are also these two helpful artic
There are also these two helpful articles on Dataflow patterns which are
largely applicable to Beam in general:
https://www.google.com/amp/s/cloudblog.withgoogle.com/products/gcp/guide-to-common-cloud-dataflow-use-case-patterns-part-1/amp/
https://www.google.com/amp/s/cloudblog.withgoogle.com/pro
Hello.
I am working on flink runner (2.15.0) and would like to ask question about
how to solve my error.
Currently , I have a remote cluster deployed as below . (please see slide1)
All master and worker nodes are installed on different server from apache
beam.
https://drive.google.com/file/d/1vB
16 matches
Mail list logo
