M
To: user@beam.apache.org
Subject: RE: kafkaIO Run with Spark Runner: "streaming-job-executor-0"
Dear all,
The related version in my running env:
Beam 2.4.0 (Spark runner with Standalone mode)
Spark 2.0.0
Kafka: 2.11-0.10.1.1
, and the setting of Pom.xml as mentioned earlier.
When we run the follo
UI_PORT=1234
Spark-default.conf
spark.driver.memory 10g
spark.executor.memory2g
spark.executor.instances 4
Rick
-Original Message-----
From: Ismaël Mejía [mailto:ieme...@gmail.com]
Sent: Wednesday, June 13, 2018 11:35 PM
To: user@beam.apache.org<mailto:user@
-Original Message-
From: Ismaël Mejía [mailto:ieme...@gmail.com]
Sent: Wednesday, June 13, 2018 11:35 PM
To: user@beam.apache.org
Subject: Re: kafkaIO Run with Spark Runner: "streaming-job-executor-0"
Can you please update the version of Beam to at least version 2.2.0.
Can you please update the version of Beam to at least version 2.2.0.
There were some important fixes in streaming after the 2.0.0 release
so this could be related. Ideally you should use the latest released
version (2.4.0). Remember that starting with Beam 2.3.0 the Spark
runner is based on Spark 2
Can you check the logs on the worker?
On Wed, Jun 13, 2018 at 2:26 AM wrote:
> Dear all,
>
>
>
> I am using the kafkaIO in my project (Beam 2.0.0 with Spark runner).
>
> My running environment is:
>
> OS: Ubuntn 14.04.3 LTS
>
> The different version for these tools:
>
> JAVA: JDK 1.8
>
> Beam 2.
