Hi Team,
Is there any difference in running the spark or Flink runners from Colab vs
Local. The code runs with no issues in Google Colab environment but it does not
run on my local environment.
This is for windows.
Steps:
1. Start Flink or Spark on local machine
2. Make sure Spark and Flink runs on local machine
3. If Spark - start docker like this -- docker run -p 8099:8099 -p 8098:8098 -p
8097:8097 apache/beam_spark_job_server:latest
--spark-master-url=spark://localhost:7077
4. python "filename.py" should run but getting raise grpc.FutureTimeoutError()
Parameters as follows
SPARK:
options = PipelineOptions([
"--runner=PortableRunner",
"--job_endpoint=localhost:8099",
"--environment_type=LOOPBACK"
])
FLINK:
options = PipelineOptions([
"--runner=FlinkRunner",
"--flink_version=1.8",
"--flink_master=localhost:8081",
"--environment_type=LOOPBACK"
])
