Leonidas Fegaras created MRQL-48:
------------------------------------
Summary: Fix global variable bindings in Spark and Flink modes
Key: MRQL-48
URL: https://issues.apache.org/jira/browse/MRQL-48
Project: MRQL
Issue Type: Bug
Affects Versions: 0.9.4
Reporter: Leonidas Fegaras
Assignee: Leonidas Fegaras
Priority: Critical
Attachments: MRQL-48.patch
The following patch fixes the bug in passing global bindings (such as the ones
set by the MRQL store statement) from master to worker nodes. It affects Spark
and Flink modes. It also fixes new bugs found in the Flink evaluator for
passing text file parsing information from master to workers. Couldn't use
Spark broadcast variables because these may not be broadcast to all workers at
all times (Spark may add workers later in the process). I think this is a Spark
bug. My solution was to pass the global environment as a serialized object from
master to workers during each DataSet operation. This works OK but may be
inefficient.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
