aicam commented on code in PR #6897:
URL: https://github.com/apache/texera/pull/6897#discussion_r4029305874
##########
amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/RegionExecutionManager.scala:
##########
@@ -352,6 +357,23 @@ class RegionExecutionManager(
)
}
+ /**
+ * Mount every repository this region's operators name before any of them
receives its code,
+ * which already refers to the mount paths. It happens here rather than in
a worker because
+ * workers are created before either phase launches. Operators carry no
mount code: they only
+ * name what they need, through `PhysicalOp.mountLocators`.
+ */
+ private def mountRegionRepositories(): Future[Unit] = {
+ val mount = Promise[Unit]()
+ if (mountFutureRef.compareAndSet(null, mount)) {
+ mount.become(Future {
+ val locators = region.getOperators.flatMap(_.mountLocators)
+ if (locators.nonEmpty) ensureMounted(locators)
Review Comment:
This is true, the following is from phase execution:
```
Future(())
.flatMap(_ => mountRegionRepositories())
.flatMap(_ => initExecutors(operatorsToRun, resourceConfig))
.flatMap(_ => assignPortsLogic())
.flatMap(_ => connectChannelsLogic())
.flatMap(_ => openOperators(operatorsToRun))
.flatMap(_ => startWorkersLogic())
.unit
```
Technically, mounting is ahead of any process and all other ones should wait
for it, the process is very quick so the latency can be ignored. There are two
problems that this order can cause:
1- If mounting fail, worst case it take 1.5 minute, which means user see
failure after 1.5min with no tuples processed
2- If user click on pause, it should wait for mounting process. This one is
not very critical because the control messages reach to region eventually.
Since mounting must happen before any operator in the region can start, we
can not skip this step or change its order. Regarding concern 1, if mounting is
working properly, we should not have any issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
