lachezar-n opened a new pull request, #2067:
URL: https://github.com/apache/systemds/pull/2067
Here are the bases for the Enumerators for Resource optimisation with the
first 2 enumerators:
- GridBasedEnumerator
- InterestBasedEnumerator
In addition to that, some utility classes/methods were created for
recompilation in the current case. It is still to be investigated how much
better this recompilation approach is compared to the existing one.
To allow the desired functionality, small changes were made to the following
classes of the existing SystemDS code base:
- src/main/java/org/apache/sysds/conf/CompilerConfig.java
* creating a new configuration type to avoid further changes that would
affect the basic functionality at script execution
- src/main/java/org/apache/sysds/hops/Hop.java
* the only place the new configuration type comes in place to allow
reassigning execution type
- src/main/java/org/apache/sysds/lops/compile/Dag.java
* resetUniqueMembers() - to be used at (re)compilation
-
src/main/java/org/apache/sysds/runtime/controlprogram/context/SparkExecutionContext.java
* initLocalSparkContext() to avoid real Spark context initialisation
-
src/main/java/org/apache/sysds/runtime/controlprogram/parfor/stat/InfrastructureAnalyzer.java
* allowing setting local parallelism from outside
Besides these required changes I have made a tiny change that looked to me
that could be useful for the SystemDS project overall:
- src/main/java/org/apache/sysds/hops/OptimizerUtils.java: replacing
`getNumMappers` and `getNumReducers` to `getNumTasks` since Spark and not
MapReduce is the new/current framework used for distributed execution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
