Hello, How feasible would it be to have spark speculatively increase the number of partitions when there is spare capacity in the system? We want to do this to increase to decrease application runtime. Initially, we will assume that function calls of the same type will have the same runtime (e.g. all maps take equal time) and that the runtime will scale linearly with the number of workers. If a numPartitions value is specified, we may increase beyond this, but if a Partitioner is specified, we would not change the number of partitions.
Some initial questions we had: * Does spark already do this? * Is there interest in supporting this functionality? * Are there any potential issues that we should be aware of? * What changes would need to be made for such a project? Thanks, Muhammed
