Hi! We are developing an application that is looking for new files on a folder, running a few Pig Scripts to prepare those files and, finally, loading them into our database.
The problem is that, for small files, the time that Pig / Tez / Yarn take to create a new application master and spawn new containers is way longer than the time it takes processing. Since Tez Sessions already allows a single Pig script to run multiple DAGs against the same application master, is there a way to reuse that application master and it´s containers for multiple Pig Scripts submissions ? Regards, Diego
