Hi Everyone, We need to deal with workflows on Spark. In our scenario, each workflow consists of multiple processing steps. Among different steps, there could be dependencies. I'm wondering if there are tools available that can help us schedule and manage workflows on Spark. I'm looking for something like pig on Hadoop, but it should fully function on Spark.
Any suggestion? Thanks in advance! Qiang
