Thanks Flavio, I get most of your points except one
- Get the list of jobs contained in jar (ideally this is is true for every engine beyond Spark or Flink) Just curious to know how you submit job via rest api, if there're multiple jobs in one jar, then do you need to submit jar one time and submit jobs multiple times ? And is there any relationship between these jobs in the same jar ? Flavio Pompermaier <pomperma...@okkam.it> 于2019年7月23日周二 下午4:01写道： > Hi Jeff, the thing about the manifest is really about to have a way to > list multiple main classes in the jart (without the need to inspect every > Java class or forcing a 1-to-1 between jar and job like it is now). > My requirements were driven by the UI we're using in our framework: > > - Get the list of jobs contained in jar (ideally this is is true for > every engine beyond Spark or Flink) > - Get the list of required/optional parameters for each job > - Besides the optionality of a parameter, each parameter should > include an help description, a type (to validate the input param), a > default value and a set of choices (when there's a limited number of > options available) > - obviously the job serve should be able to submit/run/cancel/monitor > a job and upload/delete the uploaded jars > - the job server should not depend on any target platform dependency > (Spark or Flink) beyond the rest client: at the moment the rest client > requires a lot of core libs (indeed because it needs to submit the job > graph/plan) > - in our vision, the flink client should be something like Apache Livy > (https://livy.apache.org/) > - One of the biggest limitations we face when running a Flink job > from the REST API is the fact that the job can't do anything after > env.execute() while we need to call an external service to signal that the > job has ended + some other details > > Best, > Flavio > > On Tue, Jul 23, 2019 at 3:44 AM Jeff Zhang <zjf...@gmail.com> wrote: > >> Hi Flavio, >> >> Based on the discussion in the tickets you mentioned above, the >> program-class attribute was a mistake and community is intended to use >> main-class to replace it. >> >> Deprecating Program interface is a part of work of flink new client api. >> IIUC, your requirements are not so complicated. We can implement that in >> the new flink client api. How about listing your requirement, and let's >> discuss how we can make it in the new flink client api. BTW, I guess most >> of your requirements are based on your flink job server, It would be >> helpful if you could provide more info about your flink job server. Thanks >> >> >> >> Flavio Pompermaier <pomperma...@okkam.it> 于2019年7月22日周一 下午8:59写道： >> >>> Hi Tison, >>> we use a modified version of the Program interface to enable a web UI do >>> properly detect and run Flink jobs contained in a jar + their parameters. >>> As stated in [1], we dected multiple Main classes per jar by handling an >>> extra comma-separeted Manifest entry (i.e. 'Main-classes'). >>> >>> As mentioned on the discussion on the dev ML, our revised Program >>> interface looks like this: >>> >>> public interface FlinkJob { >>> String getDescription(); >>> List<FlinkJobParameter> getParameters(); >>> boolean isStreamingOrBatch(); >>> } >>> >>> public class FlinkJobParameter { >>> private String paramName; >>> private String paramType = "string"; >>> private String paramDesc; >>> private String paramDefaultValue; >>> private Set<String> choices; >>> private boolean mandatory; >>> } >>> >>> I've also opened some JIRA issues related to this topic: >>> >>> [1] https://issues.apache.org/jira/browse/FLINK-10864 >>> [2] https://issues.apache.org/jira/browse/FLINK-10862 >>> [3] https://issues.apache.org/jira/browse/FLINK-10879. >>> >>> Best, >>> Flavio >>> >>> >>> On Mon, Jul 22, 2019 at 1:46 PM Zili Chen <wander4...@gmail.com> wrote: >>> >>>> Hi guys, >>>> >>>> We want to have an accurate idea of how many people are implementing >>>> Flink job based on the interface Program, and how they actually >>>> implement it. >>>> >>>> The reason I ask for the survey is from this thread[1] where we notice >>>> this codepath is stale and less useful than it should be. As it is an >>>> interface marked as @PublicEvolving it is originally aimed at serving >>>> as user interface. Thus before doing deprecation or dropping, we'd like >>>> to see if there are users implementing their job based on this >>>> interface(org.apache.flink.api.common.Program) and if there is any, >>>> we are curious about how it is used. >>>> >>>> If little or none of Flink user based on this interface, we would >>>> propose deprecating or dropping it. >>>> >>>> I really appreciate your time and your insight. >>>> >>>> Best, >>>> tison. >>>> >>>> [1] >>>> https://lists.apache.org/thread.html/7ffc9936a384b891dbcf0a481d26c6d13b2125607c200577780d1e18@%3Cdev.flink.apache.org%3E >>>> >>> >>> >>> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > > -- Best Regards Jeff Zhang
