File resource plug-in, the current DS file resource storage supports HDFS, S3 and NONE, making plug-in allows users to customize storage methods to meet various storage needs ====================================================================
文件资源插件化，目前DS的文件资源存储是支持HDFS，S3 和 NONE，做成插件化可以使得用户可自定义存储方式，满足用户个性化存储需求 On 2020/11/03 10:40:30, CalvinKirs <[email protected]> wrote: > > > Great planning! > But I have a little question, what are the specific requirements of the log > SPI? I am not very clear at present, are we only implementing SPI for data > storage? If this is the case, is it necessary? I think this user can use > logagent (or other technologies) for related implementations. Different users > have different needs. Some users may also involve aggregation, calculation, > and even different magnitudes, and may use additional components. Therefore, > if we store the original data in this piece, a lot of redundant data may be > generated. > > > 非常棒的规划！ > 但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术） > > 来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。 > > > Best wishes！ > CalvinKirs > > > On 11/3/2020 11:47，leon bao<[email protected]> wrote： > Hello Everyone: > > DS has good horizontal scalability with its non central design > architecture, which attracts many developers. With more and more users, the > demand for scheduling is becoming more and more complex. > At the same time, the functional design of DS is required to be more > scalable，for example: the plug-in function of alarm mode. > So we can discuss what parts of plug-ins DS can do at present. We can > reconstruct DolphinScheduler according to the results of this discussion. > At present, there are several parts of demand: > > - alert model: > refer to: > https://github.com/apache/incubator-dolphinscheduler/issues/3049 > > - task plugin: > refer to: > https://github.com/apache/incubator-dolphinscheduler/issues/2869 > > - register center: > refer to: > > https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E > https://github.com/apache/incubator-dolphinscheduler/issues/3961 > > - log model: > The current log is recorded by writing local files of the server. > Can we make this plug-in type, which can facilitate users to extend the > log reading and writing types, such as writing to the database or other > third-party systems. > > - global task queue > At present, tasks are stored in the memory queue of the master, which > results in the priority of a task can only work within the scope of a > master. > In order to make the priority of a task effective globally, we need a > global queue to make the global priority work. > (in version 1.2, we used zookeeper as the global queue, which was removed > because of the delay of ZK operation) > > Implementation details can be discussed within each topic. Here, we only > discuss the requirements. > Very appreciate you can put forward more opinions. > > ================================================================================================================================================================== > > DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性 > 比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。 > 目前已经有需求的几个部分： > > - 告警插件(running) > 相关讨论： > https://github.com/apache/incubator-dolphinscheduler/issues/3049 > > - 任务插件 > https://github.com/apache/incubator-dolphinscheduler/issues/2869 > > - 注册中心 > 相关讨论： > > https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E > https://github.com/apache/incubator-dolphinscheduler/issues/3961 > > - 日志插件 > 目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。 > > - 全局队列插件 > > 目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。 > > 实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。 > > -- > DolphinScheduler(Incubator) PPMC > BaoLiang 鲍亮 > [email protected] >
