I understand what you mean, but what I want to express is whether it would be better for users to do this part of the work. Generally, companies have their own log collection system. If we were to do this part of the storage, the workload would be great, and the benefits would not be great.
我明白您的意思，但我想表达的是，这部分工作交给用户来做会不会更好一点，一般企业都会有自己的日志收集系统。如果我们来做这部分存储的话工作量会很大，并且收益并不是很可观。 Best wishes！ CalvinKirs On 11/3/2020 22:09，leon bao<[email protected]> wrote： @CalvinKirs about log spi , now have a requirement in scalable services(master/worker), this kind of application scenario requires that the task log cannot be stored in the worker / master, but needs to be stored in a third-party place, which maybe database or other storage. Therefore, if the dolphin scheduler can provide this plug-in function, users can read and write logs according to their own needs. 关于日志插件，我们现在有一个需求是可伸缩的服务(master/worker)，这种应用场景就要求任务的日志不能存在某一个worker/master上， 而是需要存在一个第三方的地方，可能是数据库或者其他存储。所以如果DolphinScheduler能提供这个插件功能，用户就可以根据自己需求来实现日志的读写. CalvinKirs <[email protected]> 于2020年11月3日周二 下午6:40写道： Great planning! But I have a little question, what are the specific requirements of the log SPI? I am not very clear at present, are we only implementing SPI for data storage? If this is the case, is it necessary? I think this user can use logagent (or other technologies) for related implementations. Different users have different needs. Some users may also involve aggregation, calculation, and even different magnitudes, and may use additional components. Therefore, if we store the original data in this piece, a lot of redundant data may be generated. 非常棒的规划！ 但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术） 来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。 Best wishes！ CalvinKirs On 11/3/2020 11:47，leon bao<[email protected]> wrote： Hello Everyone: DS has good horizontal scalability with its non central design architecture, which attracts many developers. With more and more users, the demand for scheduling is becoming more and more complex. At the same time, the functional design of DS is required to be more scalable，for example: the plug-in function of alarm mode. So we can discuss what parts of plug-ins DS can do at present. We can reconstruct DolphinScheduler according to the results of this discussion. At present, there are several parts of demand: - alert model: refer to: https://github.com/apache/incubator-dolphinscheduler/issues/3049 - task plugin: refer to: https://github.com/apache/incubator-dolphinscheduler/issues/2869 - register center: refer to: https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E https://github.com/apache/incubator-dolphinscheduler/issues/3961 - log model: The current log is recorded by writing local files of the server. Can we make this plug-in type, which can facilitate users to extend the log reading and writing types, such as writing to the database or other third-party systems. - global task queue At present, tasks are stored in the memory queue of the master, which results in the priority of a task can only work within the scope of a master. In order to make the priority of a task effective globally, we need a global queue to make the global priority work. (in version 1.2, we used zookeeper as the global queue, which was removed because of the delay of ZK operation) Implementation details can be discussed within each topic. Here, we only discuss the requirements. Very appreciate you can put forward more opinions. ================================================================================================================================================================== DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性 比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。 目前已经有需求的几个部分： - 告警插件(running) 相关讨论： https://github.com/apache/incubator-dolphinscheduler/issues/3049 - 任务插件 https://github.com/apache/incubator-dolphinscheduler/issues/2869 - 注册中心 相关讨论： https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E https://github.com/apache/incubator-dolphinscheduler/issues/3961 - 日志插件 目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。 - 全局队列插件 目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。 实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。 -- DolphinScheduler(Incubator) PPMC BaoLiang 鲍亮 [email protected] -- DolphinScheduler(Incubator) PPMC BaoLiang 鲍亮 [email protected]
