File resource SPI , the current DS file resource storage supports HDFS, S3 and
NONE, making plug-in allows users to customize storage methods to meet various
storage needs
And the task spi should be the highest priority
===============================================================================================================
????????????????????DS????????????????????HDFS??S3 and
NONE??????????????????????????????????????????????????????????
Task SPI ??????????????????????
------------------ ???????? ------------------
??????:
"dev"
<[email protected]>;
????????: 2020??11??4??(??????) ????10:20
??????: "dev"<[email protected]>;
????: Re:Re: [DISCUSS]Which part of DolphinScheduler can be more scalable?
log SPI +1
?? 2020-11-04 10:07:58??"JUN GAO" <[email protected]> ??????
>log SPI +1
>
>But I think this is a low priority task, we should give priority to
>complete the registration center SPI and task SPI.
>
>
>
>CalvinKirs <[email protected]> ??2020??11??3?????? ????11:04??????
>
>> Thank you very much for your answer, I misunderstood what you meant.
>>
>>
>> log SPI+1
>>
>>
>> Best wishes??
>> CalvinKirs
>>
>>
>> On 11/3/2020 22:55??leon bao<[email protected]> wrote??
>> Maybe it didn't express my meaning well.
>> Yes, DolphinScheduler doesn't need to do this storage. We just need to
do
>> the log plug-in function to support users to implement their own log
>> storage mode, just like we do alert plug-ins.
>>
>>
>> ????????????????????????????
>>
??????DolphinScheduler??????????????????????????????????????????????????????????????????????????????????????????????????????
>>
>>
>> boyi <[email protected]> ??2020??11??3?????? ????10:37??????
>>
>> hi:
>>
>>
>> Agree with Calvin KIRs
>>
>>
>>
>>
>>
>>
>> Generally, there will be a unified log collection for such files, such
as
>> ELK. Especially when deployed in the docker environment, we can
consider
>> focusing on more important things
>>
>>
>> --------------------------------------
>> BoYi ZhangE-mail : [email protected]
>> On 11/3/2020 22:23??CalvinKirs<[email protected]> wrote??
>> I understand what you mean, but what I want to express is whether it
would
>> be better for users to do this part of the work. Generally, companies
have
>> their own log collection system. If we were to do this part of the
storage,
>> the workload would be great, and the benefits would not be great.
>>
>>
>>
>>
>>
??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
>>
>>
>> Best wishes??
>> CalvinKirs
>>
>>
>> On 11/3/2020 22:09??leon bao<[email protected]> wrote??
>> @CalvinKirs
>>
>> about log spi , now have a requirement in scalable
>> services(master/worker), this kind of application scenario requires
that
>> the task log cannot be stored in the worker / master, but needs to be
>> stored in a third-party place, which maybe database or other
>> storage. Therefore, if the dolphin scheduler can provide this plug-in
>> function, users can read and write logs according to their own needs.
>>
>>
>>
??????????????????????????????????????????????(master/worker)????????????????????????????????????????????worker/master????
>>
>>
>>
????????????????????????????????????????????????????????????????DolphinScheduler??????????????????????????????????????????????????????????.
>>
>> CalvinKirs <[email protected]> ??2020??11??3?????? ????6:40??????
>>
>>
>>
>> Great planning!
>> But I have a little question, what are the specific requirements of the
>> log SPI? I am not very clear at present, are we only implementing SPI
for
>> data storage? If this is the case, is it necessary? I think this user
can
>> use logagent (or other technologies) for related implementations.
Different
>> users have different needs. Some users may also involve aggregation,
>> calculation, and even different magnitudes, and may use additional
>> components. Therefore, if we store the original data in this piece, a
lot
>> of redundant data may be generated.
>>
>>
>> ??????????????
>>
>>
>>
??????????????????????SPI????????????????????????????????????????????????????????????SPI????????????????????????????????????????????????????????????????logagent????????????????
>>
>>
>>
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
>>
>>
>> Best wishes??
>> CalvinKirs
>>
>>
>> On 11/3/2020 11:47??leon bao<[email protected]> wrote??
>> Hello Everyone:
>>
>> DS has good horizontal scalability with its non central design
>> architecture, which attracts many developers. With more and more
users, the
>> demand for scheduling is becoming more and more complex.
>> At the same time, the functional design of DS is required to be more
>> scalable??for example: the plug-in function of alarm mode.
>> So we can discuss what parts of plug-ins DS can do at present.
We can
>> reconstruct DolphinScheduler according to the results of this
discussion.
>> At present, there are several parts of demand:
>>
>> - alert model:
>> refer to:
>> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>>
>> - task plugin:
>> refer to:
>> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>>
>> - register center:
>> refer to:
>>
>>
>>
>>
>>
https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
>> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>>
>> - log model:
>> The current log is recorded by writing local files of the server.
>> Can we make this plug-in type, which can facilitate users to extend the
>> log reading and writing types, such as writing to the database or other
>> third-party systems.
>>
>> - global task queue
>> At present, tasks are stored in the memory queue of the master, which
>> results in the priority of a task can only work within the scope of a
>> master.
>> In order to make the priority of a task effective globally, we need a
>> global queue to make the global priority work.
>> (in version 1.2, we used zookeeper as the global queue, which was
removed
>> because of the delay of ZK operation)
>>
>> Implementation details can be discussed within each topic. Here, we
only
>> discuss the requirements.
>> Very appreciate you can put forward more opinions.
>>
>>
>>
>>
>>
==================================================================================================================================================================
>>
>>
>>
>>
>>
DS??????????????????????????????????????????????????????????????????????????????DS????????????????????????????????????????????????DS????????????????????????????
>>
??????????????????????????????????????????????????DS????????????????????????????????????????????????????????DS????????????????????
>> ??????????????????????????
>>
>> - ????????(running)
>> ??????????
>> https://github.com/apache/incubator-dolphinscheduler/issues/3049
>>
>> - ????????
>> https://github.com/apache/incubator-dolphinscheduler/issues/2869
>>
>> - ????????
>> ??????????
>>
>>
>>
>>
>>
https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E
>> https://github.com/apache/incubator-dolphinscheduler/issues/3961
>>
>> - ????????
>>
??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
>>
>> - ????????????
>>
>>
>>
>>
>>
??????????????????master??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????(????1.2??????????????zookeeper????????????,????????zk??????????????????????????)??
>>
>>
????????????????????????????????????????????????????????????????????????????????????
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> BaoLiang ????
>> [email protected]
>>
>>
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> BaoLiang ????
>> [email protected]
>>
>>
>>
>> --
>> DolphinScheduler(Incubator) PPMC
>> BaoLiang ????
>> [email protected]
>>
>
>
>--
>
>DolphinScheduler(Incubator) PPMC
>Jun Gao ????
>[email protected]
