回复： [DISCUSSION] Incubating Proposal of Uniffle

BLAST Wed, 25 May 2022 20:41:28 -0700

+1, Good Luck!




------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                            
                                            "general"                           
                                                         
<[email protected]&gt;;
发送时间:&nbsp;2022年5月25日(星期三) 下午2:26
收件人:&nbsp;"Incubator"<[email protected]&gt;;

主题:&nbsp;Re: [DISCUSSION] Incubating Proposal of Uniffle



+1 (binding)

An interesting project. Good luck!

Best,
tison.


Jungtaek Lim <[email protected]&gt; 于2022年5月25日周三 14:22写道：

&gt; +1 (non-binding)
&gt;
&gt; Good luck!
&gt;
&gt; On Wed, May 25, 2022 at 2:42 PM Daniel Widdis <[email protected]&gt; wrote:
&gt;
&gt; &gt; This was stated in the other thread: Unified/Universal Shuffle
&gt; &gt;
&gt; &gt; On 5/24/22, 10:04 PM, "XiaoYu" <[email protected]&gt; wrote:
&gt; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; Hi
&gt; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; Uniffle&nbsp; as a project name， What does he 
mean~
&gt; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; thanks
&gt; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; Weiwei Yang <[email protected]&gt; 
于2022年5月25日周三 12:57写道：
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; +1 (binding)
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; Good luck!
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; On Tue, May 24, 2022 at 8:49 PM Ye 
Xianjin <[email protected]&gt;
&gt; &gt; wrote:
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; +1 (non-binding).
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; Sent from my iPhone
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt; On May 25, 2022, at 9:59 AM, 
Goson zhang <
&gt; [email protected]&gt;
&gt; &gt; wrote:
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt; +1 (non-binding)
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt; Good luck!
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt; Daniel Widdis 
<[email protected]&gt; 于2022年5月25日周三 09:53写道：
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; +1 (non-binding) from 
me!&nbsp; Good luck!
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; On 5/24/22, 9:05 AM, 
"Jerry Shao" <[email protected]&gt; wrote:
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; Hi all,
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; Due to 
the name issue in thread (
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt; https://lists.apache.org/thread/y07xjkqzvpchncym9zr1hgm3c4l4ql0f),
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; we
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; figured 
out a new project name "Uniffle" and created a new
&gt; &gt; Thread.
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; Please
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; help to 
discuss.
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; We would 
like to propose Uniffle[1] as a new Apache
&gt; incubator
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; project,
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; you
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; can find 
the proposal here [2] for more details.
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; Uniffle 
is a high performance, general purpose Remote
&gt; &gt; Shuffle Service
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; for
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; 
distributed compute engines like Apache Spark
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; 
<https://spark.apache.org/&gt;, Apache
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; Hadoop 
MapReduce <https://hadoop.apache.org/&gt;, Apache
&gt; Flink
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; 
<https://flink.apache.org/&gt; and so on. We are aiming to
&gt; make
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; Firestorm a
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; 
universal shuffle service for distributed compute engines.
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; Shuffle 
is the key part for a distributed compute engine to
&gt; &gt; exchange
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; the
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; data 
between distributed tasks, the performance and
&gt; &gt; stability of
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; shuffle
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; will 
directly affect the whole job. Current “local file
&gt; &gt; pull-like
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; shuffle
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; style” 
has several limitations:
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1. Current shuffle is hard to 
support super large
&gt; &gt; workloads,
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; especially
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; in a high load environment, the 
major problem is IO
&gt; &gt; problem
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; (random
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; disk IO
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; issue, network congestion and 
timeout).
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2. Current shuffle is hard to 
deploy on the
&gt; disaggregated
&gt; &gt; compute
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; storage environment, as disk 
capacity is quite limited
&gt; on
&gt; &gt; compute
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; nodes.
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3. The constraint of storing 
shuffle data locally makes
&gt; &gt; it hard to
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; scale
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; elastically.
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; Remote 
Shuffle Service is the key technology for
&gt; enterprises
&gt; &gt; to build
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; big
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; data 
platforms, to expand big data applications to
&gt; &gt; disaggregated,
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; 
online-offline hybrid environments, and to solve above
&gt; &gt; problems.
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; The 
implementation of Remote Shuffle Service -&nbsp; “Uniffle”
&gt; -
&gt; &gt; is
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; heavily
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; adopted 
in Tencent, and shows its advantages in production.
&gt; &gt; Other
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; 
enterprises also adopted or prepared to adopt Firestorm in
&gt; &gt; their
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; 
environments.
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; 
Uniffle's key idea is brought from Salfish shuffle
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; <
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt;
&gt; &gt;
&gt; 
https://www.researchgate.net/publication/262241541_Sailfish_a_framework_for_large_scale_data_processing
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&gt; ,
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; it has 
several key design goals:
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1. High performance. Firestorm’s 
performance is close
&gt; &gt; enough to
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; local
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; file based shuffle style for small 
workloads. For large
&gt; &gt; workloads,
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; it is
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; far better than the current 
shuffle style.
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2. Fault tolerance. Firestorm 
provides high availability
&gt; &gt; for
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; Coordinated
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; nodes, and failover for Shuffle 
nodes.
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3. Pluggable. Firestorm is highly 
pluggable, which could
&gt; &gt; be suited
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; to
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; different compute engines, 
different backend storages,
&gt; and
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; different
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; wire-protocols.
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; We 
believe that Uniffle project will provide the great
&gt; value
&gt; &gt; for the
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; 
community if it is accepted by the Apache incubator.
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; I will 
help this project as champion and many thanks to
&gt; the 3
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; mentors:
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Felix Cheung 
([email protected])
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Junping du 
([email protected])
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Weiwei Yang ([email protected])
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Xun liu ([email protected])
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Zhankun Tang ([email protected])
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; [1] 
https://github.com/Tencent/Firestorm
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; [2]
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt; https://cwiki.apache.org/confluence/display/INCUBATOR/UniffleProposal
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; Best 
regards,
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;&nbsp;&nbsp;&nbsp; Jerry
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt; ---------------------------------------------------------------------
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; To unsubscribe, e-mail:
&gt; &gt; [email protected]
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt; For additional commands, 
e-mail:
&gt; &gt; [email protected]
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; &gt;&gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt;
&gt; &gt; ---------------------------------------------------------------------
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; To unsubscribe, e-mail: 
[email protected]
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt; For additional commands, e-mail:
&gt; [email protected]
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; &gt; &gt;
&gt; &gt;
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; 
---------------------------------------------------------------------
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; To unsubscribe, e-mail: 
[email protected]
&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; For additional commands, e-mail: 
[email protected]
&gt; &gt;
&gt; &gt;
&gt; &gt;
&gt; &gt;
&gt; &gt; ---------------------------------------------------------------------
&gt; &gt; To unsubscribe, e-mail: [email protected]
&gt; &gt; For additional commands, e-mail: [email protected]
&gt; &gt;
&gt; &gt;
&gt;

回复： [DISCUSSION] Incubating Proposal of Uniffle

Reply via email to