[
https://issues.apache.org/jira/browse/IGNITE-10619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16849660#comment-16849660
]
Maxim Muzafarov commented on IGNITE-10619:
------------------------------------------
[~agoncharuk], [~DmitriyGovorukhin]
The PR has been reworked. Please, see my comments below.
{quote}The current API looks too complex to me. Are there any reasons to
introduce our own Channel, ChannelId, IgniteSocketChannel, etc... ? Ignite is
not a framework to transfer files, so internal API should be kept as simple as
possible.
{quote}
You are right, this is my misunderstanding. All of these definitely must be
removed. I've kept {{CommunicationListenterEx}} as an extended internal
interface for listening SPIs channel connection events and only it.
{quote}I would suggest the following change to the GridIOManager:
{quote}
Alexey, from my point of view we should provide a single ChannelHandler
(Listener) per single different Ignite technical process (for instance, file
sending, resource deployment et.c), so my suggestion is a reusing
{{GridIoManager's}} topics machinery and add single handler per single topic.
According to your suggestion and current thoughts, the io API could look like
this:
{code:java}
public IgniteInternalFuture<Channel> openChannel(ClusterNode node, Object
topic, Message initMsg) throws IgniteCheckedException;
public void addChannelListener(Object topic, GridChannelListener lsnr);
interface GridChannelListener {
public void onOpened(UUID nodeId, Message initMsg, Channel channel);
}
{code}
What do you think?
The PR prepared according to the suggestions above.
{quote}The IgniteFileTransmitProcessor API also feels too redundant. The only
thing that needs to be provided on the supplier side is an input
stream/channel. I see no need in TransmitSessionFactory and sibling classes.
{quote}
I think for the Supplier/Demander it is better to provide a full transmission
file service, rather than a raw channel. The service can be reused by other
Ignites components and will support such important things like:
- upload\download speed limiting,
- transferring files by chunks with further continue from the last transmitted
part in case of unstable network connection.
The current API includes two different types of input channel handler
{{ChunkHandler}}, {{FileHandler}} and a {{TransmissionSessionHandler}} to
support a set of files to send to remote:
- The {{ChunkHandler}} can be used to handle reading incoming data from an
opened channel and process them `on the fly` (e.g. applying pages from delta
partition file to the local cache partition [Checkpointer: copy partition on
the
fly|http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Design-document-Rebalance-caches-by-transferring-partition-files-tp38388p38553.html]).
- The {{FileHandler}} is the default and used for handle copying file from one
node to another with the _zero-copy_ algorithm.
Would the service be better than raw channel?
> Add support files transmission between nodes over connection via
> CommunicationSpi
> ---------------------------------------------------------------------------------
>
> Key: IGNITE-10619
> URL: https://issues.apache.org/jira/browse/IGNITE-10619
> Project: Ignite
> Issue Type: Sub-task
> Components: persistence
> Reporter: Maxim Muzafarov
> Assignee: Maxim Muzafarov
> Priority: Major
> Labels: iep-28
>
> Partition preloader must support cache partition file relocation from one
> cluster node to another (the zero copy algorithm [1] assume to be used by
> default). To achieve this, the file transfer machinery must be implemented at
> Apache Ignite over Communication SPI.
> _CommunicationSpi_
> Ignite's Comminication SPI must support to:
> * establishing channel connections to the remote node to an arbitrary topic
> (GridTopic) with predefined processing policy;
> * listening incoming channel creation events and registering connection
> handlers on the particular node;
> * an arbitrary set of channel parameters on connection handshake;
> _FileTransmitProcessor_
> The file transmission manager must support to:
> * using different approaches of incoming data handling – buffered and direct
> (zero-copy approach of FileChannel#transferTo);
> * transferring data by chunks of predefined size with saving intermediate
> results;
> * re-establishing connection if an error occurs and continue file
> upload\download;
> * limiting connection bandwidth (upload and download) at runtime;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
