hansva commented on code in PR #9358:
URL: https://github.com/apache/hop/pull/9358#discussion_r4537067707
##########
plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/MailConnection.java:
##########
@@ -962,8 +962,9 @@ public void moveMessage() throws HopException {
// move all messages
this.folder.copyMessages(new Message[] {this.message},
this.destinationIMAPFolder);
updatedMovedMessagesCounter();
- // Make sure to delete messages
- deleteMessage();
+ // Flag the original as deleted. Do not call deleteMessage(): a moved
message only counts
+ // as moved, not also as deleted.
Review Comment:
`moveMessage()` no longer adds to `nrDeletedMessages`, which changes the
public Lines deleted result for existing workflows that move messages. A
workflow that checks `previous_result.getNrLinesDeleted() > 0` after a move
used to get N and now gets 0. If this is intended, it deserves a note in the
upgrade/release notes, not only the doc edit.
##########
plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/MailConnection.java:
##########
@@ -1063,7 +1064,7 @@ private void updatedMovedMessagesCounter() {
/** Set count of moved messages. */
private void setMovedMessagesCounter() {
- this.nrMovedMessages = getMessagesCount();
+ this.nrMovedMessages += getMessagesCount();
Review Comment:
Now that the moved/deleted counters are summed over all folders, this can
over-count: with Include subfolders the folder list can contain the move target
itself. E.g. INBOX, Include subfolders = Y, move to `INBOX.sub` (the setup of
the new IT): INBOX moves 3 messages into INBOX.sub, then INBOX.sub is processed
as a subfolder and its 4 messages are copied into themselves, so
`nrMovedMessages` = 7 for 3 real moves, and the server may end up with
duplicate copies. Could the move target (and its children) be excluded from the
folders being processed?
##########
plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/ActionGetPOP.java:
##########
@@ -903,7 +910,8 @@ void fetchOneFolder(
}
if (isSaveMessage()) {
// get local message filename
- String localfilenameMessage =
replaceTokens(realFilenamePattern, i);
+ String localfilenameMessage =
+ replaceTokens(realFilenamePattern,
savedMessageIndex++);
Review Comment:
Same root cause on the get side: with Include subfolders and 'Move message
to folder' after retrieval into a subfolder of the source, each moved message
is fetched again from that subfolder and saved with a new `#IdFile#`. INBOX
with 3 messages, move to INBOX.sub: the INBOX pass saves msg_1..3 and moves
them, the INBOX.sub pass saves the same messages again as msg_4..6, so 6 files
and Lines written = 6 for 3 messages. Before, the duplicates at least overwrote
msg_1..3.
##########
plugins/misc/mail/src/main/java/org/apache/hop/mail/metadata/MailServerConnection.java:
##########
@@ -636,7 +636,7 @@ private void updateDeletedMessagesCounter() {
}
private void setDeletedMessagesCounter() {
- this.nrDeletedMessages = getMessagesCount();
+ this.nrDeletedMessages += getMessagesCount();
Review Comment:
These counter and `moveMessage` fixes are made by hand in both
`MailServerConnection` and `getpop/MailConnection`, each with its own tests,
and the IT runs every scenario twice to cover both. Every future change to
move/delete counting will have to be made twice. Could this go into one shared
helper both classes use?
##########
plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/MailConnection.java:
##########
@@ -962,8 +962,9 @@ public void moveMessage() throws HopException {
// move all messages
this.folder.copyMessages(new Message[] {this.message},
this.destinationIMAPFolder);
updatedMovedMessagesCounter();
Review Comment:
The moved counter is incremented before `setFlag(DELETED)`. If flagging
fails (read-only folder, dropped connection) after `copyMessages` succeeded,
the message is counted as moved, an exception is thrown, and the original is
still in the source folder. Incrementing after the flag succeeds would keep the
count honest.
##########
integration-tests/mail/metadata/MailServerConnection/james-imap-folders.json:
##########
@@ -0,0 +1,17 @@
+{
Review Comment:
The metadata key is `mail-server-connection` now (with
`MailServerConnection` only in `legacyKeys` for old objects), so new objects
should go in `metadata/mail-server-connection/`. A file in the legacy folder
still loads, but it keeps the old layout going and moves the first time it's
saved.
##########
plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/ActionGetPOP.java:
##########
@@ -222,6 +222,12 @@ public class ActionGetPOP extends ActionBase implements
Cloneable, IAction {
private MailServerConnection connection;
private MailConnection mailConn;
+ /**
+ * Number of messages saved so far in this run, over all folders. Replaces
#IdFile# so a message
+ * of a subfolder does not overwrite one of an earlier folder.
+ */
+ private int savedMessageIndex;
Review Comment:
`savedMessageIndex` tracks the same thing as the connection's saved-messages
counter (`getSavedMessagesCounter()`, incremented in
`saveMessageContentToFile`). Having both means two sources of truth that can
drift (this field is bumped before the save, the counter after it succeeds).
`replaceTokens(realFilenamePattern, connection.getSavedMessagesCounter())`
should give the same numbering without the extra field and the reset in
`execute()`.
##########
integration-tests/mail/main-0010-get-mails-subfolders.hwf:
##########
@@ -0,0 +1,852 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<workflow>
+ <name>main-0010-get-mails-subfolders</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>Get mails over a folder and its subfolder: unique #IdFile#,
move and delete counters (#9208)</description>
+ <extended_description/>
+ <workflow_version/>
+ <created_user>-</created_user>
+ <created_date>2026/11/07 12:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/11/07 12:00:00.000</modified_date>
+ <parameters>
+ </parameters>
+ <actions>
+ <action>
+ <name>Start</name>
+ <description/>
+ <type>SPECIAL</type>
+ <attributes/>
+ <DayOfMonth>1</DayOfMonth>
+ <hour>12</hour>
+ <intervalMinutes>60</intervalMinutes>
+ <intervalSeconds>0</intervalSeconds>
+ <minutes>0</minutes>
+ <repeat>N</repeat>
+ <schedulerType>0</schedulerType>
+ <weekDay>1</weekDay>
+ <parallel>N</parallel>
+ <xloc>64</xloc>
+ <yloc>64</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>prepare mailbox (inline)</name>
+ <description/>
+ <type>SHELL</type>
+ <attributes/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <arg_from_previous>N</arg_from_previous>
+ <arguments>
+</arguments>
+ <exec_per_row>N</exec_per_row>
+ <insertScript>Y</insertScript>
+ <loglevel>Basic</loglevel>
+ <script>#!/bin/bash
+# Fresh mailbox for [email protected]: INBOX with 4 messages and an empty
INBOX.sub.
+WEBADMIN="http://${MAIL_SMTP_SERVER}:8000"
+MAILUSER="[email protected]"
+OUT="${PROJECT_HOME}/output/0010-inline"
+
+ready=0
+for attempt in $(seq 1 60); do
+ if curl -sf -o /dev/null --max-time 2 "$WEBADMIN/domains"; then
+ ready=1
+ break
+ fi
+ sleep 2
+done
+if [ "$ready" != 1 ]; then
+ echo "James WebAdmin did not accept connections on port 8000"
+ exit 1
+fi
+
+curl -s -XPUT "$WEBADMIN/domains/example.com"
+curl -s -XPUT "$WEBADMIN/users/$MAILUSER" -d '{"password":"password"}' -H
"Content-Type: application/json"
+curl -sf -XDELETE "$WEBADMIN/users/$MAILUSER/mailboxes" || exit 1
+curl -sf -XPUT "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX" || exit 1
+curl -sf -XPUT "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX.sub" || exit 1
+
+rm -rf "$OUT"
+mkdir -p "$OUT"
+
+for n in 1 2 3 4; do
+ printf 'From: [email protected]\r\nTo: %s\r\nSubject: Message
%s\r\n\r\nBody of message %s\r\n' "$MAILUSER" "$n" "$n" \
+ | curl -sS --url "smtp://${MAIL_SMTP_SERVER}:25" --mail-from
[email protected] --mail-rcpt "$MAILUSER" -T - || exit 1
+done
+
+# Delivery is asynchronous: wait until all 4 messages are in INBOX
+for attempt in $(seq 1 60); do
+ count=$(curl -s "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX/messageCount")
+ if [ "$count" = 4 ]; then
+ echo "INBOX of $MAILUSER holds 4 messages"
+ exit 0
+ fi
+ sleep 1
+done
+echo "Expected 4 messages in INBOX of $MAILUSER, found $count"
+exit 1
+</script>
+ <set_append_logfile>N</set_append_logfile>
+ <set_logfile>N</set_logfile>
+ <parallel>N</parallel>
+ <xloc>240</xloc>
+ <yloc>64</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>move first mail to INBOX.sub (inline)</name>
+ <description/>
+ <type>GET_POP</type>
+ <attributes/>
+ <password>Encrypted 2be98afc86aa7f2e4bb18bd63c99dbdde</password>
+ <servername>${MAIL_SMTP_SERVER}</servername>
+ <sslport>993</sslport>
+ <username>[email protected]</username>
+ <usessl>Y</usessl>
+ <attachmentfolder/>
+ <attachmentwildcard/>
+ <bodysearch/>
+ <createlocalfolder>Y</createlocalfolder>
+ <createmovetofolder>N</createmovetofolder>
+ <delete>N</delete>
+ <filenamepattern>msg_#IdFile#.mail</filenamepattern>
+ <firstmails/>
+ <imapfirstmails>1</imapfirstmails>
+ <imapfolder>INBOX</imapfolder>
+ <includesubfolders>N</includesubfolders>
+ <movetoimapfolder>INBOX.sub</movetoimapfolder>
+ <nottermbodysearch>N</nottermbodysearch>
+ <nottermreceipientsearh>N</nottermreceipientsearh>
+ <nottermreceiveddatesearch>N</nottermreceiveddatesearch>
+ <nottermsendersearch>N</nottermsendersearch>
+ <nottermsubjectsearch>N</nottermsubjectsearch>
+ <outputdirectory>${PROJECT_HOME}/output/0010-inline</outputdirectory>
+ <protocol>IMAP</protocol>
+ <proxyusername/>
+ <receivedDate1/>
+ <receivedDate2/>
+ <recipientsearch/>
+ <retrievemails>0</retrievemails>
+ <actiontype>move</actiontype>
+ <valueimaplist>imaplistall</valueimaplist>
+ <conditionreceiveddate>ignore</conditionreceiveddate>
+ <aftergetimap>nothing</aftergetimap>
+ <saveattachment>N</saveattachment>
+ <savemessage>Y</savemessage>
+ <sendersearch/>
+ <subjectsearch/>
+ <usedifferentfolderforattachment>N</usedifferentfolderforattachment>
+ <useproxy>N</useproxy>
+ <usexoauth2>N</usexoauth2>
+ <parallel>N</parallel>
+ <xloc>464</xloc>
+ <yloc>64</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>check move counters (inline)</name>
+ <description/>
+ <type>EVAL</type>
+ <attributes/>
+ <script>var updated = previous_result.getNrLinesUpdated();
Review Comment:
The move step runs with Include subfolders = N, and the get step uses
after-get = nothing, so the combination most likely to break (move with Include
subfolders = Y and the target below the source, see the comments on
`MailConnection` line 1067 and `ActionGetPOP` line 914) isn't covered. Adding
that case would have caught both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]