hansva commented on code in PR #9358:
URL: https://github.com/apache/hop/pull/9358#discussion_r4537067707


##########
plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/MailConnection.java:
##########
@@ -962,8 +962,9 @@ public void moveMessage() throws HopException {
       // move all messages
       this.folder.copyMessages(new Message[] {this.message}, 
this.destinationIMAPFolder);
       updatedMovedMessagesCounter();
-      // Make sure to delete messages
-      deleteMessage();
+      // Flag the original as deleted. Do not call deleteMessage(): a moved 
message only counts
+      // as moved, not also as deleted.

Review Comment:
   `moveMessage()` no longer adds to `nrDeletedMessages`, which changes the 
public Lines deleted result for existing workflows that move messages. A 
workflow that checks `previous_result.getNrLinesDeleted() > 0` after a move 
used to get N and now gets 0. If this is intended, it deserves a note in the 
upgrade/release notes, not only the doc edit.



##########
plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/MailConnection.java:
##########
@@ -1063,7 +1064,7 @@ private void updatedMovedMessagesCounter() {
 
   /** Set count of moved messages. */
   private void setMovedMessagesCounter() {
-    this.nrMovedMessages = getMessagesCount();
+    this.nrMovedMessages += getMessagesCount();

Review Comment:
   Now that the moved/deleted counters are summed over all folders, this can 
over-count: with Include subfolders the folder list can contain the move target 
itself. E.g. INBOX, Include subfolders = Y, move to `INBOX.sub` (the setup of 
the new IT): INBOX moves 3 messages into INBOX.sub, then INBOX.sub is processed 
as a subfolder and its 4 messages are copied into themselves, so 
`nrMovedMessages` = 7 for 3 real moves, and the server may end up with 
duplicate copies. Could the move target (and its children) be excluded from the 
folders being processed?



##########
plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/ActionGetPOP.java:
##########
@@ -903,7 +910,8 @@ void fetchOneFolder(
                   }
                   if (isSaveMessage()) {
                     // get local message filename
-                    String localfilenameMessage = 
replaceTokens(realFilenamePattern, i);
+                    String localfilenameMessage =
+                        replaceTokens(realFilenamePattern, 
savedMessageIndex++);

Review Comment:
   Same root cause on the get side: with Include subfolders and 'Move message 
to folder' after retrieval into a subfolder of the source, each moved message 
is fetched again from that subfolder and saved with a new `#IdFile#`. INBOX 
with 3 messages, move to INBOX.sub: the INBOX pass saves msg_1..3 and moves 
them, the INBOX.sub pass saves the same messages again as msg_4..6, so 6 files 
and Lines written = 6 for 3 messages. Before, the duplicates at least overwrote 
msg_1..3.



##########
plugins/misc/mail/src/main/java/org/apache/hop/mail/metadata/MailServerConnection.java:
##########
@@ -636,7 +636,7 @@ private void updateDeletedMessagesCounter() {
   }
 
   private void setDeletedMessagesCounter() {
-    this.nrDeletedMessages = getMessagesCount();
+    this.nrDeletedMessages += getMessagesCount();

Review Comment:
   These counter and `moveMessage` fixes are made by hand in both 
`MailServerConnection` and `getpop/MailConnection`, each with its own tests, 
and the IT runs every scenario twice to cover both. Every future change to 
move/delete counting will have to be made twice. Could this go into one shared 
helper both classes use?



##########
plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/MailConnection.java:
##########
@@ -962,8 +962,9 @@ public void moveMessage() throws HopException {
       // move all messages
       this.folder.copyMessages(new Message[] {this.message}, 
this.destinationIMAPFolder);
       updatedMovedMessagesCounter();

Review Comment:
   The moved counter is incremented before `setFlag(DELETED)`. If flagging 
fails (read-only folder, dropped connection) after `copyMessages` succeeded, 
the message is counted as moved, an exception is thrown, and the original is 
still in the source folder. Incrementing after the flag succeeds would keep the 
count honest.



##########
integration-tests/mail/metadata/MailServerConnection/james-imap-folders.json:
##########
@@ -0,0 +1,17 @@
+{

Review Comment:
   The metadata key is `mail-server-connection` now (with 
`MailServerConnection` only in `legacyKeys` for old objects), so new objects 
should go in `metadata/mail-server-connection/`. A file in the legacy folder 
still loads, but it keeps the old layout going and moves the first time it's 
saved.



##########
plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/ActionGetPOP.java:
##########
@@ -222,6 +222,12 @@ public class ActionGetPOP extends ActionBase implements 
Cloneable, IAction {
   private MailServerConnection connection;
   private MailConnection mailConn;
 
+  /**
+   * Number of messages saved so far in this run, over all folders. Replaces 
#IdFile# so a message
+   * of a subfolder does not overwrite one of an earlier folder.
+   */
+  private int savedMessageIndex;

Review Comment:
   `savedMessageIndex` tracks the same thing as the connection's saved-messages 
counter (`getSavedMessagesCounter()`, incremented in 
`saveMessageContentToFile`). Having both means two sources of truth that can 
drift (this field is bumped before the save, the counter after it succeeds). 
`replaceTokens(realFilenamePattern, connection.getSavedMessagesCounter())` 
should give the same numbering without the extra field and the reset in 
`execute()`.



##########
integration-tests/mail/main-0010-get-mails-subfolders.hwf:
##########
@@ -0,0 +1,852 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<workflow>
+  <name>main-0010-get-mails-subfolders</name>
+  <name_sync_with_filename>Y</name_sync_with_filename>
+  <description>Get mails over a folder and its subfolder: unique #IdFile#, 
move and delete counters (#9208)</description>
+  <extended_description/>
+  <workflow_version/>
+  <created_user>-</created_user>
+  <created_date>2026/11/07 12:00:00.000</created_date>
+  <modified_user>-</modified_user>
+  <modified_date>2026/11/07 12:00:00.000</modified_date>
+  <parameters>
+    </parameters>
+  <actions>
+    <action>
+      <name>Start</name>
+      <description/>
+      <type>SPECIAL</type>
+      <attributes/>
+      <DayOfMonth>1</DayOfMonth>
+      <hour>12</hour>
+      <intervalMinutes>60</intervalMinutes>
+      <intervalSeconds>0</intervalSeconds>
+      <minutes>0</minutes>
+      <repeat>N</repeat>
+      <schedulerType>0</schedulerType>
+      <weekDay>1</weekDay>
+      <parallel>N</parallel>
+      <xloc>64</xloc>
+      <yloc>64</yloc>
+      <attributes_hac/>
+    </action>
+    <action>
+      <name>prepare mailbox (inline)</name>
+      <description/>
+      <type>SHELL</type>
+      <attributes/>
+      <add_date>N</add_date>
+      <add_time>N</add_time>
+      <arg_from_previous>N</arg_from_previous>
+      <arguments>
+</arguments>
+      <exec_per_row>N</exec_per_row>
+      <insertScript>Y</insertScript>
+      <loglevel>Basic</loglevel>
+      <script>#!/bin/bash
+# Fresh mailbox for [email protected]: INBOX with 4 messages and an empty 
INBOX.sub.
+WEBADMIN="http://${MAIL_SMTP_SERVER}:8000";
+MAILUSER="[email protected]"
+OUT="${PROJECT_HOME}/output/0010-inline"
+
+ready=0
+for attempt in $(seq 1 60); do
+  if curl -sf -o /dev/null --max-time 2 "$WEBADMIN/domains"; then
+    ready=1
+    break
+  fi
+  sleep 2
+done
+if [ "$ready" != 1 ]; then
+  echo "James WebAdmin did not accept connections on port 8000"
+  exit 1
+fi
+
+curl -s -XPUT "$WEBADMIN/domains/example.com"
+curl -s -XPUT "$WEBADMIN/users/$MAILUSER" -d '{"password":"password"}' -H 
"Content-Type: application/json"
+curl -sf -XDELETE "$WEBADMIN/users/$MAILUSER/mailboxes" || exit 1
+curl -sf -XPUT "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX" || exit 1
+curl -sf -XPUT "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX.sub" || exit 1
+
+rm -rf "$OUT"
+mkdir -p "$OUT"
+
+for n in 1 2 3 4; do
+  printf 'From: [email protected]\r\nTo: %s\r\nSubject: Message 
%s\r\n\r\nBody of message %s\r\n' "$MAILUSER" "$n" "$n" \
+    | curl -sS --url "smtp://${MAIL_SMTP_SERVER}:25" --mail-from 
[email protected] --mail-rcpt "$MAILUSER" -T - || exit 1
+done
+
+# Delivery is asynchronous: wait until all 4 messages are in INBOX
+for attempt in $(seq 1 60); do
+  count=$(curl -s "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX/messageCount")
+  if [ "$count" = 4 ]; then
+    echo "INBOX of $MAILUSER holds 4 messages"
+    exit 0
+  fi
+  sleep 1
+done
+echo "Expected 4 messages in INBOX of $MAILUSER, found $count"
+exit 1
+</script>
+      <set_append_logfile>N</set_append_logfile>
+      <set_logfile>N</set_logfile>
+      <parallel>N</parallel>
+      <xloc>240</xloc>
+      <yloc>64</yloc>
+      <attributes_hac/>
+    </action>
+    <action>
+      <name>move first mail to INBOX.sub (inline)</name>
+      <description/>
+      <type>GET_POP</type>
+      <attributes/>
+      <password>Encrypted 2be98afc86aa7f2e4bb18bd63c99dbdde</password>
+      <servername>${MAIL_SMTP_SERVER}</servername>
+      <sslport>993</sslport>
+      <username>[email protected]</username>
+      <usessl>Y</usessl>
+      <attachmentfolder/>
+      <attachmentwildcard/>
+      <bodysearch/>
+      <createlocalfolder>Y</createlocalfolder>
+      <createmovetofolder>N</createmovetofolder>
+      <delete>N</delete>
+      <filenamepattern>msg_#IdFile#.mail</filenamepattern>
+      <firstmails/>
+      <imapfirstmails>1</imapfirstmails>
+      <imapfolder>INBOX</imapfolder>
+      <includesubfolders>N</includesubfolders>
+      <movetoimapfolder>INBOX.sub</movetoimapfolder>
+      <nottermbodysearch>N</nottermbodysearch>
+      <nottermreceipientsearh>N</nottermreceipientsearh>
+      <nottermreceiveddatesearch>N</nottermreceiveddatesearch>
+      <nottermsendersearch>N</nottermsendersearch>
+      <nottermsubjectsearch>N</nottermsubjectsearch>
+      <outputdirectory>${PROJECT_HOME}/output/0010-inline</outputdirectory>
+      <protocol>IMAP</protocol>
+      <proxyusername/>
+      <receivedDate1/>
+      <receivedDate2/>
+      <recipientsearch/>
+      <retrievemails>0</retrievemails>
+      <actiontype>move</actiontype>
+      <valueimaplist>imaplistall</valueimaplist>
+      <conditionreceiveddate>ignore</conditionreceiveddate>
+      <aftergetimap>nothing</aftergetimap>
+      <saveattachment>N</saveattachment>
+      <savemessage>Y</savemessage>
+      <sendersearch/>
+      <subjectsearch/>
+      <usedifferentfolderforattachment>N</usedifferentfolderforattachment>
+      <useproxy>N</useproxy>
+      <usexoauth2>N</usexoauth2>
+      <parallel>N</parallel>
+      <xloc>464</xloc>
+      <yloc>64</yloc>
+      <attributes_hac/>
+    </action>
+    <action>
+      <name>check move counters (inline)</name>
+      <description/>
+      <type>EVAL</type>
+      <attributes/>
+      <script>var updated = previous_result.getNrLinesUpdated();

Review Comment:
   The move step runs with Include subfolders = N, and the get step uses 
after-get = nothing, so the combination most likely to break (move with Include 
subfolders = Y and the target below the source, see the comments on 
`MailConnection` line 1067 and `ActionGetPOP` line 914) isn't covered. Adding 
that case would have caught both.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to