This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 5de6a6a253 Get mails: #IdFile# restarts in each folder and overwrites
files, sin… (#9358)
5de6a6a253 is described below
commit 5de6a6a253daea7025c6a87d3cb7978d2c65a2c0
Author: Bart Maertens <[email protected]>
AuthorDate: Thu Nov 12 11:54:22 2026 +0100
Get mails: #IdFile# restarts in each folder and overwrites files, sin…
(#9358)
* Get mails: #IdFile# restarts in each folder and overwrites files, single
moves count as deletes, fixes #9208
* Get mails: unit tests for #IdFile# over folders and move/delete counters,
#9208
* Get mails: skip the move target in subfolders, count moves after the
flag, shared folder operations, #9208
---
.../ROOT/pages/workflow/actions/getpop.adoc | 10 +-
.../mail/main-0010-get-mails-subfolders.hwf | 1504 ++++++++++++++++++++
.../mail-server-connection/james-imap-folders.json | 17 +
.../hop/mail/common/MailFolderOperations.java | 109 ++
.../hop/mail/metadata/MailServerConnection.java | 67 +-
.../mail/workflow/actions/getpop/ActionGetPOP.java | 93 +-
.../workflow/actions/getpop/MailConnection.java | 68 +-
.../getpop/messages/messages_en_US.properties | 1 +
.../hop/mail/common/MailFolderOperationsTest.java | 115 ++
.../mail/metadata/MailServerConnectionTest.java | 67 +
.../workflow/actions/getpop/ActionGetPOPTest.java | 78 +-
.../actions/getpop/MailConnectionTest.java | 48 +
12 files changed, 2035 insertions(+), 142 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/getpop.adoc
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/getpop.adoc
index 49d7e7c4f8..5c55f0cd69 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/getpop.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/getpop.adoc
@@ -52,7 +52,7 @@ The `Get mails (POP3/IMAP)` action retrieves mails from a
POP3 or IMAP server an
2+|Target folder
|Target directory|The directory in which you want to put the mail files.
|Create folder|check to create the specified target directory if it doesn't
exist
-|Target filename pattern|The filename pattern (start of filename). The default
pattern is `name_{SYS\|hhmmss_MMddyyyy\|}_#IdFile#.mail`
+|Target filename pattern|The filename pattern (start of filename). The default
pattern is `name_{SYS\|hhmmss_MMddyyyy\|}_#IdFile#.mail`. `#IdFile#` is
replaced with the number of the saved message, starting at 1 and counting over
all folders, so with *Include subfolders* every message gets its own file.
|Save message content|check to save the message body text for retrieved email
messages
|Get mail attachment|check to save the attachments for retrieved email messages
|Different folder for attachment|check to save the attachments for retrieved
email messages in a different folder than the message body
@@ -81,7 +81,7 @@ a|Select the e-mails to retrieve:
|Delete emails after retrieval|check to delete the mails on the pop server
after retrieval.
2+|IMAP settings
|IMAP folder|the folder on the IMAP server to retrieve messages from
-|Include subfolders|check to select messages for subfolders from the selected
IMAP folder
+|Include subfolders|check to select messages for subfolders from the selected
IMAP folder. Folders are handled in alphabetical order. When messages are
moved, the folder they are moved into is skipped, so a moved message isn't
read, saved or moved a second time.
|Retrieve a|choose one of
* Get all messages
@@ -148,12 +148,12 @@ What this action passes on to the next action. See
xref:workflow/action-results.
|Result rows|Passed on unchanged.
|Lines written|The number of messages saved with *Save message content*.
|Files retrieved|The number of attachments saved.
-|Lines deleted|The number of messages deleted, by the *Delete messages*
action, *Delete emails after retrieval* or *Delete message* after retrieval. A
message moved on its own also counts here: with *Move message to folder* after
retrieval, or with the *Move messages to folder* action and *Retrieve the
first...emails*.
-|Lines updated|The number of messages moved to another IMAP folder.
+|Lines deleted|The number of messages deleted, by the *Delete messages*
action, *Delete emails after retrieval* or *Delete message* after retrieval.
+|Lines updated|The number of messages moved to another IMAP folder. A moved
message only counts here, not in *Lines deleted*.
|===
When the action is false, these metrics keep the values of the previous action.
-NOTE: The `#IdFile#` number in *Target filename pattern* starts again in each
folder: with *Include subfolders*, a message of a subfolder can overwrite the
file of a message of the first folder.
+NOTE: Up to Hop 2.20, a message moved on its own (with *Move message to
folder* after retrieval, or with the *Move messages to folder* action and
*Retrieve the first...emails*) also counted in *Lines deleted*. From 2.21 it
only counts in *Lines updated*, like a message moved together with the others.
A workflow that checks *Lines deleted* after a move needs to check *Lines
updated* instead.
NOTE: *Retrieve the first...emails* also applies to the *Delete messages* and
*Move messages to folder* actions: only that number of messages is deleted or
moved.
diff --git a/integration-tests/mail/main-0010-get-mails-subfolders.hwf
b/integration-tests/mail/main-0010-get-mails-subfolders.hwf
new file mode 100644
index 0000000000..6a68f3849b
--- /dev/null
+++ b/integration-tests/mail/main-0010-get-mails-subfolders.hwf
@@ -0,0 +1,1504 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<workflow>
+ <name>main-0010-get-mails-subfolders</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>Get mails over a folder and its subfolders: unique #IdFile#,
move and delete counters, move target below the source folder
(#9208)</description>
+ <extended_description/>
+ <workflow_version/>
+ <created_user>-</created_user>
+ <created_date>2026/11/07 12:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/11/07 12:00:00.000</modified_date>
+ <parameters>
+ </parameters>
+ <actions>
+ <action>
+ <name>Start</name>
+ <description/>
+ <type>SPECIAL</type>
+ <attributes/>
+ <DayOfMonth>1</DayOfMonth>
+ <hour>12</hour>
+ <intervalMinutes>60</intervalMinutes>
+ <intervalSeconds>0</intervalSeconds>
+ <minutes>0</minutes>
+ <repeat>N</repeat>
+ <schedulerType>0</schedulerType>
+ <weekDay>1</weekDay>
+ <parallel>N</parallel>
+ <xloc>64</xloc>
+ <yloc>64</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>prepare mailbox (inline)</name>
+ <description/>
+ <type>SHELL</type>
+ <attributes/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <arg_from_previous>N</arg_from_previous>
+ <arguments>
+</arguments>
+ <exec_per_row>N</exec_per_row>
+ <insertScript>Y</insertScript>
+ <loglevel>Basic</loglevel>
+ <script>#!/bin/bash
+# Fresh mailbox for [email protected]: INBOX with 4 messages and an empty
INBOX.sub.
+WEBADMIN="http://${MAIL_SMTP_SERVER}:8000"
+MAILUSER="[email protected]"
+OUT="${PROJECT_HOME}/output/0010-inline"
+
+ready=0
+for attempt in $(seq 1 60); do
+ if curl -sf -o /dev/null --max-time 2 "$WEBADMIN/domains"; then
+ ready=1
+ break
+ fi
+ sleep 2
+done
+if [ "$ready" != 1 ]; then
+ echo "James WebAdmin did not accept connections on port 8000"
+ exit 1
+fi
+
+curl -s -XPUT "$WEBADMIN/domains/example.com"
+curl -s -XPUT "$WEBADMIN/users/$MAILUSER" -d '{"password":"password"}' -H
"Content-Type: application/json"
+curl -sf -XDELETE "$WEBADMIN/users/$MAILUSER/mailboxes" || exit 1
+curl -sf -XPUT "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX" || exit 1
+curl -sf -XPUT "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX.sub" || exit 1
+
+rm -rf "$OUT"
+mkdir -p "$OUT"
+
+for n in 1 2 3 4; do
+ printf 'From: [email protected]\r\nTo: %s\r\nSubject: Message
%s\r\n\r\nBody of message %s\r\n' "$MAILUSER" "$n" "$n" \
+ | curl -sS --url "smtp://${MAIL_SMTP_SERVER}:25" --mail-from
[email protected] --mail-rcpt "$MAILUSER" -T - || exit 1
+done
+
+# Delivery is asynchronous: wait until all 4 messages are in INBOX
+for attempt in $(seq 1 60); do
+ count=$(curl -s "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX/messageCount")
+ if [ "$count" = 4 ]; then
+ echo "INBOX of $MAILUSER holds 4 messages"
+ exit 0
+ fi
+ sleep 1
+done
+echo "Expected 4 messages in INBOX of $MAILUSER, found $count"
+exit 1
+</script>
+ <set_append_logfile>N</set_append_logfile>
+ <set_logfile>N</set_logfile>
+ <parallel>N</parallel>
+ <xloc>240</xloc>
+ <yloc>64</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>move first mail to INBOX.sub (inline)</name>
+ <description/>
+ <type>GET_POP</type>
+ <attributes/>
+ <password>Encrypted 2be98afc86aa7f2e4bb18bd63c99dbdde</password>
+ <servername>${MAIL_SMTP_SERVER}</servername>
+ <sslport>993</sslport>
+ <username>[email protected]</username>
+ <usessl>Y</usessl>
+ <attachmentfolder/>
+ <attachmentwildcard/>
+ <bodysearch/>
+ <createlocalfolder>Y</createlocalfolder>
+ <createmovetofolder>N</createmovetofolder>
+ <delete>N</delete>
+ <filenamepattern>msg_#IdFile#.mail</filenamepattern>
+ <firstmails/>
+ <imapfirstmails>1</imapfirstmails>
+ <imapfolder>INBOX</imapfolder>
+ <includesubfolders>N</includesubfolders>
+ <movetoimapfolder>INBOX.sub</movetoimapfolder>
+ <nottermbodysearch>N</nottermbodysearch>
+ <nottermreceipientsearh>N</nottermreceipientsearh>
+ <nottermreceiveddatesearch>N</nottermreceiveddatesearch>
+ <nottermsendersearch>N</nottermsendersearch>
+ <nottermsubjectsearch>N</nottermsubjectsearch>
+ <outputdirectory>${PROJECT_HOME}/output/0010-inline</outputdirectory>
+ <protocol>IMAP</protocol>
+ <proxyusername/>
+ <receivedDate1/>
+ <receivedDate2/>
+ <recipientsearch/>
+ <retrievemails>0</retrievemails>
+ <actiontype>move</actiontype>
+ <valueimaplist>imaplistall</valueimaplist>
+ <conditionreceiveddate>ignore</conditionreceiveddate>
+ <aftergetimap>nothing</aftergetimap>
+ <saveattachment>N</saveattachment>
+ <savemessage>Y</savemessage>
+ <sendersearch/>
+ <subjectsearch/>
+ <usedifferentfolderforattachment>N</usedifferentfolderforattachment>
+ <useproxy>N</useproxy>
+ <usexoauth2>N</usexoauth2>
+ <parallel>N</parallel>
+ <xloc>464</xloc>
+ <yloc>64</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>check move counters (inline)</name>
+ <description/>
+ <type>EVAL</type>
+ <attributes/>
+ <script>var updated = previous_result.getNrLinesUpdated();
+var deleted = previous_result.getNrLinesDeleted();
+log.logBasic("Move first message (inline): lines updated = " + updated + ",
lines deleted = " + deleted);
+updated == 1 && deleted == 0;</script>
+ <parallel>N</parallel>
+ <xloc>688</xloc>
+ <yloc>64</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>get mails incl. subfolders (inline)</name>
+ <description/>
+ <type>GET_POP</type>
+ <attributes/>
+ <password>Encrypted 2be98afc86aa7f2e4bb18bd63c99dbdde</password>
+ <servername>${MAIL_SMTP_SERVER}</servername>
+ <sslport>993</sslport>
+ <username>[email protected]</username>
+ <usessl>Y</usessl>
+ <attachmentfolder/>
+ <attachmentwildcard/>
+ <bodysearch/>
+ <createlocalfolder>Y</createlocalfolder>
+ <createmovetofolder>N</createmovetofolder>
+ <delete>N</delete>
+ <filenamepattern>msg_#IdFile#.mail</filenamepattern>
+ <firstmails/>
+ <imapfirstmails>0</imapfirstmails>
+ <imapfolder>INBOX</imapfolder>
+ <includesubfolders>Y</includesubfolders>
+ <movetoimapfolder></movetoimapfolder>
+ <nottermbodysearch>N</nottermbodysearch>
+ <nottermreceipientsearh>N</nottermreceipientsearh>
+ <nottermreceiveddatesearch>N</nottermreceiveddatesearch>
+ <nottermsendersearch>N</nottermsendersearch>
+ <nottermsubjectsearch>N</nottermsubjectsearch>
+ <outputdirectory>${PROJECT_HOME}/output/0010-inline</outputdirectory>
+ <protocol>IMAP</protocol>
+ <proxyusername/>
+ <receivedDate1/>
+ <receivedDate2/>
+ <recipientsearch/>
+ <retrievemails>0</retrievemails>
+ <actiontype>get</actiontype>
+ <valueimaplist>imaplistall</valueimaplist>
+ <conditionreceiveddate>ignore</conditionreceiveddate>
+ <aftergetimap>nothing</aftergetimap>
+ <saveattachment>N</saveattachment>
+ <savemessage>Y</savemessage>
+ <sendersearch/>
+ <subjectsearch/>
+ <usedifferentfolderforattachment>N</usedifferentfolderforattachment>
+ <useproxy>N</useproxy>
+ <usexoauth2>N</usexoauth2>
+ <parallel>N</parallel>
+ <xloc>912</xloc>
+ <yloc>64</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>check saved files (inline)</name>
+ <description/>
+ <type>SHELL</type>
+ <attributes/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <arg_from_previous>N</arg_from_previous>
+ <arguments>
+</arguments>
+ <exec_per_row>N</exec_per_row>
+ <insertScript>Y</insertScript>
+ <loglevel>Basic</loglevel>
+ <script>#!/bin/bash
+# #IdFile# must be unique over all folders: 4 files, each message saved
exactly once.
+OUT="${PROJECT_HOME}/output/0010-inline"
+errors=0
+files=$(ls "$OUT")
+nfiles=$(ls "$OUT" | wc -l)
+echo "Saved message files: " $files
+if [ "$nfiles" != 4 ]; then
+ echo "Expected 4 message files, found $nfiles"
+ errors=$((errors + 1))
+fi
+for n in 1 2 3 4; do
+ found=$(grep -l "^Subject: Message $n" "$OUT"/*.mail 2>/dev/null | wc -l)
+ if [ "$found" != 1 ]; then
+ echo "Message $n was found in $found files, expected 1"
+ errors=$((errors + 1))
+ fi
+done
+exit $errors
+</script>
+ <set_append_logfile>N</set_append_logfile>
+ <set_logfile>N</set_logfile>
+ <parallel>N</parallel>
+ <xloc>1136</xloc>
+ <yloc>64</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>delete mails incl. subfolders (inline)</name>
+ <description/>
+ <type>GET_POP</type>
+ <attributes/>
+ <password>Encrypted 2be98afc86aa7f2e4bb18bd63c99dbdde</password>
+ <servername>${MAIL_SMTP_SERVER}</servername>
+ <sslport>993</sslport>
+ <username>[email protected]</username>
+ <usessl>Y</usessl>
+ <attachmentfolder/>
+ <attachmentwildcard/>
+ <bodysearch/>
+ <createlocalfolder>Y</createlocalfolder>
+ <createmovetofolder>N</createmovetofolder>
+ <delete>N</delete>
+ <filenamepattern>msg_#IdFile#.mail</filenamepattern>
+ <firstmails/>
+ <imapfirstmails>0</imapfirstmails>
+ <imapfolder>INBOX</imapfolder>
+ <includesubfolders>Y</includesubfolders>
+ <movetoimapfolder></movetoimapfolder>
+ <nottermbodysearch>N</nottermbodysearch>
+ <nottermreceipientsearh>N</nottermreceipientsearh>
+ <nottermreceiveddatesearch>N</nottermreceiveddatesearch>
+ <nottermsendersearch>N</nottermsendersearch>
+ <nottermsubjectsearch>N</nottermsubjectsearch>
+ <outputdirectory>${PROJECT_HOME}/output/0010-inline</outputdirectory>
+ <protocol>IMAP</protocol>
+ <proxyusername/>
+ <receivedDate1/>
+ <receivedDate2/>
+ <recipientsearch/>
+ <retrievemails>0</retrievemails>
+ <actiontype>delete</actiontype>
+ <valueimaplist>imaplistall</valueimaplist>
+ <conditionreceiveddate>ignore</conditionreceiveddate>
+ <aftergetimap>nothing</aftergetimap>
+ <saveattachment>N</saveattachment>
+ <savemessage>Y</savemessage>
+ <sendersearch/>
+ <subjectsearch/>
+ <usedifferentfolderforattachment>N</usedifferentfolderforattachment>
+ <useproxy>N</useproxy>
+ <usexoauth2>N</usexoauth2>
+ <parallel>N</parallel>
+ <xloc>1360</xloc>
+ <yloc>64</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>check delete counters (inline)</name>
+ <description/>
+ <type>EVAL</type>
+ <attributes/>
+ <script>var updated = previous_result.getNrLinesUpdated();
+var deleted = previous_result.getNrLinesDeleted();
+log.logBasic("Delete all messages (inline): lines updated = " + updated + ",
lines deleted = " + deleted);
+updated == 0 && deleted == 4;</script>
+ <parallel>N</parallel>
+ <xloc>1584</xloc>
+ <yloc>64</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>prepare mailbox (connection)</name>
+ <description/>
+ <type>SHELL</type>
+ <attributes/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <arg_from_previous>N</arg_from_previous>
+ <arguments>
+</arguments>
+ <exec_per_row>N</exec_per_row>
+ <insertScript>Y</insertScript>
+ <loglevel>Basic</loglevel>
+ <script>#!/bin/bash
+# Fresh mailbox for [email protected]: INBOX with 4 messages and an empty
INBOX.sub.
+WEBADMIN="http://${MAIL_SMTP_SERVER}:8000"
+MAILUSER="[email protected]"
+OUT="${PROJECT_HOME}/output/0010-connection"
+
+ready=0
+for attempt in $(seq 1 60); do
+ if curl -sf -o /dev/null --max-time 2 "$WEBADMIN/domains"; then
+ ready=1
+ break
+ fi
+ sleep 2
+done
+if [ "$ready" != 1 ]; then
+ echo "James WebAdmin did not accept connections on port 8000"
+ exit 1
+fi
+
+curl -s -XPUT "$WEBADMIN/domains/example.com"
+curl -s -XPUT "$WEBADMIN/users/$MAILUSER" -d '{"password":"password"}' -H
"Content-Type: application/json"
+curl -sf -XDELETE "$WEBADMIN/users/$MAILUSER/mailboxes" || exit 1
+curl -sf -XPUT "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX" || exit 1
+curl -sf -XPUT "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX.sub" || exit 1
+
+rm -rf "$OUT"
+mkdir -p "$OUT"
+
+for n in 1 2 3 4; do
+ printf 'From: [email protected]\r\nTo: %s\r\nSubject: Message
%s\r\n\r\nBody of message %s\r\n' "$MAILUSER" "$n" "$n" \
+ | curl -sS --url "smtp://${MAIL_SMTP_SERVER}:25" --mail-from
[email protected] --mail-rcpt "$MAILUSER" -T - || exit 1
+done
+
+# Delivery is asynchronous: wait until all 4 messages are in INBOX
+for attempt in $(seq 1 60); do
+ count=$(curl -s "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX/messageCount")
+ if [ "$count" = 4 ]; then
+ echo "INBOX of $MAILUSER holds 4 messages"
+ exit 0
+ fi
+ sleep 1
+done
+echo "Expected 4 messages in INBOX of $MAILUSER, found $count"
+exit 1
+</script>
+ <set_append_logfile>N</set_append_logfile>
+ <set_logfile>N</set_logfile>
+ <parallel>N</parallel>
+ <xloc>240</xloc>
+ <yloc>224</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>move first mail to INBOX.sub (connection)</name>
+ <description/>
+ <type>GET_POP</type>
+ <attributes/>
+ <connectionName>james-imap-folders</connectionName>
+ <password/>
+ <servername/>
+ <sslport/>
+ <username/>
+ <usessl>N</usessl>
+ <attachmentfolder/>
+ <attachmentwildcard/>
+ <bodysearch/>
+ <createlocalfolder>Y</createlocalfolder>
+ <createmovetofolder>N</createmovetofolder>
+ <delete>N</delete>
+ <filenamepattern>msg_#IdFile#.mail</filenamepattern>
+ <firstmails/>
+ <imapfirstmails>1</imapfirstmails>
+ <imapfolder>INBOX</imapfolder>
+ <includesubfolders>N</includesubfolders>
+ <movetoimapfolder>INBOX.sub</movetoimapfolder>
+ <nottermbodysearch>N</nottermbodysearch>
+ <nottermreceipientsearh>N</nottermreceipientsearh>
+ <nottermreceiveddatesearch>N</nottermreceiveddatesearch>
+ <nottermsendersearch>N</nottermsendersearch>
+ <nottermsubjectsearch>N</nottermsubjectsearch>
+ <outputdirectory>${PROJECT_HOME}/output/0010-connection</outputdirectory>
+ <protocol>IMAP</protocol>
+ <proxyusername/>
+ <receivedDate1/>
+ <receivedDate2/>
+ <recipientsearch/>
+ <retrievemails>0</retrievemails>
+ <actiontype>move</actiontype>
+ <valueimaplist>imaplistall</valueimaplist>
+ <conditionreceiveddate>ignore</conditionreceiveddate>
+ <aftergetimap>nothing</aftergetimap>
+ <saveattachment>N</saveattachment>
+ <savemessage>Y</savemessage>
+ <sendersearch/>
+ <subjectsearch/>
+ <usedifferentfolderforattachment>N</usedifferentfolderforattachment>
+ <useproxy>N</useproxy>
+ <usexoauth2>N</usexoauth2>
+ <parallel>N</parallel>
+ <xloc>464</xloc>
+ <yloc>224</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>check move counters (connection)</name>
+ <description/>
+ <type>EVAL</type>
+ <attributes/>
+ <script>var updated = previous_result.getNrLinesUpdated();
+var deleted = previous_result.getNrLinesDeleted();
+log.logBasic("Move first message (connection): lines updated = " + updated +
", lines deleted = " + deleted);
+updated == 1 && deleted == 0;</script>
+ <parallel>N</parallel>
+ <xloc>688</xloc>
+ <yloc>224</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>get mails incl. subfolders (connection)</name>
+ <description/>
+ <type>GET_POP</type>
+ <attributes/>
+ <connectionName>james-imap-folders</connectionName>
+ <password/>
+ <servername/>
+ <sslport/>
+ <username/>
+ <usessl>N</usessl>
+ <attachmentfolder/>
+ <attachmentwildcard/>
+ <bodysearch/>
+ <createlocalfolder>Y</createlocalfolder>
+ <createmovetofolder>N</createmovetofolder>
+ <delete>N</delete>
+ <filenamepattern>msg_#IdFile#.mail</filenamepattern>
+ <firstmails/>
+ <imapfirstmails>0</imapfirstmails>
+ <imapfolder>INBOX</imapfolder>
+ <includesubfolders>Y</includesubfolders>
+ <movetoimapfolder></movetoimapfolder>
+ <nottermbodysearch>N</nottermbodysearch>
+ <nottermreceipientsearh>N</nottermreceipientsearh>
+ <nottermreceiveddatesearch>N</nottermreceiveddatesearch>
+ <nottermsendersearch>N</nottermsendersearch>
+ <nottermsubjectsearch>N</nottermsubjectsearch>
+ <outputdirectory>${PROJECT_HOME}/output/0010-connection</outputdirectory>
+ <protocol>IMAP</protocol>
+ <proxyusername/>
+ <receivedDate1/>
+ <receivedDate2/>
+ <recipientsearch/>
+ <retrievemails>0</retrievemails>
+ <actiontype>get</actiontype>
+ <valueimaplist>imaplistall</valueimaplist>
+ <conditionreceiveddate>ignore</conditionreceiveddate>
+ <aftergetimap>nothing</aftergetimap>
+ <saveattachment>N</saveattachment>
+ <savemessage>Y</savemessage>
+ <sendersearch/>
+ <subjectsearch/>
+ <usedifferentfolderforattachment>N</usedifferentfolderforattachment>
+ <useproxy>N</useproxy>
+ <usexoauth2>N</usexoauth2>
+ <parallel>N</parallel>
+ <xloc>912</xloc>
+ <yloc>224</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>check saved files (connection)</name>
+ <description/>
+ <type>SHELL</type>
+ <attributes/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <arg_from_previous>N</arg_from_previous>
+ <arguments>
+</arguments>
+ <exec_per_row>N</exec_per_row>
+ <insertScript>Y</insertScript>
+ <loglevel>Basic</loglevel>
+ <script>#!/bin/bash
+# #IdFile# must be unique over all folders: 4 files, each message saved
exactly once.
+OUT="${PROJECT_HOME}/output/0010-connection"
+errors=0
+files=$(ls "$OUT")
+nfiles=$(ls "$OUT" | wc -l)
+echo "Saved message files: " $files
+if [ "$nfiles" != 4 ]; then
+ echo "Expected 4 message files, found $nfiles"
+ errors=$((errors + 1))
+fi
+for n in 1 2 3 4; do
+ found=$(grep -l "^Subject: Message $n" "$OUT"/*.mail 2>/dev/null | wc -l)
+ if [ "$found" != 1 ]; then
+ echo "Message $n was found in $found files, expected 1"
+ errors=$((errors + 1))
+ fi
+done
+exit $errors
+</script>
+ <set_append_logfile>N</set_append_logfile>
+ <set_logfile>N</set_logfile>
+ <parallel>N</parallel>
+ <xloc>1136</xloc>
+ <yloc>224</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>delete mails incl. subfolders (connection)</name>
+ <description/>
+ <type>GET_POP</type>
+ <attributes/>
+ <connectionName>james-imap-folders</connectionName>
+ <password/>
+ <servername/>
+ <sslport/>
+ <username/>
+ <usessl>N</usessl>
+ <attachmentfolder/>
+ <attachmentwildcard/>
+ <bodysearch/>
+ <createlocalfolder>Y</createlocalfolder>
+ <createmovetofolder>N</createmovetofolder>
+ <delete>N</delete>
+ <filenamepattern>msg_#IdFile#.mail</filenamepattern>
+ <firstmails/>
+ <imapfirstmails>0</imapfirstmails>
+ <imapfolder>INBOX</imapfolder>
+ <includesubfolders>Y</includesubfolders>
+ <movetoimapfolder></movetoimapfolder>
+ <nottermbodysearch>N</nottermbodysearch>
+ <nottermreceipientsearh>N</nottermreceipientsearh>
+ <nottermreceiveddatesearch>N</nottermreceiveddatesearch>
+ <nottermsendersearch>N</nottermsendersearch>
+ <nottermsubjectsearch>N</nottermsubjectsearch>
+ <outputdirectory>${PROJECT_HOME}/output/0010-connection</outputdirectory>
+ <protocol>IMAP</protocol>
+ <proxyusername/>
+ <receivedDate1/>
+ <receivedDate2/>
+ <recipientsearch/>
+ <retrievemails>0</retrievemails>
+ <actiontype>delete</actiontype>
+ <valueimaplist>imaplistall</valueimaplist>
+ <conditionreceiveddate>ignore</conditionreceiveddate>
+ <aftergetimap>nothing</aftergetimap>
+ <saveattachment>N</saveattachment>
+ <savemessage>Y</savemessage>
+ <sendersearch/>
+ <subjectsearch/>
+ <usedifferentfolderforattachment>N</usedifferentfolderforattachment>
+ <useproxy>N</useproxy>
+ <usexoauth2>N</usexoauth2>
+ <parallel>N</parallel>
+ <xloc>1360</xloc>
+ <yloc>224</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>check delete counters (connection)</name>
+ <description/>
+ <type>EVAL</type>
+ <attributes/>
+ <script>var updated = previous_result.getNrLinesUpdated();
+var deleted = previous_result.getNrLinesDeleted();
+log.logBasic("Delete all messages (connection): lines updated = " + updated +
", lines deleted = " + deleted);
+updated == 0 && deleted == 4;</script>
+ <parallel>N</parallel>
+ <xloc>1584</xloc>
+ <yloc>224</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>prepare mailbox for move (move target below INBOX, inline)</name>
+ <description/>
+ <type>SHELL</type>
+ <attributes/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <arg_from_previous>N</arg_from_previous>
+ <arguments>
+</arguments>
+ <exec_per_row>N</exec_per_row>
+ <insertScript>Y</insertScript>
+ <loglevel>Basic</loglevel>
+ <script>#!/bin/bash
+# Fresh mailbox for [email protected]: INBOX with 3 messages, INBOX.a with 1
and an empty
+# INBOX.z. INBOX.z is the move target: it sorts after INBOX.a, so it is
reached as a subfolder
+# after messages were moved into it.
+WEBADMIN="http://${MAIL_SMTP_SERVER}:8000"
+MAILUSER="[email protected]"
+OUT="${PROJECT_HOME}/output/0010-move-inline"
+
+curl -s -XPUT "$WEBADMIN/domains/example.com"
+curl -s -XPUT "$WEBADMIN/users/$MAILUSER" -d '{"password":"password"}' -H
"Content-Type: application/json"
+curl -sf -XDELETE "$WEBADMIN/users/$MAILUSER/mailboxes" || exit 1
+for mailbox in INBOX INBOX.a INBOX.z; do
+ curl -sf -XPUT "$WEBADMIN/users/$MAILUSER/mailboxes/$mailbox" || exit 1
+done
+
+rm -rf "$OUT"
+mkdir -p "$OUT"
+
+for n in 1 2 3; do
+ printf 'From: [email protected]\r\nTo: %s\r\nSubject: Message
%s\r\n\r\nBody of message %s\r\n' "$MAILUSER" "$n" "$n" \
+ | curl -sS --url "smtp://${MAIL_SMTP_SERVER}:25" --mail-from
[email protected] --mail-rcpt "$MAILUSER" -T - || exit 1
+done
+# Put message 4 straight into INBOX.a with an IMAP APPEND (curl needs a file
of known size)
+printf 'From: [email protected]\r\nTo: %s\r\nSubject: Message 4\r\n\r\nBody
of message 4\r\n' "$MAILUSER" > "$OUT/../0010-message-4.eml"
+curl -sSk --url "imaps://${MAIL_SMTP_SERVER}:993/INBOX.a" --user
"$MAILUSER:password" -T "$OUT/../0010-message-4.eml" || exit 1
+rm -f "$OUT/../0010-message-4.eml"
+
+# Delivery is asynchronous: wait until all 3 messages are in INBOX
+for attempt in $(seq 1 60); do
+ count=$(curl -s "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX/messageCount")
+ sub=$(curl -s "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX.a/messageCount")
+ if [ "$count" = 3 ] && [ "$sub" = 1 ]; then
+ echo "INBOX of $MAILUSER holds 3 messages, INBOX.a holds 1"
+ exit 0
+ fi
+ sleep 1
+done
+echo "Expected 3 messages in INBOX and 1 in INBOX.a of $MAILUSER, found $count
and $sub"
+exit 1
+</script>
+ <set_append_logfile>N</set_append_logfile>
+ <set_logfile>N</set_logfile>
+ <parallel>N</parallel>
+ <xloc>240</xloc>
+ <yloc>544</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>get and move to INBOX.z incl. subfolders (inline)</name>
+ <description/>
+ <type>GET_POP</type>
+ <attributes/>
+ <password>Encrypted 2be98afc86aa7f2e4bb18bd63c99dbdde</password>
+ <servername>${MAIL_SMTP_SERVER}</servername>
+ <sslport>993</sslport>
+ <username>[email protected]</username>
+ <usessl>Y</usessl>
+ <attachmentfolder/>
+ <attachmentwildcard/>
+ <bodysearch/>
+ <createlocalfolder>Y</createlocalfolder>
+ <createmovetofolder>N</createmovetofolder>
+ <delete>N</delete>
+ <filenamepattern>msg_#IdFile#.mail</filenamepattern>
+ <firstmails/>
+ <imapfirstmails>0</imapfirstmails>
+ <imapfolder>INBOX</imapfolder>
+ <includesubfolders>Y</includesubfolders>
+ <movetoimapfolder>INBOX.z</movetoimapfolder>
+ <nottermbodysearch>N</nottermbodysearch>
+ <nottermreceipientsearh>N</nottermreceipientsearh>
+ <nottermreceiveddatesearch>N</nottermreceiveddatesearch>
+ <nottermsendersearch>N</nottermsendersearch>
+ <nottermsubjectsearch>N</nottermsubjectsearch>
+
<outputdirectory>${PROJECT_HOME}/output/0010-move-inline</outputdirectory>
+ <protocol>IMAP</protocol>
+ <proxyusername/>
+ <receivedDate1/>
+ <receivedDate2/>
+ <recipientsearch/>
+ <retrievemails>0</retrievemails>
+ <actiontype>get</actiontype>
+ <valueimaplist>imaplistall</valueimaplist>
+ <conditionreceiveddate>ignore</conditionreceiveddate>
+ <aftergetimap>move</aftergetimap>
+ <saveattachment>N</saveattachment>
+ <savemessage>Y</savemessage>
+ <sendersearch/>
+ <subjectsearch/>
+ <usedifferentfolderforattachment>N</usedifferentfolderforattachment>
+ <useproxy>N</useproxy>
+ <usexoauth2>N</usexoauth2>
+ <parallel>N</parallel>
+ <xloc>464</xloc>
+ <yloc>544</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>check get and move counters (inline)</name>
+ <description/>
+ <type>EVAL</type>
+ <attributes/>
+ <script>var written = previous_result.getNrLinesWritten();
+var updated = previous_result.getNrLinesUpdated();
+var deleted = previous_result.getNrLinesDeleted();
+log.logBasic("Get and move incl. subfolders (inline): lines written = " +
written + ", lines updated = " + updated + ", lines deleted = " + deleted);
+written == 4 && updated == 4 && deleted == 0;</script>
+ <parallel>N</parallel>
+ <xloc>688</xloc>
+ <yloc>544</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>check saved files and folders after move (inline)</name>
+ <description/>
+ <type>SHELL</type>
+ <attributes/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <arg_from_previous>N</arg_from_previous>
+ <arguments>
+</arguments>
+ <exec_per_row>N</exec_per_row>
+ <insertScript>Y</insertScript>
+ <loglevel>Basic</loglevel>
+ <script>#!/bin/bash
+# Every message saved once, and all 4 now in INBOX.z: none read or copied
again from INBOX.z.
+WEBADMIN="http://${MAIL_SMTP_SERVER}:8000"
+MAILUSER="[email protected]"
+OUT="${PROJECT_HOME}/output/0010-move-inline"
+errors=0
+files=$(ls "$OUT")
+nfiles=$(ls "$OUT" | wc -l)
+echo "Saved message files: " $files
+if [ "$nfiles" != 4 ]; then
+ echo "Expected 4 message files, found $nfiles"
+ errors=$((errors + 1))
+fi
+for n in 1 2 3 4; do
+ found=$(grep -l "^Subject: Message $n" "$OUT"/*.mail 2>/dev/null | wc -l)
+ if [ "$found" != 1 ]; then
+ echo "Message $n was found in $found files, expected 1"
+ errors=$((errors + 1))
+ fi
+done
+for expected in INBOX:0 INBOX.a:0 INBOX.z:4; do
+ mailbox=${expected%%:*}
+ count=$(curl -s "$WEBADMIN/users/$MAILUSER/mailboxes/$mailbox/messageCount")
+ echo "$mailbox holds $count messages"
+ if [ "$count" != "${expected##*:}" ]; then
+ echo "Expected ${expected##*:} messages in $mailbox"
+ errors=$((errors + 1))
+ fi
+done
+exit $errors
+</script>
+ <set_append_logfile>N</set_append_logfile>
+ <set_logfile>N</set_logfile>
+ <parallel>N</parallel>
+ <xloc>912</xloc>
+ <yloc>544</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>move all to INBOX.z incl. subfolders (inline)</name>
+ <description/>
+ <type>GET_POP</type>
+ <attributes/>
+ <password>Encrypted 2be98afc86aa7f2e4bb18bd63c99dbdde</password>
+ <servername>${MAIL_SMTP_SERVER}</servername>
+ <sslport>993</sslport>
+ <username>[email protected]</username>
+ <usessl>Y</usessl>
+ <attachmentfolder/>
+ <attachmentwildcard/>
+ <bodysearch/>
+ <createlocalfolder>Y</createlocalfolder>
+ <createmovetofolder>N</createmovetofolder>
+ <delete>N</delete>
+ <filenamepattern>msg_#IdFile#.mail</filenamepattern>
+ <firstmails/>
+ <imapfirstmails>0</imapfirstmails>
+ <imapfolder>INBOX</imapfolder>
+ <includesubfolders>Y</includesubfolders>
+ <movetoimapfolder>INBOX.z</movetoimapfolder>
+ <nottermbodysearch>N</nottermbodysearch>
+ <nottermreceipientsearh>N</nottermreceipientsearh>
+ <nottermreceiveddatesearch>N</nottermreceiveddatesearch>
+ <nottermsendersearch>N</nottermsendersearch>
+ <nottermsubjectsearch>N</nottermsubjectsearch>
+
<outputdirectory>${PROJECT_HOME}/output/0010-move-inline</outputdirectory>
+ <protocol>IMAP</protocol>
+ <proxyusername/>
+ <receivedDate1/>
+ <receivedDate2/>
+ <recipientsearch/>
+ <retrievemails>0</retrievemails>
+ <actiontype>move</actiontype>
+ <valueimaplist>imaplistall</valueimaplist>
+ <conditionreceiveddate>ignore</conditionreceiveddate>
+ <aftergetimap>nothing</aftergetimap>
+ <saveattachment>N</saveattachment>
+ <savemessage>Y</savemessage>
+ <sendersearch/>
+ <subjectsearch/>
+ <usedifferentfolderforattachment>N</usedifferentfolderforattachment>
+ <useproxy>N</useproxy>
+ <usexoauth2>N</usexoauth2>
+ <parallel>N</parallel>
+ <xloc>1136</xloc>
+ <yloc>544</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>check move all counters (inline)</name>
+ <description/>
+ <type>EVAL</type>
+ <attributes/>
+ <script>var written = previous_result.getNrLinesWritten();
+var updated = previous_result.getNrLinesUpdated();
+var deleted = previous_result.getNrLinesDeleted();
+log.logBasic("Move all incl. subfolders, all messages already in INBOX.z
(inline): lines written = " + written + ", lines updated = " + updated + ",
lines deleted = " + deleted);
+updated == 0 && deleted == 0;</script>
+ <parallel>N</parallel>
+ <xloc>1360</xloc>
+ <yloc>544</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>prepare mailbox for move (move target below INBOX,
connection)</name>
+ <description/>
+ <type>SHELL</type>
+ <attributes/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <arg_from_previous>N</arg_from_previous>
+ <arguments>
+</arguments>
+ <exec_per_row>N</exec_per_row>
+ <insertScript>Y</insertScript>
+ <loglevel>Basic</loglevel>
+ <script>#!/bin/bash
+# Fresh mailbox for [email protected]: INBOX with 3 messages, INBOX.a with 1
and an empty
+# INBOX.z. INBOX.z is the move target: it sorts after INBOX.a, so it is
reached as a subfolder
+# after messages were moved into it.
+WEBADMIN="http://${MAIL_SMTP_SERVER}:8000"
+MAILUSER="[email protected]"
+OUT="${PROJECT_HOME}/output/0010-move-connection"
+
+curl -s -XPUT "$WEBADMIN/domains/example.com"
+curl -s -XPUT "$WEBADMIN/users/$MAILUSER" -d '{"password":"password"}' -H
"Content-Type: application/json"
+curl -sf -XDELETE "$WEBADMIN/users/$MAILUSER/mailboxes" || exit 1
+for mailbox in INBOX INBOX.a INBOX.z; do
+ curl -sf -XPUT "$WEBADMIN/users/$MAILUSER/mailboxes/$mailbox" || exit 1
+done
+
+rm -rf "$OUT"
+mkdir -p "$OUT"
+
+for n in 1 2 3; do
+ printf 'From: [email protected]\r\nTo: %s\r\nSubject: Message
%s\r\n\r\nBody of message %s\r\n' "$MAILUSER" "$n" "$n" \
+ | curl -sS --url "smtp://${MAIL_SMTP_SERVER}:25" --mail-from
[email protected] --mail-rcpt "$MAILUSER" -T - || exit 1
+done
+# Put message 4 straight into INBOX.a with an IMAP APPEND (curl needs a file
of known size)
+printf 'From: [email protected]\r\nTo: %s\r\nSubject: Message 4\r\n\r\nBody
of message 4\r\n' "$MAILUSER" > "$OUT/../0010-message-4.eml"
+curl -sSk --url "imaps://${MAIL_SMTP_SERVER}:993/INBOX.a" --user
"$MAILUSER:password" -T "$OUT/../0010-message-4.eml" || exit 1
+rm -f "$OUT/../0010-message-4.eml"
+
+# Delivery is asynchronous: wait until all 3 messages are in INBOX
+for attempt in $(seq 1 60); do
+ count=$(curl -s "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX/messageCount")
+ sub=$(curl -s "$WEBADMIN/users/$MAILUSER/mailboxes/INBOX.a/messageCount")
+ if [ "$count" = 3 ] && [ "$sub" = 1 ]; then
+ echo "INBOX of $MAILUSER holds 3 messages, INBOX.a holds 1"
+ exit 0
+ fi
+ sleep 1
+done
+echo "Expected 3 messages in INBOX and 1 in INBOX.a of $MAILUSER, found $count
and $sub"
+exit 1
+</script>
+ <set_append_logfile>N</set_append_logfile>
+ <set_logfile>N</set_logfile>
+ <parallel>N</parallel>
+ <xloc>240</xloc>
+ <yloc>704</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>get and move to INBOX.z incl. subfolders (connection)</name>
+ <description/>
+ <type>GET_POP</type>
+ <attributes/>
+ <connectionName>james-imap-folders</connectionName>
+ <password/>
+ <servername/>
+ <sslport/>
+ <username/>
+ <usessl>N</usessl>
+ <attachmentfolder/>
+ <attachmentwildcard/>
+ <bodysearch/>
+ <createlocalfolder>Y</createlocalfolder>
+ <createmovetofolder>N</createmovetofolder>
+ <delete>N</delete>
+ <filenamepattern>msg_#IdFile#.mail</filenamepattern>
+ <firstmails/>
+ <imapfirstmails>0</imapfirstmails>
+ <imapfolder>INBOX</imapfolder>
+ <includesubfolders>Y</includesubfolders>
+ <movetoimapfolder>INBOX.z</movetoimapfolder>
+ <nottermbodysearch>N</nottermbodysearch>
+ <nottermreceipientsearh>N</nottermreceipientsearh>
+ <nottermreceiveddatesearch>N</nottermreceiveddatesearch>
+ <nottermsendersearch>N</nottermsendersearch>
+ <nottermsubjectsearch>N</nottermsubjectsearch>
+
<outputdirectory>${PROJECT_HOME}/output/0010-move-connection</outputdirectory>
+ <protocol>IMAP</protocol>
+ <proxyusername/>
+ <receivedDate1/>
+ <receivedDate2/>
+ <recipientsearch/>
+ <retrievemails>0</retrievemails>
+ <actiontype>get</actiontype>
+ <valueimaplist>imaplistall</valueimaplist>
+ <conditionreceiveddate>ignore</conditionreceiveddate>
+ <aftergetimap>move</aftergetimap>
+ <saveattachment>N</saveattachment>
+ <savemessage>Y</savemessage>
+ <sendersearch/>
+ <subjectsearch/>
+ <usedifferentfolderforattachment>N</usedifferentfolderforattachment>
+ <useproxy>N</useproxy>
+ <usexoauth2>N</usexoauth2>
+ <parallel>N</parallel>
+ <xloc>464</xloc>
+ <yloc>704</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>check get and move counters (connection)</name>
+ <description/>
+ <type>EVAL</type>
+ <attributes/>
+ <script>var written = previous_result.getNrLinesWritten();
+var updated = previous_result.getNrLinesUpdated();
+var deleted = previous_result.getNrLinesDeleted();
+log.logBasic("Get and move incl. subfolders (connection): lines written = " +
written + ", lines updated = " + updated + ", lines deleted = " + deleted);
+written == 4 && updated == 4 && deleted == 0;</script>
+ <parallel>N</parallel>
+ <xloc>688</xloc>
+ <yloc>704</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>check saved files and folders after move (connection)</name>
+ <description/>
+ <type>SHELL</type>
+ <attributes/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <arg_from_previous>N</arg_from_previous>
+ <arguments>
+</arguments>
+ <exec_per_row>N</exec_per_row>
+ <insertScript>Y</insertScript>
+ <loglevel>Basic</loglevel>
+ <script>#!/bin/bash
+# Every message saved once, and all 4 now in INBOX.z: none read or copied
again from INBOX.z.
+WEBADMIN="http://${MAIL_SMTP_SERVER}:8000"
+MAILUSER="[email protected]"
+OUT="${PROJECT_HOME}/output/0010-move-connection"
+errors=0
+files=$(ls "$OUT")
+nfiles=$(ls "$OUT" | wc -l)
+echo "Saved message files: " $files
+if [ "$nfiles" != 4 ]; then
+ echo "Expected 4 message files, found $nfiles"
+ errors=$((errors + 1))
+fi
+for n in 1 2 3 4; do
+ found=$(grep -l "^Subject: Message $n" "$OUT"/*.mail 2>/dev/null | wc -l)
+ if [ "$found" != 1 ]; then
+ echo "Message $n was found in $found files, expected 1"
+ errors=$((errors + 1))
+ fi
+done
+for expected in INBOX:0 INBOX.a:0 INBOX.z:4; do
+ mailbox=${expected%%:*}
+ count=$(curl -s "$WEBADMIN/users/$MAILUSER/mailboxes/$mailbox/messageCount")
+ echo "$mailbox holds $count messages"
+ if [ "$count" != "${expected##*:}" ]; then
+ echo "Expected ${expected##*:} messages in $mailbox"
+ errors=$((errors + 1))
+ fi
+done
+exit $errors
+</script>
+ <set_append_logfile>N</set_append_logfile>
+ <set_logfile>N</set_logfile>
+ <parallel>N</parallel>
+ <xloc>912</xloc>
+ <yloc>704</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>move all to INBOX.z incl. subfolders (connection)</name>
+ <description/>
+ <type>GET_POP</type>
+ <attributes/>
+ <connectionName>james-imap-folders</connectionName>
+ <password/>
+ <servername/>
+ <sslport/>
+ <username/>
+ <usessl>N</usessl>
+ <attachmentfolder/>
+ <attachmentwildcard/>
+ <bodysearch/>
+ <createlocalfolder>Y</createlocalfolder>
+ <createmovetofolder>N</createmovetofolder>
+ <delete>N</delete>
+ <filenamepattern>msg_#IdFile#.mail</filenamepattern>
+ <firstmails/>
+ <imapfirstmails>0</imapfirstmails>
+ <imapfolder>INBOX</imapfolder>
+ <includesubfolders>Y</includesubfolders>
+ <movetoimapfolder>INBOX.z</movetoimapfolder>
+ <nottermbodysearch>N</nottermbodysearch>
+ <nottermreceipientsearh>N</nottermreceipientsearh>
+ <nottermreceiveddatesearch>N</nottermreceiveddatesearch>
+ <nottermsendersearch>N</nottermsendersearch>
+ <nottermsubjectsearch>N</nottermsubjectsearch>
+
<outputdirectory>${PROJECT_HOME}/output/0010-move-connection</outputdirectory>
+ <protocol>IMAP</protocol>
+ <proxyusername/>
+ <receivedDate1/>
+ <receivedDate2/>
+ <recipientsearch/>
+ <retrievemails>0</retrievemails>
+ <actiontype>move</actiontype>
+ <valueimaplist>imaplistall</valueimaplist>
+ <conditionreceiveddate>ignore</conditionreceiveddate>
+ <aftergetimap>nothing</aftergetimap>
+ <saveattachment>N</saveattachment>
+ <savemessage>Y</savemessage>
+ <sendersearch/>
+ <subjectsearch/>
+ <usedifferentfolderforattachment>N</usedifferentfolderforattachment>
+ <useproxy>N</useproxy>
+ <usexoauth2>N</usexoauth2>
+ <parallel>N</parallel>
+ <xloc>1136</xloc>
+ <yloc>704</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>check move all counters (connection)</name>
+ <description/>
+ <type>EVAL</type>
+ <attributes/>
+ <script>var written = previous_result.getNrLinesWritten();
+var updated = previous_result.getNrLinesUpdated();
+var deleted = previous_result.getNrLinesDeleted();
+log.logBasic("Move all incl. subfolders, all messages already in INBOX.z
(connection): lines written = " + written + ", lines updated = " + updated + ",
lines deleted = " + deleted);
+updated == 0 && deleted == 0;</script>
+ <parallel>N</parallel>
+ <xloc>1360</xloc>
+ <yloc>704</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>abort</name>
+ <description/>
+ <type>ABORT</type>
+ <attributes/>
+ <always_log_rows>N</always_log_rows>
+ <message>Get mails over INBOX and its subfolders did not give the
expected result</message>
+ <parallel>N</parallel>
+ <xloc>1808</xloc>
+ <yloc>384</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>success</name>
+ <description/>
+ <type>SUCCESS</type>
+ <attributes/>
+
+ <parallel>N</parallel>
+ <xloc>1584</xloc>
+ <yloc>704</yloc>
+ <attributes_hac/>
+ </action>
+ </actions>
+ <hops>
+ <hop>
+ <from>Start</from>
+ <to>prepare mailbox (inline)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>Y</unconditional>
+ </hop>
+ <hop>
+ <from>prepare mailbox (inline)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>prepare mailbox (inline)</from>
+ <to>move first mail to INBOX.sub (inline)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>move first mail to INBOX.sub (inline)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>move first mail to INBOX.sub (inline)</from>
+ <to>check move counters (inline)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check move counters (inline)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check move counters (inline)</from>
+ <to>get mails incl. subfolders (inline)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>get mails incl. subfolders (inline)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>get mails incl. subfolders (inline)</from>
+ <to>check saved files (inline)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check saved files (inline)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check saved files (inline)</from>
+ <to>delete mails incl. subfolders (inline)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>delete mails incl. subfolders (inline)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>delete mails incl. subfolders (inline)</from>
+ <to>check delete counters (inline)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check delete counters (inline)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check delete counters (inline)</from>
+ <to>prepare mailbox (connection)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>prepare mailbox (connection)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>prepare mailbox (connection)</from>
+ <to>move first mail to INBOX.sub (connection)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>move first mail to INBOX.sub (connection)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>move first mail to INBOX.sub (connection)</from>
+ <to>check move counters (connection)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check move counters (connection)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check move counters (connection)</from>
+ <to>get mails incl. subfolders (connection)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>get mails incl. subfolders (connection)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>get mails incl. subfolders (connection)</from>
+ <to>check saved files (connection)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check saved files (connection)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check saved files (connection)</from>
+ <to>delete mails incl. subfolders (connection)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>delete mails incl. subfolders (connection)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>delete mails incl. subfolders (connection)</from>
+ <to>check delete counters (connection)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check delete counters (connection)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check delete counters (connection)</from>
+ <to>prepare mailbox for move (move target below INBOX, inline)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>prepare mailbox for move (move target below INBOX, inline)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>prepare mailbox for move (move target below INBOX, inline)</from>
+ <to>get and move to INBOX.z incl. subfolders (inline)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>get and move to INBOX.z incl. subfolders (inline)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>get and move to INBOX.z incl. subfolders (inline)</from>
+ <to>check get and move counters (inline)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check get and move counters (inline)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check get and move counters (inline)</from>
+ <to>check saved files and folders after move (inline)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check saved files and folders after move (inline)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check saved files and folders after move (inline)</from>
+ <to>move all to INBOX.z incl. subfolders (inline)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>move all to INBOX.z incl. subfolders (inline)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>move all to INBOX.z incl. subfolders (inline)</from>
+ <to>check move all counters (inline)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check move all counters (inline)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check move all counters (inline)</from>
+ <to>prepare mailbox for move (move target below INBOX, connection)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>prepare mailbox for move (move target below INBOX,
connection)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>prepare mailbox for move (move target below INBOX,
connection)</from>
+ <to>get and move to INBOX.z incl. subfolders (connection)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>get and move to INBOX.z incl. subfolders (connection)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>get and move to INBOX.z incl. subfolders (connection)</from>
+ <to>check get and move counters (connection)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check get and move counters (connection)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check get and move counters (connection)</from>
+ <to>check saved files and folders after move (connection)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check saved files and folders after move (connection)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check saved files and folders after move (connection)</from>
+ <to>move all to INBOX.z incl. subfolders (connection)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>move all to INBOX.z incl. subfolders (connection)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>move all to INBOX.z incl. subfolders (connection)</from>
+ <to>check move all counters (connection)</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check move all counters (connection)</from>
+ <to>abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>check move all counters (connection)</from>
+ <to>success</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ </hops>
+ <notepads>
+ </notepads>
+ <attributes/>
+</workflow>
diff --git
a/integration-tests/mail/metadata/mail-server-connection/james-imap-folders.json
b/integration-tests/mail/metadata/mail-server-connection/james-imap-folders.json
new file mode 100644
index 0000000000..be12b58119
--- /dev/null
+++
b/integration-tests/mail/metadata/mail-server-connection/james-imap-folders.json
@@ -0,0 +1,17 @@
+{
+ "proxyUsername" : "",
+ "trustedHosts" : "",
+ "checkServerIdentity" : false,
+ "useProxy" : false,
+ "serverPort" : "993",
+ "serverHost" : "${MAIL_SMTP_SERVER}",
+ "useXOAuth2" : false,
+ "password" : "Encrypted 2be98afc86aa7f2e4bb18bd63c99dbdde",
+ "protocol" : "IMAP",
+ "virtualPath" : "",
+ "secureConnectionType" : "SSL",
+ "useAuthentication" : true,
+ "name" : "james-imap-folders",
+ "useSecureAuthentication" : true,
+ "username" : "[email protected]"
+}
\ No newline at end of file
diff --git
a/plugins/misc/mail/src/main/java/org/apache/hop/mail/common/MailFolderOperations.java
b/plugins/misc/mail/src/main/java/org/apache/hop/mail/common/MailFolderOperations.java
new file mode 100644
index 0000000000..0773a6fe4b
--- /dev/null
+++
b/plugins/misc/mail/src/main/java/org/apache/hop/mail/common/MailFolderOperations.java
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.mail.common;
+
+import jakarta.mail.Flags;
+import jakarta.mail.Folder;
+import jakarta.mail.Message;
+import jakarta.mail.MessagingException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import lombok.experimental.UtilityClass;
+
+/**
+ * Moving, deleting and listing messages and folders, shared by {@code
MailServerConnection} and the
+ * legacy {@code MailConnection}. Each method returns the number of messages
to add to the caller's
+ * counter, and only once the whole operation succeeded. A moved message
counts as moved, not also
+ * as deleted.
+ */
+@UtilityClass
+public class MailFolderOperations {
+
+ /**
+ * Flag one message as deleted.
+ *
+ * @return the number of deleted messages (1)
+ */
+ public static int deleteMessage(Message message) throws MessagingException {
+ message.setFlag(Flags.Flag.DELETED, true);
+ return 1;
+ }
+
+ /**
+ * Flag messages of a folder as deleted.
+ *
+ * @return the number of deleted messages
+ */
+ public static int deleteMessages(Folder folder, Message[] messages) throws
MessagingException {
+ folder.setFlags(messages, new Flags(Flags.Flag.DELETED), true);
+ return messages.length;
+ }
+
+ /**
+ * Copy one message to the destination folder and flag the original as
deleted.
+ *
+ * @return the number of moved messages (1)
+ */
+ public static int moveMessage(Folder source, Message message, Folder
destination)
+ throws MessagingException {
+ source.copyMessages(new Message[] {message}, destination);
+ deleteMessage(message);
+ return 1;
+ }
+
+ /**
+ * Copy messages to the destination folder and flag the originals as deleted.
+ *
+ * @return the number of moved messages
+ */
+ public static int moveMessages(Folder source, Message[] messages, Folder
destination)
+ throws MessagingException {
+ source.copyMessages(messages, destination);
+ deleteMessages(source, messages);
+ return messages.length;
+ }
+
+ /**
+ * Full names of all folders below a folder, depth first and sorted by name
on each level, so the
+ * order is the same in every run. Leaf folders that cannot hold subfolders
are included too.
+ *
+ * @param folder the parent folder
+ * @return the full names of the subfolders
+ */
+ public static List<String> listSubfolders(Folder folder) throws
MessagingException {
+ List<String> names = new ArrayList<>();
+ addSubfolders(folder, names);
+ return names;
+ }
+
+ private static void addSubfolders(Folder folder, List<String> names) throws
MessagingException {
+ if ((folder.getType() & Folder.HOLDS_FOLDERS) == 0) {
+ return;
+ }
+ Folder[] children = folder.list();
+ Arrays.sort(children, Comparator.comparing(Folder::getFullName));
+ for (Folder child : children) {
+ if ((child.getType() & (Folder.HOLDS_FOLDERS | Folder.HOLDS_MESSAGES))
!= 0) {
+ names.add(child.getFullName());
+ addSubfolders(child, names);
+ }
+ }
+ }
+}
diff --git
a/plugins/misc/mail/src/main/java/org/apache/hop/mail/metadata/MailServerConnection.java
b/plugins/misc/mail/src/main/java/org/apache/hop/mail/metadata/MailServerConnection.java
index 46794c8d98..c2534831a2 100644
---
a/plugins/misc/mail/src/main/java/org/apache/hop/mail/metadata/MailServerConnection.java
+++
b/plugins/misc/mail/src/main/java/org/apache/hop/mail/metadata/MailServerConnection.java
@@ -47,7 +47,6 @@ import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Date;
-import java.util.HashSet;
import java.util.Properties;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@@ -64,6 +63,7 @@ import org.apache.hop.core.variables.IVariables;
import org.apache.hop.core.vfs.HopVfs;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.mail.common.MailConst;
+import org.apache.hop.mail.common.MailFolderOperations;
import org.apache.hop.metadata.api.HopMetadata;
import org.apache.hop.metadata.api.HopMetadataBase;
import org.apache.hop.metadata.api.HopMetadataCategory;
@@ -528,34 +528,18 @@ public class MailServerConnection extends HopMetadataBase
implements IHopMetadat
addSearchTerm(new FlagTerm(new Flags(Flags.Flag.DRAFT), false));
}
- private HashSet<String> returnSubfolders(Folder folder) throws HopException {
- HashSet<String> list = new HashSet<>();
- try {
- if ((folder.getType() & Folder.HOLDS_FOLDERS) != 0) {
- Folder[] f = folder.list();
- for (Folder value : f) {
- // Search for sub folders
- if ((value.getType() & Folder.HOLDS_FOLDERS) != 0) {
- list.add(value.getFullName());
- list.addAll(returnSubfolders(value));
- }
- }
- }
- } catch (MessagingException m) {
- throw new HopException(m);
- }
- return list;
- }
-
/**
- * Returns all subfolders of the specified folder
+ * Returns all subfolders of the specified folder, in the same order in
every run
*
* @param folder parent folder
* @return sub folders
*/
public String[] returnAllFolders(Folder folder) throws HopException {
- HashSet<String> list = returnSubfolders(folder);
- return list.toArray(new String[0]);
+ try {
+ return MailFolderOperations.listSubfolders(folder).toArray(new
String[0]);
+ } catch (MessagingException m) {
+ throw new HopException(m);
+ }
}
/**
@@ -631,24 +615,6 @@ public class MailServerConnection extends HopMetadataBase
implements IHopMetadat
return this.messagenr;
}
- private void updateDeletedMessagesCounter() {
- this.nrDeletedMessages++;
- }
-
- private void setDeletedMessagesCounter() {
- this.nrDeletedMessages = getMessagesCount();
- }
-
- /** Update count of moved messages. */
- private void updatedMovedMessagesCounter() {
- this.nrMovedMessages++;
- }
-
- /** Set count of moved messages. */
- private void setMovedMessagesCounter() {
- this.nrMovedMessages = getMessagesCount();
- }
-
public void updateSavedMessagesCounter() {
this.nrSavedMessages++;
}
@@ -889,8 +855,7 @@ public class MailServerConnection extends HopMetadataBase
implements IHopMetadat
*/
public void deleteMessage() throws HopException {
try {
- this.message.setFlag(Flags.Flag.DELETED, true);
- updateDeletedMessagesCounter();
+ this.nrDeletedMessages +=
MailFolderOperations.deleteMessage(this.message);
} catch (Exception e) {
throw new HopException(
BaseMessages.getString(
@@ -908,9 +873,9 @@ public class MailServerConnection extends HopMetadataBase
implements IHopMetadat
*/
public void deleteMessages(boolean setCounter) throws HopException {
try {
- this.folder.setFlags(this.messages, new Flags(Flags.Flag.DELETED), true);
+ int deleted = MailFolderOperations.deleteMessages(this.folder,
this.messages);
if (setCounter) {
- setDeletedMessagesCounter();
+ this.nrDeletedMessages += deleted;
}
} catch (Exception e) {
throw new HopException(
@@ -926,11 +891,8 @@ public class MailServerConnection extends HopMetadataBase
implements IHopMetadat
*/
public void moveMessage() throws HopException {
try {
- // move all messages
- this.folder.copyMessages(new Message[] {this.message},
this.destinationIMAPFolder);
- updatedMovedMessagesCounter();
- // Make sure to delete messages
- deleteMessage();
+ this.nrMovedMessages +=
+ MailFolderOperations.moveMessage(this.folder, this.message,
this.destinationIMAPFolder);
} catch (Exception e) {
throw new HopException(
BaseMessages.getString(
@@ -949,9 +911,8 @@ public class MailServerConnection extends HopMetadataBase
implements IHopMetadat
*/
public void moveMessages() throws HopException {
try {
- this.folder.copyMessages(this.messages, this.destinationIMAPFolder);
- deleteMessages(false);
- setMovedMessagesCounter();
+ this.nrMovedMessages +=
+ MailFolderOperations.moveMessages(this.folder, this.messages,
this.destinationIMAPFolder);
} catch (Exception e) {
throw new HopException(
BaseMessages.getString(
diff --git
a/plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/ActionGetPOP.java
b/plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/ActionGetPOP.java
index 669962dd87..f6b2063add 100644
---
a/plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/ActionGetPOP.java
+++
b/plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/ActionGetPOP.java
@@ -18,6 +18,7 @@
package org.apache.hop.mail.workflow.actions.getpop;
import jakarta.mail.Flags.Flag;
+import jakarta.mail.Folder;
import jakarta.mail.Session;
import java.io.IOException;
import java.text.DateFormat;
@@ -660,33 +661,14 @@ public class ActionGetPOP extends ActionBase implements
Cloneable, IAction {
df);
if (isIncludeSubFolders()) {
- // Fetch also sub folders?
- if (isDebug()) {
- logDebug(BaseMessages.getString(PKG,
"ActionGetPOP.FetchingSubFolders"));
- }
- String[] subfolders = null;
- if (connection != null) {
- subfolders = connection.returnAllFolders();
- } else {
- subfolders = mailConn.returnAllFolders();
- }
- if (subfolders.length == 0) {
- if (isDebug()) {
- logDebug(BaseMessages.getString(PKG, "ActionGetPOP.NoSubFolders"));
- }
- } else {
- for (String subfolder : subfolders) {
- fetchOneFolder(
- usePOP3,
- subfolder,
- realOutputFolder,
- targetAttachmentFolder,
- realMoveToIMAPFolder,
- realFilenamePattern,
- nbrmailtoretrieve,
- df);
- }
- }
+ fetchSubfolders(
+ usePOP3,
+ realOutputFolder,
+ targetAttachmentFolder,
+ realMoveToIMAPFolder,
+ realFilenamePattern,
+ nbrmailtoretrieve,
+ df);
}
result.setResult(true);
@@ -748,6 +730,57 @@ public class ActionGetPOP extends ActionBase implements
Cloneable, IAction {
return result;
}
+ /**
+ * Fetch the subfolders of the current folder. The folder that messages are
moved into is skipped:
+ * its messages would otherwise be read, saved and moved a second time.
+ */
+ void fetchSubfolders(
+ boolean usePOP3,
+ String realOutputFolder,
+ String targetAttachmentFolder,
+ String realMoveToIMAPFolder,
+ String realFilenamePattern,
+ int nbrmailtoretrieve,
+ SimpleDateFormat df)
+ throws HopException {
+ if (isDebug()) {
+ logDebug(BaseMessages.getString(PKG, "ActionGetPOP.FetchingSubFolders"));
+ }
+ String[] subfolders;
+ Folder moveTarget;
+ if (connection != null) {
+ subfolders = connection.returnAllFolders();
+ moveTarget = connection.getDestinationIMAPFolder();
+ } else {
+ subfolders = mailConn.returnAllFolders();
+ moveTarget = mailConn.getDestinationIMAPFolder();
+ }
+ if (subfolders.length == 0) {
+ if (isDebug()) {
+ logDebug(BaseMessages.getString(PKG, "ActionGetPOP.NoSubFolders"));
+ }
+ return;
+ }
+ for (String subfolder : subfolders) {
+ if (moveTarget != null && subfolder.equals(moveTarget.getFullName())) {
+ if (isDetailed()) {
+ logDetailed(
+ BaseMessages.getString(PKG,
"ActionGetPOP.SkippingMoveTargetFolder", subfolder));
+ }
+ continue;
+ }
+ fetchOneFolder(
+ usePOP3,
+ subfolder,
+ realOutputFolder,
+ targetAttachmentFolder,
+ realMoveToIMAPFolder,
+ realFilenamePattern,
+ nbrmailtoretrieve,
+ df);
+ }
+ }
+
void fetchOneFolder(
boolean usePOP3,
String realIMAPFolder,
@@ -903,7 +936,8 @@ public class ActionGetPOP extends ActionBase implements
Cloneable, IAction {
}
if (isSaveMessage()) {
// get local message filename
- String localfilenameMessage =
replaceTokens(realFilenamePattern, i);
+ String localfilenameMessage =
+ replaceTokens(realFilenamePattern,
connection.getSavedMessagesCounter());
if (isDebug()) {
logDebug(
@@ -1072,7 +1106,8 @@ public class ActionGetPOP extends ActionBase implements
Cloneable, IAction {
}
if (isSaveMessage()) {
// get local message filename
- String localfilenameMessage =
replaceTokens(realFilenamePattern, i);
+ String localfilenameMessage =
+ replaceTokens(realFilenamePattern,
mailConn.getSavedMessagesCounter());
if (isDebug()) {
logDebug(
diff --git
a/plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/MailConnection.java
b/plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/MailConnection.java
index 23590f4b95..a786fd7ab5 100644
---
a/plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/MailConnection.java
+++
b/plugins/misc/mail/src/main/java/org/apache/hop/mail/workflow/actions/getpop/MailConnection.java
@@ -46,7 +46,6 @@ import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Date;
-import java.util.HashSet;
import java.util.Properties;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@@ -60,6 +59,7 @@ import org.apache.hop.core.logging.ILogChannel;
import org.apache.hop.core.util.Utils;
import org.apache.hop.core.vfs.HopVfs;
import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.mail.common.MailFolderOperations;
import org.eclipse.angus.mail.imap.IMAPSSLStore;
import org.eclipse.angus.mail.pop3.POP3SSLStore;
@@ -911,8 +911,7 @@ public class MailConnection {
*/
public void deleteMessage() throws HopException {
try {
- this.message.setFlag(Flag.DELETED, true);
- updateDeletedMessagesCounter();
+ this.nrDeletedMessages +=
MailFolderOperations.deleteMessage(this.message);
} catch (Exception e) {
throw new HopException(
BaseMessages.getString(
@@ -959,11 +958,8 @@ public class MailConnection {
*/
public void moveMessage() throws HopException {
try {
- // move all messages
- this.folder.copyMessages(new Message[] {this.message},
this.destinationIMAPFolder);
- updatedMovedMessagesCounter();
- // Make sure to delete messages
- deleteMessage();
+ this.nrMovedMessages +=
+ MailFolderOperations.moveMessage(this.folder, this.message,
this.destinationIMAPFolder);
} catch (Exception e) {
throw new HopException(
BaseMessages.getString(
@@ -1039,14 +1035,6 @@ public class MailConnection {
return this.nrDeletedMessages;
}
- private void updateDeletedMessagesCounter() {
- this.nrDeletedMessages++;
- }
-
- private void setDeletedMessagesCounter() {
- this.nrDeletedMessages = getMessagesCount();
- }
-
/**
* Returns count of moved messages.
*
@@ -1056,16 +1044,6 @@ public class MailConnection {
return this.nrMovedMessages;
}
- /** Update count of moved messages. */
- private void updatedMovedMessagesCounter() {
- this.nrMovedMessages++;
- }
-
- /** Set count of moved messages. */
- private void setMovedMessagesCounter() {
- this.nrMovedMessages = getMessagesCount();
- }
-
/**
* Delete messages.
*
@@ -1073,9 +1051,9 @@ public class MailConnection {
*/
public void deleteMessages(boolean setCounter) throws HopException {
try {
- this.folder.setFlags(this.messages, new Flags(Flag.DELETED), true);
+ int deleted = MailFolderOperations.deleteMessages(this.folder,
this.messages);
if (setCounter) {
- setDeletedMessagesCounter();
+ this.nrDeletedMessages += deleted;
}
} catch (Exception e) {
throw new HopException(
@@ -1090,9 +1068,8 @@ public class MailConnection {
*/
public void moveMessages() throws HopException {
try {
- this.folder.copyMessages(this.messages, this.destinationIMAPFolder);
- deleteMessages(false);
- setMovedMessagesCounter();
+ this.nrMovedMessages +=
+ MailFolderOperations.moveMessages(this.folder, this.messages,
this.destinationIMAPFolder);
} catch (Exception e) {
throw new HopException(
BaseMessages.getString(
@@ -1130,35 +1107,18 @@ public class MailConnection {
return retval;
}
- private HashSet<String> returnSubfolders(Folder folder) throws HopException {
- HashSet<String> list = new HashSet<>();
- try {
- if ((folder.getType() & Folder.HOLDS_FOLDERS) != 0) {
- Folder[] f = folder.list();
- for (Folder value : f) {
- // Search for sub folders
- if ((value.getType() & Folder.HOLDS_FOLDERS) != 0) {
- list.add(value.getFullName());
- list.addAll(returnSubfolders(value));
- }
- }
- }
- } catch (MessagingException m) {
- throw new HopException(m);
- }
- return list;
- }
-
/**
- * Returns all subfolders of the specified folder
+ * Returns all subfolders of the specified folder, in the same order in
every run
*
* @param folder parent folder
* @return sub folders
*/
public String[] returnAllFolders(Folder folder) throws HopException {
- HashSet<String> list = new HashSet<>();
- list = returnSubfolders(folder);
- return list.toArray(new String[list.size()]);
+ try {
+ return MailFolderOperations.listSubfolders(folder).toArray(new
String[0]);
+ } catch (MessagingException m) {
+ throw new HopException(m);
+ }
}
/**
diff --git
a/plugins/misc/mail/src/main/resources/org/apache/hop/mail/workflow/actions/getpop/messages/messages_en_US.properties
b/plugins/misc/mail/src/main/resources/org/apache/hop/mail/workflow/actions/getpop/messages/messages_en_US.properties
index 3cc40c4bfc..6c7597f074 100644
---
a/plugins/misc/mail/src/main/resources/org/apache/hop/mail/workflow/actions/getpop/messages/messages_en_US.properties
+++
b/plugins/misc/mail/src/main/resources/org/apache/hop/mail/workflow/actions/getpop/messages/messages_en_US.properties
@@ -141,6 +141,7 @@ ActionGetPOP.NegateSubject.Tooltip=Negate subject
ActionGetPOP.NoNameMessageBox.Message=Please name this action
ActionGetPOP.NoNameMessageBox.Text=Unnamed action
ActionGetPOP.NoSubFolders=No subfolder found!
+ActionGetPOP.SkippingMoveTargetFolder=Skipping folder {0}: messages are moved
into it
ActionGetPOP.OpenCalendar=Get calendar
ActionGetPOP.OutputDirectory.Label=Target directory
ActionGetPOP.OutputDirectory.Tooltip=Target directory
diff --git
a/plugins/misc/mail/src/test/java/org/apache/hop/mail/common/MailFolderOperationsTest.java
b/plugins/misc/mail/src/test/java/org/apache/hop/mail/common/MailFolderOperationsTest.java
new file mode 100644
index 0000000000..70207e46d2
--- /dev/null
+++
b/plugins/misc/mail/src/test/java/org/apache/hop/mail/common/MailFolderOperationsTest.java
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.mail.common;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import jakarta.mail.Flags;
+import jakarta.mail.Folder;
+import jakarta.mail.Message;
+import jakarta.mail.MessagingException;
+import java.util.List;
+import org.junit.jupiter.api.Test;
+
+class MailFolderOperationsTest {
+
+ @Test
+ void moveMessageCopiesFlagsAndCountsOne() throws Exception {
+ Folder source = mock(Folder.class);
+ Folder destination = mock(Folder.class);
+ Message message = mock(Message.class);
+
+ assertEquals(1, MailFolderOperations.moveMessage(source, message,
destination));
+
+ verify(source).copyMessages(new Message[] {message}, destination);
+ verify(message).setFlag(Flags.Flag.DELETED, true);
+ }
+
+ /** A copied message whose original could not be flagged is not reported as
moved. */
+ @Test
+ void moveMessageThrowsWhenFlaggingFails() throws Exception {
+ Folder source = mock(Folder.class);
+ Message message = mock(Message.class);
+ doThrow(new
MessagingException("read-only")).when(message).setFlag(Flags.Flag.DELETED,
true);
+
+ assertThrows(
+ MessagingException.class,
+ () -> MailFolderOperations.moveMessage(source, message,
mock(Folder.class)));
+ }
+
+ @Test
+ void moveMessagesDoesNotCopyWhenNothingFlagged() throws Exception {
+ Folder source = mock(Folder.class);
+ Folder destination = mock(Folder.class);
+ Message[] messages = {mock(Message.class), mock(Message.class)};
+ doThrow(new
MessagingException("down")).when(source).copyMessages(messages, destination);
+
+ assertThrows(
+ MessagingException.class,
+ () -> MailFolderOperations.moveMessages(source, messages,
destination));
+ verify(source, never()).setFlags(eq(messages), any(Flags.class), eq(true));
+ }
+
+ @Test
+ void moveMessagesCountsAll() throws Exception {
+ Folder source = mock(Folder.class);
+ Folder destination = mock(Folder.class);
+ Message[] messages = {mock(Message.class), mock(Message.class)};
+
+ assertEquals(2, MailFolderOperations.moveMessages(source, messages,
destination));
+
+ verify(source).copyMessages(messages, destination);
+ verify(source).setFlags(messages, new Flags(Flags.Flag.DELETED), true);
+ }
+
+ /**
+ * Subfolders come back depth first and sorted on each level, whatever order
the server lists them
+ * in, and leaf folders that cannot hold subfolders (\Noinferiors) are
included.
+ */
+ @Test
+ void listSubfoldersIsSortedAndIncludesLeafFolders() throws Exception {
+ Folder z = folder("INBOX.z", Folder.HOLDS_MESSAGES);
+ Folder bx = folder("INBOX.b.x", Folder.HOLDS_MESSAGES |
Folder.HOLDS_FOLDERS);
+ Folder b = folder("INBOX.b", Folder.HOLDS_MESSAGES | Folder.HOLDS_FOLDERS,
bx);
+ Folder a = folder("INBOX.a", Folder.HOLDS_MESSAGES);
+ Folder inbox = folder("INBOX", Folder.HOLDS_MESSAGES |
Folder.HOLDS_FOLDERS, z, b, a);
+
+ List<String> names = MailFolderOperations.listSubfolders(inbox);
+
+ assertEquals(List.of("INBOX.a", "INBOX.b", "INBOX.b.x", "INBOX.z"), names);
+ verify(a, never()).list();
+ verify(z, never()).list();
+ }
+
+ private static Folder folder(String fullName, int type, Folder... children)
+ throws MessagingException {
+ Folder folder = mock(Folder.class);
+ when(folder.getFullName()).thenReturn(fullName);
+ when(folder.getType()).thenReturn(type);
+ when(folder.list()).thenReturn(children);
+ return folder;
+ }
+}
diff --git
a/plugins/misc/mail/src/test/java/org/apache/hop/mail/metadata/MailServerConnectionTest.java
b/plugins/misc/mail/src/test/java/org/apache/hop/mail/metadata/MailServerConnectionTest.java
index ee698a2d5e..586ebd9b0f 100644
---
a/plugins/misc/mail/src/test/java/org/apache/hop/mail/metadata/MailServerConnectionTest.java
+++
b/plugins/misc/mail/src/test/java/org/apache/hop/mail/metadata/MailServerConnectionTest.java
@@ -19,10 +19,20 @@ package org.apache.hop.mail.metadata;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import jakarta.mail.Flags;
+import jakarta.mail.Folder;
+import jakarta.mail.Message;
+import jakarta.mail.MessagingException;
import java.util.Properties;
+import org.apache.commons.lang3.reflect.FieldUtils;
import org.apache.hop.core.HopClientEnvironment;
+import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.variables.Variables;
import org.apache.hop.mail.common.MailConst;
import org.apache.hop.metadata.api.IHopMetadataSerializer;
@@ -199,4 +209,61 @@ class MailServerConnectionTest {
assertEquals("true", props.getProperty("mail.smtp.starttls.enable"));
assertEquals(MailConst.SSL_TLS_V12,
props.getProperty("mail.smtp.ssl.protocols"));
}
+
+ /** A message moved on its own counts as moved, not also as deleted. */
+ @Test
+ void moveMessageCountsAsMovedNotDeleted() throws Exception {
+ MailServerConnection connection = new MailServerConnection(new
Variables());
+ Folder folder = mock(Folder.class);
+ Folder destination = mock(Folder.class);
+ Message message = mock(Message.class);
+ FieldUtils.writeField(connection, "folder", folder, true);
+ FieldUtils.writeField(connection, "destinationIMAPFolder", destination,
true);
+ FieldUtils.writeField(connection, "message", message, true);
+
+ connection.moveMessage();
+
+ verify(folder).copyMessages(new Message[] {message}, destination);
+ verify(message).setFlag(Flags.Flag.DELETED, true);
+ assertEquals(1, connection.getMovedMessagesCounter());
+ assertEquals(0, connection.getDeletedMessagesCounter());
+ }
+
+ /** A message that is copied but whose original can't be flagged is not
counted as moved. */
+ @Test
+ void moveMessageNotCountedWhenFlagFails() throws Exception {
+ MailServerConnection connection = new MailServerConnection(new
Variables());
+ Message message = mock(Message.class);
+ doThrow(new
MessagingException("read-only")).when(message).setFlag(Flags.Flag.DELETED,
true);
+ FieldUtils.writeField(connection, "folder", mock(Folder.class), true);
+ FieldUtils.writeField(connection, "destinationIMAPFolder",
mock(Folder.class), true);
+ FieldUtils.writeField(connection, "message", message, true);
+
+ assertThrows(HopException.class, connection::moveMessage);
+ assertEquals(0, connection.getMovedMessagesCounter());
+ assertEquals(0, connection.getDeletedMessagesCounter());
+ }
+
+ /** Deleting or moving all messages of several folders adds up the counts of
all folders. */
+ @Test
+ void bulkCountersAddUpOverFolders() throws Exception {
+ MailServerConnection connection = new MailServerConnection(new
Variables());
+ FieldUtils.writeField(connection, "folder", mock(Folder.class), true);
+ FieldUtils.writeField(connection, "destinationIMAPFolder",
mock(Folder.class), true);
+ Message[] firstFolder = {mock(Message.class), mock(Message.class)};
+ Message[] secondFolder = {mock(Message.class)};
+
+ for (Message[] messages : new Message[][] {firstFolder, secondFolder}) {
+ FieldUtils.writeField(connection, "messages", messages, true);
+ connection.deleteMessages(true);
+ }
+ assertEquals(3, connection.getDeletedMessagesCounter());
+
+ for (Message[] messages : new Message[][] {firstFolder, secondFolder}) {
+ FieldUtils.writeField(connection, "messages", messages, true);
+ connection.moveMessages();
+ }
+ assertEquals(3, connection.getMovedMessagesCounter());
+ assertEquals(3, connection.getDeletedMessagesCounter());
+ }
}
diff --git
a/plugins/misc/mail/src/test/java/org/apache/hop/mail/workflow/actions/getpop/ActionGetPOPTest.java
b/plugins/misc/mail/src/test/java/org/apache/hop/mail/workflow/actions/getpop/ActionGetPOPTest.java
index f5cd7d7dee..2870563d21 100644
---
a/plugins/misc/mail/src/test/java/org/apache/hop/mail/workflow/actions/getpop/ActionGetPOPTest.java
+++
b/plugins/misc/mail/src/test/java/org/apache/hop/mail/workflow/actions/getpop/ActionGetPOPTest.java
@@ -19,18 +19,34 @@ package org.apache.hop.mail.workflow.actions.getpop;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
-
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import jakarta.mail.Folder;
+import jakarta.mail.Message;
+import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.commons.lang3.reflect.FieldUtils;
import org.apache.hop.core.HopClientEnvironment;
import org.apache.hop.core.ICheckResult;
+import org.apache.hop.core.logging.LogLevel;
import org.apache.hop.core.variables.Variables;
import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
import org.apache.hop.metadata.serializer.memory.MemoryMetadataProvider;
import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
+import org.mockito.InOrder;
+import org.mockito.Mockito;
/** Unit test for {@link ActionGetPOP} */
class ActionGetPOPTest {
@@ -138,4 +154,64 @@ class ActionGetPOPTest {
assertEquals("Archive", action.getImapFolder());
assertTrue(action.isIncludeSubFolders());
}
+
+ /** #IdFile# keeps counting over the folders, so a subfolder doesn't
overwrite earlier files. */
+ @Test
+ void idFileCountsOverAllFolders() throws Exception {
+ ActionGetPOP action = newActionWithParentWorkflow();
+ MailConnection mailConn = mockMailConnectionCountingSaves();
+ FieldUtils.writeField(action, "mailConn", mailConn, true);
+
+ SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd");
+ action.fetchOneFolder(false, "INBOX", "/out", null, null,
"m_#IdFile#.mail", 0, df);
+ action.fetchOneFolder(false, "INBOX/sub", "/out", null, null,
"m_#IdFile#.mail", 0, df);
+
+ InOrder inOrder = Mockito.inOrder(mailConn);
+ for (int i = 1; i <= 4; i++) {
+ inOrder.verify(mailConn).saveMessageContentToFile(eq("m_" + i +
".mail"), anyString());
+ }
+ verify(mailConn, Mockito.times(4)).saveMessageContentToFile(anyString(),
anyString());
+ }
+
+ /** With Include subfolders, the folder that messages are moved into is not
fetched again. */
+ @Test
+ void subfoldersSkipTheMoveTarget() throws Exception {
+ ActionGetPOP action = newActionWithParentWorkflow();
+ MailConnection mailConn = mockMailConnectionCountingSaves();
+ Folder moveTarget = mock(Folder.class);
+ when(moveTarget.getFullName()).thenReturn("INBOX.z");
+ when(mailConn.getDestinationIMAPFolder()).thenReturn(moveTarget);
+ when(mailConn.returnAllFolders()).thenReturn(new String[] {"INBOX.a",
"INBOX.z", "INBOX.z.x"});
+ FieldUtils.writeField(action, "mailConn", mailConn, true);
+
+ SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd");
+ action.fetchSubfolders(false, "/out", null, "INBOX.z", "m_#IdFile#.mail",
0, df);
+
+ verify(mailConn).openFolder("INBOX.a", true);
+ verify(mailConn).openFolder("INBOX.z.x", true);
+ verify(mailConn, never()).openFolder("INBOX.z", true);
+ verify(mailConn, Mockito.times(4)).saveMessageContentToFile(anyString(),
anyString());
+ }
+
+ private static ActionGetPOP newActionWithParentWorkflow() {
+ ActionGetPOP action = new ActionGetPOP("mails");
+ action.setSaveAttachment(false);
+ IWorkflowEngine<WorkflowMeta> workflow = mock(IWorkflowEngine.class);
+ when(workflow.getLogLevel()).thenReturn(LogLevel.BASIC);
+ action.setParentWorkflow(workflow);
+ return action;
+ }
+
+ /** A connection with 2 messages in every folder, that counts saved messages
like the real one. */
+ private static MailConnection mockMailConnectionCountingSaves() throws
Exception {
+ MailConnection mailConn = mock(MailConnection.class);
+ when(mailConn.getMessagesCount()).thenReturn(2);
+ when(mailConn.getMessage()).thenReturn(mock(Message.class));
+ AtomicInteger saved = new AtomicInteger();
+ when(mailConn.getSavedMessagesCounter()).thenAnswer(invocation ->
saved.get());
+ doAnswer(invocation -> saved.incrementAndGet())
+ .when(mailConn)
+ .saveMessageContentToFile(anyString(), anyString());
+ return mailConn;
+ }
}
diff --git
a/plugins/misc/mail/src/test/java/org/apache/hop/mail/workflow/actions/getpop/MailConnectionTest.java
b/plugins/misc/mail/src/test/java/org/apache/hop/mail/workflow/actions/getpop/MailConnectionTest.java
index ac39416d0f..476f56bdf2 100644
---
a/plugins/misc/mail/src/test/java/org/apache/hop/mail/workflow/actions/getpop/MailConnectionTest.java
+++
b/plugins/misc/mail/src/test/java/org/apache/hop/mail/workflow/actions/getpop/MailConnectionTest.java
@@ -16,11 +16,15 @@
*/
package org.apache.hop.mail.workflow.actions.getpop;
+import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assertions.fail;
+import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
+import jakarta.mail.Flags;
import jakarta.mail.Folder;
+import jakarta.mail.Message;
import jakarta.mail.MessagingException;
import jakarta.mail.Store;
import java.io.File;
@@ -169,6 +173,50 @@ class MailConnectionTest {
}
}
+ /** A message moved on its own counts as moved, not also as deleted. */
+ @Test
+ void moveMessageCountsAsMovedNotDeleted() throws Exception {
+ Message message = Mockito.mock(Message.class);
+ when(conn.b.getMessages()).thenReturn(new Message[] {message});
+ conn.openFolder("a/b", false, true);
+ conn.setDestinationFolder("a/b/c", true);
+ conn.retrieveMessages();
+ conn.fetchNext();
+
+ conn.moveMessage();
+
+ verify(conn.b).copyMessages(new Message[] {message}, conn.c);
+ verify(message).setFlag(Flags.Flag.DELETED, true);
+ assertEquals(1, conn.getMovedMessagesCounter());
+ assertEquals(0, conn.getDeletedMessagesCounter());
+ }
+
+ /** Deleting or moving all messages of several folders adds up the counts of
all folders. */
+ @Test
+ void bulkCountersAddUpOverFolders() throws Exception {
+ when(conn.a.getMessages())
+ .thenReturn(new Message[] {Mockito.mock(Message.class),
Mockito.mock(Message.class)});
+ when(conn.b.getMessages()).thenReturn(new Message[]
{Mockito.mock(Message.class)});
+ conn.setDestinationFolder("a/b/c", true);
+
+ conn.openFolder("a", false, true);
+ conn.retrieveMessages();
+ conn.deleteMessages(true);
+ conn.openFolder("a/b", false, true);
+ conn.retrieveMessages();
+ conn.deleteMessages(true);
+ assertEquals(3, conn.getDeletedMessagesCounter());
+
+ conn.openFolder("a", false, true);
+ conn.retrieveMessages();
+ conn.moveMessages();
+ conn.openFolder("a/b", false, true);
+ conn.retrieveMessages();
+ conn.moveMessages();
+ assertEquals(3, conn.getMovedMessagesCounter());
+ assertEquals(3, conn.getDeletedMessagesCounter());
+ }
+
/** Test {@link MailConnection#folderExists(String)} method. */
@Test
void folderExistsTest() {