Hello,
I am trying to poll a gmail account to process emails using the camel-mail
version 2.23.0 imaps component and the Java DSL.
There are thousands of emails I need to process but I am trying to split it up
so as to not overload anything.
I have tried setting maxMessagesPerPoll and fetchSize parameters to different
numbers but no matter what it seems to try to peek every message first ( I
think ) and doesn't seem to retrieve just the number of messages I have
specified. I have tried setting peek to false as well but this does not seem
to help. I have also tried setting just the maxMessagesPerPoll without
fetchSize and peek as well as setting just fetchSize without maxMessagesPerPoll
or peek set but the same occurs.
below is my from statement using the imaps component:
from("imaps://{{env:IMAPS_HOSTNAME}}:{{env:IMAPS_PORT}}?" +
"maxMessagesPerPoll=10" +
"&fetchSize=10" +
"&peek=false" +
"&username={{env:EMAIL_ADDRESS}}" +
"&password=" + password +
"&debugMode={{env:IMAP_DEBUG_MODE}}" +
"&consumer.initialDelay=5000")
The following is the output from the connection to the gmail account and the
following '(BODY.PEEK[HEADER.FIELDS (Message-ID)])' messages that seem to go
on and on for every message before retrieval of the message body has even begun.
Any explanation as to what is happening and what I could do in order to process
say 1000 emails at a time and stop it from doing this pre-processing on all
emails instead of the first 1000 per poll would be really appreciated.
2019-03-05 21:10:01,531 INFO [main] (DefaultCamelContext.java:3197) - Apache
Camel 2.23.1 (CamelContext: consume-emails) started in 0.969 seconds
DEBUG: getProvider() returning
javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle]
DEBUG IMAPS: mail.imap.fetchsize: 16384
DEBUG IMAPS: mail.imap.ignorebodystructuresize: false
DEBUG IMAPS: mail.imap.statuscachetimeout: 1000
DEBUG IMAPS: mail.imap.appendbuffersize: -1
DEBUG IMAPS: mail.imap.minidletime: 10
DEBUG IMAPS: closeFoldersOnStoreFailure
DEBUG IMAPS: trying to connect to host "imap.gmail.com", port 993, isSSL true
* OK Gimap ready for requests from 45.2.196.68 n3mb457716515jap
A0 CAPABILITY
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN
X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN
AUTH=OAUTHBEARER AUTH=XOAUTH
A0 OK Thats all she wrote! n3mb457716515jap
DEBUG IMAPS: AUTH: XOAUTH2
DEBUG IMAPS: AUTH: PLAIN
DEBUG IMAPS: AUTH: PLAIN-CLIENTTOKEN
DEBUG IMAPS: AUTH: OAUTHBEARER
DEBUG IMAPS: AUTH: XOAUTH
DEBUG IMAPS: protocolConnect login, host=imap.gmail.com, user=<removed by me>,
password=<non-null>
DEBUG IMAPS: AUTHENTICATE PLAIN command trace suppressed
DEBUG IMAPS: AUTHENTICATE PLAIN command result: A1 OK <removed by me>
authenticated (Success)
A2 ENABLE UTF8=ACCEPT
* ENABLED UTF8=ACCEPT
A2 OK Success [THROTTLED]
A3 LIST "" INBOX
* LIST (\HasNoChildren) "/" "INBOX"
A3 OK Success [THROTTLED]
DEBUG IMAPS: connection available -- size: 1
A4 SELECT INBOX
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing
$Phishing \*)] Flags permitted.
* OK [UIDVALIDITY 1] UIDs valid.
* 78900 EXISTS
* 0 RECENT
* OK [UIDNEXT 78907] Predicted next UID.
* OK [HIGHESTMODSEQ 3405665]
A4 OK [READ-WRITE] INBOX selected. (Success) [THROTTLED]
A5 SEARCH UNSEEN 1:78900
* SEARCH 3 4 5 6 10 20 26 39 58 76 78 97 102 105 116 117 118 119 120 121 122
123 124 125 ... <truncated to shorten message> ... 78890 78891 78892 78893
78894 78895 78896 78897 78898 78899 78900
A5 OK SEARCH completed (Success) [THROTTLED]
A6 FETCH 3 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 3 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>
)
A6 OK Success [THROTTLED]
A7 FETCH 4 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 4 FETCH (BODY[HEADER.FIELDS (Message-ID)] {65}
Message-Id: <removed by me>
)
A7 OK Success [THROTTLED]
A8 FETCH 5 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 5 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>
)
A8 OK Success [THROTTLED]
A9 FETCH 6 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 6 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>
)
A9 OK Success [THROTTLED]
A10 FETCH 10 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 10 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>
)
A10 OK Success [THROTTLED]
A11 FETCH 20 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 20 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>
)
A11 OK Success [THROTTLED]
A12 FETCH 26 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 26 FETCH (BODY[HEADER.FIELDS (Message-ID)] {65}
Message-Id: <removed by me>
)
A12 OK Success [THROTTLED]
A13 FETCH 39 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 39 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>
)
A13 OK Success [THROTTLED]
A14 FETCH 58 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 58 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>
)
A14 OK Success [THROTTLED]
A15 FETCH 76 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 76 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>
)
A15 OK Success [THROTTLED]
A16 FETCH 78 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
* 78 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
Message-ID: <removed by me>
and so on until for over an hour until I received ..
2019-03-06 00:07:58,263 WARN [Camel (consume-emails) thread #4 -
imaps://imap.gmail.com:993] (DefaultMailUidGenerator.java:59) - Cannot read
headers from mail message. This exception will be ignored.
javax.mail.FolderClosedException: * BYE JavaMail Exception:
java.io.IOException: Connection dropped by server?
at com.sun.mail.imap.IMAPMessage.getHeader(IMAPMessage.java:958)
~[javax.mail-1.6.1.jar:1.6.1]
at
org.apache.camel.component.mail.DefaultMailUidGenerator.generateMessageIdHeader(DefaultMailUidGenerator.java:52)
[camel-mail-2.23.0.jar:2.23.0]
at
org.apache.camel.component.mail.DefaultMailUidGenerator.generateUuid(DefaultMailUidGenerator.java:36)
[camel-mail-2.23.0.jar:2.23.0]
at
org.apache.camel.component.mail.MailConsumer.retrieveMessages(MailConsumer.java:273)
[camel-mail-2.23.0.jar:2.23.0]
at org.apache.camel.component.mail.MailConsumer.poll(MailConsumer.java:119)
[camel-mail-2.23.0.jar:2.23.0]
at
org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
[camel-core-2.23.0.jar:2.23.0]
at
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)
[camel-core-2.23.0.jar:2.23.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_181]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_181]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_181]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2019-03-06 00:07:58,275 WARN [Camel (consume-emails) thread #4 -
imaps://imap.gmail.com:993] (DefaultMailUidGenerator.java:83) - Cannot read
headers from mail message. This exception will be ignored.
javax.mail.FolderClosedException: null
at com.sun.mail.imap.IMAPMessage.getProtocol(IMAPMessage.java:169)
~[javax.mail-1.6.1.jar:1.6.1]
at com.sun.mail.imap.IMAPMessage.loadHeaders(IMAPMessage.java:1581)
~[javax.mail-1.6.1.jar:1.6.1]
at com.sun.mail.imap.IMAPMessage.getAllHeaders(IMAPMessage.java:1016)
~[javax.mail-1.6.1.jar:1.6.1]
at
org.apache.camel.component.mail.DefaultMailUidGenerator.generateMessageHash(DefaultMailUidGenerator.java:73)
[camel-mail-2.23.0.jar:2.23.0]
at
org.apache.camel.component.mail.DefaultMailUidGenerator.generateUuid(DefaultMailUidGenerator.java:38)
[camel-mail-2.23.0.jar:2.23.0]
at
org.apache.camel.component.mail.MailConsumer.retrieveMessages(MailConsumer.java:273)
[camel-mail-2.23.0.jar:2.23.0]
at org.apache.camel.component.mail.MailConsumer.poll(MailConsumer.java:119)
[camel-mail-2.23.0.jar:2.23.0]
at
org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
[camel-core-2.23.0.jar:2.23.0]
at
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)
[camel-core-2.23.0.jar:2.23.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_181]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_181]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_181]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
