Hi
Yeah we can likely improve this to avoid fetching all the mails. You
are welcome to log a JIRA and work on a patch as github PR.
http://camel.apache.org/support.html
https://github.com/apache/camel/blob/master/CONTRIBUTING.md
On Wed, Mar 6, 2019 at 9:54 AM js504 <js...@protonmail.com.invalid> wrote:
>
> Hello,
>
> I am trying to poll a gmail account to process emails using the camel-mail
> version 2.23.0 imaps component and the Java DSL.
>
> There are thousands of emails I need to process but I am trying to split it
> up so as to not overload anything.
>
> I have tried setting maxMessagesPerPoll and fetchSize parameters to different
> numbers but no matter what it seems to try to peek every message first ( I
> think ) and doesn't seem to retrieve just the number of messages I have
> specified. I have tried setting peek to false as well but this does not seem
> to help. I have also tried setting just the maxMessagesPerPoll without
> fetchSize and peek as well as setting just fetchSize without
> maxMessagesPerPoll or peek set but the same occurs.
>
> below is my from statement using the imaps component:
>
> from("imaps://{{env:IMAPS_HOSTNAME}}:{{env:IMAPS_PORT}}?" +
> "maxMessagesPerPoll=10" +
> "&fetchSize=10" +
> "&peek=false" +
> "&username={{env:EMAIL_ADDRESS}}" +
> "&password=" + password +
> "&debugMode={{env:IMAP_DEBUG_MODE}}" +
> "&consumer.initialDelay=5000")
>
> The following is the output from the connection to the gmail account and the
> following '(BODY.PEEK[HEADER.FIELDS (Message-ID)])' messages that seem to go
> on and on for every message before retrieval of the message body has even
> begun.
> Any explanation as to what is happening and what I could do in order to
> process say 1000 emails at a time and stop it from doing this pre-processing
> on all emails instead of the first 1000 per poll would be really appreciated.
>
> 2019-03-05 21:10:01,531 INFO [main] (DefaultCamelContext.java:3197) - Apache
> Camel 2.23.1 (CamelContext: consume-emails) started in 0.969 seconds
> DEBUG: getProvider() returning
> javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle]
> DEBUG IMAPS: mail.imap.fetchsize: 16384
> DEBUG IMAPS: mail.imap.ignorebodystructuresize: false
> DEBUG IMAPS: mail.imap.statuscachetimeout: 1000
> DEBUG IMAPS: mail.imap.appendbuffersize: -1
> DEBUG IMAPS: mail.imap.minidletime: 10
> DEBUG IMAPS: closeFoldersOnStoreFailure
> DEBUG IMAPS: trying to connect to host "imap.gmail.com", port 993, isSSL true
> * OK Gimap ready for requests from 45.2.196.68 n3mb457716515jap
> A0 CAPABILITY
> * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN
> X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN
> AUTH=OAUTHBEARER AUTH=XOAUTH
> A0 OK Thats all she wrote! n3mb457716515jap
> DEBUG IMAPS: AUTH: XOAUTH2
> DEBUG IMAPS: AUTH: PLAIN
> DEBUG IMAPS: AUTH: PLAIN-CLIENTTOKEN
> DEBUG IMAPS: AUTH: OAUTHBEARER
> DEBUG IMAPS: AUTH: XOAUTH
> DEBUG IMAPS: protocolConnect login, host=imap.gmail.com, user=<removed by
> me>, password=<non-null>
> DEBUG IMAPS: AUTHENTICATE PLAIN command trace suppressed
> DEBUG IMAPS: AUTHENTICATE PLAIN command result: A1 OK <removed by me>
> authenticated (Success)
> A2 ENABLE UTF8=ACCEPT
> * ENABLED UTF8=ACCEPT
> A2 OK Success [THROTTLED]
> A3 LIST "" INBOX
> * LIST (\HasNoChildren) "/" "INBOX"
> A3 OK Success [THROTTLED]
> DEBUG IMAPS: connection available -- size: 1
> A4 SELECT INBOX
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing
> $Phishing \*)] Flags permitted.
> * OK [UIDVALIDITY 1] UIDs valid.
> * 78900 EXISTS
> * 0 RECENT
> * OK [UIDNEXT 78907] Predicted next UID.
> * OK [HIGHESTMODSEQ 3405665]
> A4 OK [READ-WRITE] INBOX selected. (Success) [THROTTLED]
> A5 SEARCH UNSEEN 1:78900
> * SEARCH 3 4 5 6 10 20 26 39 58 76 78 97 102 105 116 117 118 119 120 121 122
> 123 124 125 ... <truncated to shorten message> ... 78890 78891 78892 78893
> 78894 78895 78896 78897 78898 78899 78900
> A5 OK SEARCH completed (Success) [THROTTLED]
> A6 FETCH 3 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 3 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A6 OK Success [THROTTLED]
> A7 FETCH 4 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 4 FETCH (BODY[HEADER.FIELDS (Message-ID)] {65}
> Message-Id: <removed by me>
>
> )
> A7 OK Success [THROTTLED]
> A8 FETCH 5 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 5 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A8 OK Success [THROTTLED]
> A9 FETCH 6 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 6 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A9 OK Success [THROTTLED]
> A10 FETCH 10 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 10 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A10 OK Success [THROTTLED]
> A11 FETCH 20 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 20 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A11 OK Success [THROTTLED]
> A12 FETCH 26 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 26 FETCH (BODY[HEADER.FIELDS (Message-ID)] {65}
> Message-Id: <removed by me>
>
> )
> A12 OK Success [THROTTLED]
> A13 FETCH 39 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 39 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A13 OK Success [THROTTLED]
> A14 FETCH 58 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 58 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A14 OK Success [THROTTLED]
> A15 FETCH 76 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 76 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> )
> A15 OK Success [THROTTLED]
> A16 FETCH 78 (BODY.PEEK[HEADER.FIELDS (Message-ID)])
> * 78 FETCH (BODY[HEADER.FIELDS (Message-ID)] {64}
> Message-ID: <removed by me>
>
> and so on until for over an hour until I received ..
>
> 2019-03-06 00:07:58,263 WARN [Camel (consume-emails) thread #4 -
> imaps://imap.gmail.com:993] (DefaultMailUidGenerator.java:59) - Cannot read
> headers from mail message. This exception will be ignored.
> javax.mail.FolderClosedException: * BYE JavaMail Exception:
> java.io.IOException: Connection dropped by server?
> at com.sun.mail.imap.IMAPMessage.getHeader(IMAPMessage.java:958)
> ~[javax.mail-1.6.1.jar:1.6.1]
> at
> org.apache.camel.component.mail.DefaultMailUidGenerator.generateMessageIdHeader(DefaultMailUidGenerator.java:52)
> [camel-mail-2.23.0.jar:2.23.0]
> at
> org.apache.camel.component.mail.DefaultMailUidGenerator.generateUuid(DefaultMailUidGenerator.java:36)
> [camel-mail-2.23.0.jar:2.23.0]
> at
> org.apache.camel.component.mail.MailConsumer.retrieveMessages(MailConsumer.java:273)
> [camel-mail-2.23.0.jar:2.23.0]
> at org.apache.camel.component.mail.MailConsumer.poll(MailConsumer.java:119)
> [camel-mail-2.23.0.jar:2.23.0]
> at
> org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
> [camel-core-2.23.0.jar:2.23.0]
> at
> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)
> [camel-core-2.23.0.jar:2.23.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_181]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [?:1.8.0_181]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [?:1.8.0_181]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> 2019-03-06 00:07:58,275 WARN [Camel (consume-emails) thread #4 -
> imaps://imap.gmail.com:993] (DefaultMailUidGenerator.java:83) - Cannot read
> headers from mail message. This exception will be ignored.
> javax.mail.FolderClosedException: null
> at com.sun.mail.imap.IMAPMessage.getProtocol(IMAPMessage.java:169)
> ~[javax.mail-1.6.1.jar:1.6.1]
> at com.sun.mail.imap.IMAPMessage.loadHeaders(IMAPMessage.java:1581)
> ~[javax.mail-1.6.1.jar:1.6.1]
> at com.sun.mail.imap.IMAPMessage.getAllHeaders(IMAPMessage.java:1016)
> ~[javax.mail-1.6.1.jar:1.6.1]
> at
> org.apache.camel.component.mail.DefaultMailUidGenerator.generateMessageHash(DefaultMailUidGenerator.java:73)
> [camel-mail-2.23.0.jar:2.23.0]
> at
> org.apache.camel.component.mail.DefaultMailUidGenerator.generateUuid(DefaultMailUidGenerator.java:38)
> [camel-mail-2.23.0.jar:2.23.0]
> at
> org.apache.camel.component.mail.MailConsumer.retrieveMessages(MailConsumer.java:273)
> [camel-mail-2.23.0.jar:2.23.0]
> at org.apache.camel.component.mail.MailConsumer.poll(MailConsumer.java:119)
> [camel-mail-2.23.0.jar:2.23.0]
> at
> org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
> [camel-core-2.23.0.jar:2.23.0]
> at
> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)
> [camel-core-2.23.0.jar:2.23.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_181]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [?:1.8.0_181]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [?:1.8.0_181]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
