Re: IMAP Search skipping results

[Mark Waddingham via use-livecode]

On 2020-03-23 18:02, Bob Sneidar via use-livecode wrote:

Ahh hah hah! That’s funny. I can’t even write a snippet without
debugging it! :-)


Debugging is just a means to discover the assumptions you made which 
were invalid! ;)


Warmest Regards,

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Bob Sneidar via use-livecode]

Ahh hah hah! That’s funny. I can’t even write a snippet without debugging it! 
:-)

Bob S


> On Mar 23, 2020, at 11:01 AM, Mark Waddingham via use-livecode 
>  wrote:
> 
> On 2020-03-23 17:49, Pi Digital via use-livecode wrote:
>> Hey, are you insinuating my code is flawed!?! :-o
>> The cheek of it! ;)
> 
> Lol - well we all write flawed code from time to time...
> 
> More often though we write code which is perfect but based on assumptions we 
> didn't realize could be invalid (that's usually much harder to discern!)
> 
> Anyway, something here is bugging me - I do wonder if it is the manner in 
> which cURL (via tsNet) is being used (in this instance) is actually causing 
> the bug to manifest. IMAP is a session-based protocol, so unless cURL does 
> something magic with its 'custom URL' support (which is how cURL implements 
> IMAP), I'm puzzled how issuing two separate URL requests can actually work 
> (unless it isn't actually meant to, but just seems to).
> 
> I'll experiment later on before I finish for the day.
> 
> Warmest Regards,
> 
> Mark.
> 
> -- 
> Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
> LiveCode: Everyone can create apps
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Mark Waddingham via use-livecode]

On 2020-03-23 17:49, Pi Digital via use-livecode wrote:

Hey, are you insinuating my code is flawed!?! :-o
The cheek of it! ;)


Lol - well we all write flawed code from time to time...

More often though we write code which is perfect but based on 
assumptions we didn't realize could be invalid (that's usually much 
harder to discern!)


Anyway, something here is bugging me - I do wonder if it is the manner 
in which cURL (via tsNet) is being used (in this instance) is actually 
causing the bug to manifest. IMAP is a session-based protocol, so unless 
cURL does something magic with its 'custom URL' support (which is how 
cURL implements IMAP), I'm puzzled how issuing two separate URL requests 
can actually work (unless it isn't actually meant to, but just seems 
to).


I'll experiment later on before I finish for the day.

Warmest Regards,

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Pi Digital via use-livecode]

Hey, are you insinuating my code is flawed!?! :-o 
The cheek of it! ;)

Sean Cole
Pi Digital

.
> 
> I'd perhaps suggest that something isn't a workaround if the original 
> approach was flawed (however irksome the more code it requires might be!) ;)
> 
> Warmest Regards,
> 
> Mark.
> 
> -- 
> Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
> LiveCode: Everyone can create apps
> 
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Mark Waddingham via use-livecode]

On 2020-03-23 17:14, Pi Digital via use-livecode wrote:

Finding the first unseen UID though seems to be the next issue I’m
going to have to overcome. A never ending stream of workarounds.


Well, unfortunately, this is a bug in cURL as far as we can see - its 
been there for 6 years and no-one has fixed it.


This is kind of surprising as cURL is generally a very well supported 
and widely used open-source library (that being said, we should probably 
take note of the whole OpenSSL Heartbleed issue a while back...)


Given that cURL is used in a lot of places, by a lot of products, that 
suggests that either


  (a) no-one uses cURL IMAP very much so there isn't any interest in 
fixing it


*or*

  (b) fixing that bug doesn't actually solve the problem (it just shifts 
it). i.e. The bug makes quick-and-dirty approaches to certain tasks 
fail; but when people come across it they realize that if they don't do 
things 'en-masse' (which is basically what the IMAP RFC suggests you 
shouldn't) then it doesn't have an effect.


Of course, I'm hypothesizing here: (a) could be the reason as much as 
(b).


I'd perhaps suggest that something isn't a workaround if the original 
approach was flawed (however irksome the more code it requires might 
be!) ;)


Warmest Regards,

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Pi Digital via use-livecode]

Thanks Brian and Mark

A combination may well help. 

I am not for this project needing ALL but have been using UNSEEN. The mailboxes 
we are referencing have around 100-1200 emails coming in from O2 every morning 
at 4am. These then get processed at 5am to update their databases. I will have 
to first find the FIRST UNSEEN UID and then run a process to get the UIDs of 
the next 100 (and repeat) till they run out I guess. Once I’ve built the list 
of UIDs I can carry on as normal processing each one in turn. 

Finding the first unseen UID though seems to be the next issue I’m going to 
have to overcome. A never ending stream of workarounds. 

Sean Cole
Pi Digital Productions Ltd


eMail Ts & Cs


> On 23 Mar 2020, at 16:48, Mark Waddingham via use-livecode 
>  wrote:
> 
> On 2020-03-23 15:21, Pi Digital via use-livecode wrote:
>> Thanks Mark, your input here is appreciated.
>> This reply in that forum wasn’t helpful, was it
>>> > Known bug #90 was reported in 2014. However it still occurs in 2019. Does
>>> > anybody know how to overcome this situation?
>>> Yes: by fixing the code!
>> In the thread it talks of pagination (as do you) but doesn’t give an
>> example of how to do it. How would we implement this in LC? If it is
>> not giving us the correct count how do we know how many pages we will
>> have to allow for and so on.
> 
> That's where I got the reference to pagination from - as I said it's not a 
> particularly useful post there. I suspect the person replying assumed the 
> person asking the question had a detailed understanding of IMAP queries (my 
> knowledge of interacting with IMAP servers is more than it was earlier on, 
> but still not really sufficient to give any detailed advice).
> 
> The tsNet support for IMAP comes entirely from cURL - so it works pretty much 
> the same as the 'curl' shell command.
> 
> I found this with a quick search which might help:
> 
>  
> 
> The queries listed there at least appear to give ways to query the count of 
> things in a mailbox.
> 
> This page also looks like it might be helpful:
> 
>  
> 
> Also this has some more info on using CURL to talk to IMAP:
> 
>  
> 
> 
> Having pondered this a bit, I'm not sure that even if cURL guys fixed the 
> above bug it would actually make a difference to what you need to do to have 
> 100% correct code. The method of fetching all UIDs of messages in a mailbox 
> at once is completely unscalable - imagine an INBOX with 100,000s of messages 
> for example over a slow/flaky connection; or an INBOX which has a lot of 
> traffic and changes exceptionally frequently; or sits on a heavily loaded 
> server (you can imagine that no server would like to be polled for complete 
> message lists all the time by lots of clients).
> 
> The IMAP protocol appears to be designed to be treated as something to use to 
> synchronize local and remote state. Indeed, there is quite an extensive 
> description of how to 'synchronize with an IMAP server' in an RFC:
> 
>  
> 
> Indeed, it states in one place:
> 
> 
>  The following is an example of the first FETCH:
> 
>   C: A011 UID fetch 131:* (FLAGS BODYSTRUCTURE INTERNALDATE
>   RFC822.SIZE)
> 
>   Note 1: The first FETCH may result in the server's sending a huge
>   volume of data.  A smart disconnected client should use message
>   ranges (see also Section 3.2.1.2 of [RFC2683]), so that the user is
>   able to execute a different operation between fetching information
>   for a group of new messages.
> 
> 
> The unique id used to perform the sync operation is the UID; you can fetch 
> ranges of those as they are 'guaranteed' to always increase and never be 
> re-used. (The only caveat is the UIDVALIDITY, which is basically a 
> cache-generation number - i.e. if the UIDVALIDITY changes you have to dump 
> your local cache and start from scratch as it means the meaning of individual 
> UIDs has changed).
> 
> Not sure how much the above helps...
> 
> Warmest Regards,
> 
> Mark.
> 
> P.S. I've been chatting to Seb about the issue (he reported the bug 
> originally) - he has confirmed that the problem affects Kognition's IMAP code 
> too (that component is still in development at this stage which is why it 
> hadn't been noticed until I asked him to check). I had hoped the approach 
> there might have been slightly different and so provided a viable workaround, 
> but it would appear that it is not the case :(
> 
> -- 
> Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
> LiveCode: Everyone can create apps
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> 

Re: IMAP Search skipping results

[Mark Waddingham via use-livecode]

On 2020-03-23 15:21, Pi Digital via use-livecode wrote:

Thanks Mark, your input here is appreciated.

This reply in that forum wasn’t helpful, was it


> Known bug #90 was reported in 2014. However it still occurs in 2019. Does
> anybody know how to overcome this situation?

Yes: by fixing the code!


In the thread it talks of pagination (as do you) but doesn’t give an
example of how to do it. How would we implement this in LC? If it is
not giving us the correct count how do we know how many pages we will
have to allow for and so on.


That's where I got the reference to pagination from - as I said it's not 
a particularly useful post there. I suspect the person replying assumed 
the person asking the question had a detailed understanding of IMAP 
queries (my knowledge of interacting with IMAP servers is more than it 
was earlier on, but still not really sufficient to give any detailed 
advice).


The tsNet support for IMAP comes entirely from cURL - so it works pretty 
much the same as the 'curl' shell command.


I found this with a quick search which might help:

  

The queries listed there at least appear to give ways to query the count 
of things in a mailbox.


This page also looks like it might be helpful:

  

Also this has some more info on using CURL to talk to IMAP:

  



Having pondered this a bit, I'm not sure that even if cURL guys fixed 
the above bug it would actually make a difference to what you need to do 
to have 100% correct code. The method of fetching all UIDs of messages 
in a mailbox at once is completely unscalable - imagine an INBOX with 
100,000s of messages for example over a slow/flaky connection; or an 
INBOX which has a lot of traffic and changes exceptionally frequently; 
or sits on a heavily loaded server (you can imagine that no server would 
like to be polled for complete message lists all the time by lots of 
clients).


The IMAP protocol appears to be designed to be treated as something to 
use to synchronize local and remote state. Indeed, there is quite an 
extensive description of how to 'synchronize with an IMAP server' in an 
RFC:


  

Indeed, it states in one place:


  The following is an example of the first FETCH:

   C: A011 UID fetch 131:* (FLAGS BODYSTRUCTURE INTERNALDATE
   RFC822.SIZE)

   Note 1: The first FETCH may result in the server's sending a huge
   volume of data.  A smart disconnected client should use message
   ranges (see also Section 3.2.1.2 of [RFC2683]), so that the user is
   able to execute a different operation between fetching information
   for a group of new messages.


The unique id used to perform the sync operation is the UID; you can 
fetch ranges of those as they are 'guaranteed' to always increase and 
never be re-used. (The only caveat is the UIDVALIDITY, which is 
basically a cache-generation number - i.e. if the UIDVALIDITY changes 
you have to dump your local cache and start from scratch as it means the 
meaning of individual UIDs has changed).


Not sure how much the above helps...

Warmest Regards,

Mark.

P.S. I've been chatting to Seb about the issue (he reported the bug 
originally) - he has confirmed that the problem affects Kognition's IMAP 
code too (that component is still in development at this stage which is 
why it hadn't been noticed until I asked him to check). I had hoped the 
approach there might have been slightly different and so provided a 
viable workaround, but it would appear that it is not the case :(


--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Brian Milby via use-livecode]

Does ‘STATUS INBOX (UNSEEN)’ give the proper count reliably?  (Would actually 
need to use MESSAGES though to page through the whole inbox, but you could stop 
paging once all we’re found.)
If so, ranges are just specified with a : so you could use the sequence numbers 
to get the UIDs (1:100 then 101:200 ...) a piece at a time as the first 
argument to the search.

Thanks,
Brian
On Mar 23, 2020, 11:22 AM -0400, Pi Digital via use-livecode 
, wrote:
> Thanks Mark, your input here is appreciated.
>
> This reply in that forum wasn’t helpful, was it
> >
> > > Known bug #90 was reported in 2014. However it still occurs in 2019. Does
> > > anybody know how to overcome this situation?
> >
> > Yes: by fixing the code!
> >
> In the thread it talks of pagination (as do you) but doesn’t give an example 
> of how to do it. How would we implement this in LC? If it is not giving us 
> the correct count how do we know how many pages we will have to allow for and 
> so on.
>
> Regards
>
> Sean Cole
> Pi Digital
>
>
> > On 23 Mar 2020, at 12:11, Mark Waddingham via use-livecode 
> >  wrote:
> >
> > On 2020-03-23 11:39, Pi Digital via use-livecode wrote:
> > > No, same thing unfortunately. Even ALL has the issue. According to an
> > > old bug report that has sat for an age untouched it is a known issue
> > > in the ood curl lib they are using. At the rate they are currently
> > > dealing with bugs for fundamental operations we might see a fix in
> > > about a year or two (or four). [dig]
> >
> > In this case the bug is in CURL - it is 'known bug #90':
> >
> > 
> >
> > There's a few threads around about it - this is the most recent I could 
> > find:
> >
> > https://curl.haxx.se/mail/archive-2019-07/0029.html
> >
> > The only real suggestion (although it is vague) there is using some for of 
> > pagination (as the UIDs should be eternal, rather than related to the 
> > position in the mailbox requested).
> >
> > I couldn't find any explicit bug in their bug tracker which is still open 
> > for this, but its still listed as a known issue so it sounds like there has 
> > been no change in it unforunately :(
> >
> > Warmest Regards,
> >
> > Mark.
> >
> > --
> > Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
> > LiveCode: Everyone can create apps
> >
> > ___
> > use-livecode mailing list
> > use-livecode@lists.runrev.com
> > Please visit this url to subscribe, unsubscribe and manage your 
> > subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Pi Digital via use-livecode]

Thanks Mark, your input here is appreciated. 

This reply in that forum wasn’t helpful, was it
> 
> > Known bug #90 was reported in 2014. However it still occurs in 2019. Does 
> > anybody know how to overcome this situation? 
> 
> Yes: by fixing the code! 
> 
In the thread it talks of pagination (as do you) but doesn’t give an example of 
how to do it. How would we implement this in LC? If it is not giving us the 
correct count how do we know how many pages we will have to allow for and so 
on. 

Regards

Sean Cole
Pi Digital


> On 23 Mar 2020, at 12:11, Mark Waddingham via use-livecode 
>  wrote:
> 
> On 2020-03-23 11:39, Pi Digital via use-livecode wrote:
>> No, same thing unfortunately. Even ALL has the issue. According to an
>> old bug report that has sat for an age untouched it is a known issue
>> in the ood curl lib they are using. At the rate they are currently
>> dealing with bugs for fundamental operations we might see a fix in
>> about a year or two (or four). [dig]
> 
> In this case the bug is in CURL - it is 'known bug #90':
> 
> 
> 
> There's a few threads around about it - this is the most recent I could find:
> 
> https://curl.haxx.se/mail/archive-2019-07/0029.html
> 
> The only real suggestion (although it is vague) there is using some for of 
> pagination (as the UIDs should be eternal, rather than related to the 
> position in the mailbox requested).
> 
> I couldn't find any explicit bug in their bug tracker which is still open for 
> this, but its still listed as a known issue so it sounds like there has been 
> no change in it unforunately :(
> 
> Warmest Regards,
> 
> Mark.
> 
> -- 
> Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
> LiveCode: Everyone can create apps
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Mark Waddingham via use-livecode]

On 2020-03-23 11:39, Pi Digital via use-livecode wrote:

No, same thing unfortunately. Even ALL has the issue. According to an
old bug report that has sat for an age untouched it is a known issue
in the ood curl lib they are using. At the rate they are currently
dealing with bugs for fundamental operations we might see a fix in
about a year or two (or four). [dig]


In this case the bug is in CURL - it is 'known bug #90':



There's a few threads around about it - this is the most recent I could 
find:


https://curl.haxx.se/mail/archive-2019-07/0029.html

The only real suggestion (although it is vague) there is using some for 
of pagination (as the UIDs should be eternal, rather than related to the 
position in the mailbox requested).


I couldn't find any explicit bug in their bug tracker which is still 
open for this, but its still listed as a known issue so it sounds like 
there has been no change in it unforunately :(


Warmest Regards,

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Pi Digital via use-livecode]

No, same thing unfortunately. Even ALL has the issue. According to an old bug 
report that has sat for an age untouched it is a known issue in the ood curl 
lib they are using. At the rate they are currently dealing with bugs for 
fundamental operations we might see a fix in about a year or two (or four). 
[dig]

Sean 


> On 23 Mar 2020, at 11:03, Charles Warwick via use-livecode 
>  wrote:
> 
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Charles Warwick via use-livecode]

Hi Sean,

For your particular issue, does using “SEARCH UNSEEN” instead of “UID SEARCH 
NOT SEEN” provide the results you are looking for?

Regards,

Charles

> On 21 Mar 2020, at 3:10 am, Pi Digital via use-livecode 
>  wrote:
> 
> Thanks Bob
> 
> Yeah, I actually already have something like that. I’d abbreviated my code to 
> keep it simple on the forum. I’m not having trouble connecting. It’s just the 
> shortage of data returned from the server. It’s actually skipping a whole 
> chunk in the middle which is the weirdest part. 
> 
> There is this bug which is yet to be followed up on. 
> 
> https://quality.livecode.com/show_bug.cgi?id=22216
> 
> So I look forward to that in a couple of years maybe. 
> 
> In the meantime I’m hoping I find a solution to this or else it’s yet another 
> reason for clients to disappear saying “you can’t even do so and so”. It gets 
> so tedious having all the workarounds on workarounds. 
> 
> Sean Cole
> Pi Digital
> 
>> On 20 Mar 2020, at 15:10, Bob Sneidar via use-livecode 
>>  wrote:
>> 
>> This version wraps ALL the tsNet calls in a try catch construct:
>> 
>> function IMAPGetHeaders pServer, pUser, pPass
>>  if pServer is empty then
>> ask "Enter IMAP Server IP or address:" as sheet
>> if the result is "Cancel" then return "Canceled."
>> put it into pServer
>>  end if
>> 
>>  if pUser is empty then
>> ask "Enter email address:" as sheet
>> if the result is "Cancel" then return "Canceled."
>> put it into pUser
>>  end if
>> 
>>  if pPass is empty then
>> ask password "Enter password:" as sheet
>> if the result is "Cancel" then return "Canceled."
>> put it into pPass
>>  end if
>> 
>>  put pUser into tSettings["username"]
>>  put pPass into tSettings["password"]
>>  put empty into xHeaders
>> 
>>  try
>> tsnetinit
>> put tsNetCustomSync(pServer, \
>>   "SELECT INBOX",xHeaders,retHeaders,retResult, \
>>   retBytes,tSettings) into tStat
>> put tsNetCustomSync(pServer, \
>>   "UID SEARCH NOT SEEN",xHeaders,retHeaders, \
>>   retResult,retBytes,tSettings) into tStat
>>  catch tError
>> answer error "ERROR: " & tError as sheet
>> breakpoint
>>  end try
>> 
>>  put word 3 to -1 of tStat into tUIDs
>>  put the number of words in tUIDs into sMsgCount
>>  return retHeaders & tStat & sMsgCount
>> end IMAPGetHeaders
>> 
>>> On Mar 20, 2020, at 08:06 , Bob Sneidar via use-livecode 
>>>  wrote:
>>> 
>>> This may be of more use. No functional changes, just something you can use 
>>> to test against different IMAP servers:
>> 
>> 
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription 
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Pi Digital via use-livecode]

Thanks Bob

Yeah, I actually already have something like that. I’d abbreviated my code to 
keep it simple on the forum. I’m not having trouble connecting. It’s just the 
shortage of data returned from the server. It’s actually skipping a whole chunk 
in the middle which is the weirdest part. 

There is this bug which is yet to be followed up on. 

https://quality.livecode.com/show_bug.cgi?id=22216

So I look forward to that in a couple of years maybe. 

In the meantime I’m hoping I find a solution to this or else it’s yet another 
reason for clients to disappear saying “you can’t even do so and so”. It gets 
so tedious having all the workarounds on workarounds. 

Sean Cole
Pi Digital

> On 20 Mar 2020, at 15:10, Bob Sneidar via use-livecode 
>  wrote:
> 
> This version wraps ALL the tsNet calls in a try catch construct:
> 
> function IMAPGetHeaders pServer, pUser, pPass
>   if pServer is empty then
>  ask "Enter IMAP Server IP or address:" as sheet
>  if the result is "Cancel" then return "Canceled."
>  put it into pServer
>   end if
> 
>   if pUser is empty then
>  ask "Enter email address:" as sheet
>  if the result is "Cancel" then return "Canceled."
>  put it into pUser
>   end if
> 
>   if pPass is empty then
>  ask password "Enter password:" as sheet
>  if the result is "Cancel" then return "Canceled."
>  put it into pPass
>   end if
> 
>   put pUser into tSettings["username"]
>   put pPass into tSettings["password"]
>   put empty into xHeaders
> 
>   try
>  tsnetinit
>  put tsNetCustomSync(pServer, \
>"SELECT INBOX",xHeaders,retHeaders,retResult, \
>retBytes,tSettings) into tStat
>  put tsNetCustomSync(pServer, \
>"UID SEARCH NOT SEEN",xHeaders,retHeaders, \
>retResult,retBytes,tSettings) into tStat
>   catch tError
>  answer error "ERROR: " & tError as sheet
>  breakpoint
>   end try
> 
>   put word 3 to -1 of tStat into tUIDs
>   put the number of words in tUIDs into sMsgCount
>   return retHeaders & tStat & sMsgCount
> end IMAPGetHeaders
> 
>> On Mar 20, 2020, at 08:06 , Bob Sneidar via use-livecode 
>>  wrote:
>> 
>> This may be of more use. No functional changes, just something you can use 
>> to test against different IMAP servers:
> 
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Bob Sneidar via use-livecode]

This version wraps ALL the tsNet calls in a try catch construct:

function IMAPGetHeaders pServer, pUser, pPass
   if pServer is empty then
  ask "Enter IMAP Server IP or address:" as sheet
  if the result is "Cancel" then return "Canceled."
  put it into pServer
   end if
   
   if pUser is empty then
  ask "Enter email address:" as sheet
  if the result is "Cancel" then return "Canceled."
  put it into pUser
   end if
   
   if pPass is empty then
  ask password "Enter password:" as sheet
  if the result is "Cancel" then return "Canceled."
  put it into pPass
   end if
   
   put pUser into tSettings["username"]
   put pPass into tSettings["password"]
   put empty into xHeaders
   
   try
  tsnetinit
  put tsNetCustomSync(pServer, \
"SELECT INBOX",xHeaders,retHeaders,retResult, \
retBytes,tSettings) into tStat
  put tsNetCustomSync(pServer, \
"UID SEARCH NOT SEEN",xHeaders,retHeaders, \
retResult,retBytes,tSettings) into tStat
   catch tError
  answer error "ERROR: " & tError as sheet
  breakpoint
   end try
   
   put word 3 to -1 of tStat into tUIDs
   put the number of words in tUIDs into sMsgCount
   return retHeaders & tStat & sMsgCount
end IMAPGetHeaders

> On Mar 20, 2020, at 08:06 , Bob Sneidar via use-livecode 
>  wrote:
> 
> This may be of more use. No functional changes, just something you can use to 
> test against different IMAP servers:


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Bob Sneidar via use-livecode]

This may be of more use. No functional changes, just something you can use to 
test against different IMAP servers:

function IMAPGetHeaders pServer, pUser, pPass
   if pServer is empty then
  ask "Enter IMAP Server IP or address:" as sheet
  if the result is "Cancel" then return "Canceled."
  put it into pServer
   end if
   
   if pUser is empty then
  ask "Enter email address:" as sheet
  if the result is "Cancel" then return "Canceled."
  put it into pUser
   end if
   
   if pPass is empty then
  ask password "Enter password:" as sheet
  if the result is "Cancel" then return "Canceled."
  put it into pPass
   end if
   
   put pUser into tSettings["username"]
   put pPass into tSettings["password"]
   put empty into xHeaders
   
   try
  tsnetinit
   catch tError
  answer error "ERROR: Unable to initialize TSNet" as sheet
  breakpoint
   end try
   
   put tsNetCustomSync(pServer, \
 "SELECT INBOX",xHeaders,retHeaders,retResult, \
 retBytes,tSettings) into tStat
   put tsNetCustomSync(pServer, \
 "UID SEARCH NOT SEEN",xHeaders,retHeaders, \
 retResult,retBytes,tSettings) into tStat
   put word 3 to -1 of tStat into tUIDs
   put the number of words in tUIDs into sMsgCount
   return retHeaders & tStat & sMsgCount
end IMAPGetHeaders

> On Mar 20, 2020, at 07:26 , Pi Digital via use-livecode 
>  wrote:
> 
> Thanks Matthias
> 
> I will give it a go. I tried some of the other settings but not that one as 
> it happens. Does that not upset the fact that you have set the inbox and also 
> download each message body one at a time? That’s why I didn’t try it. 
> 
> In fact, just gave it a go but by the time the second call is requested it 
> fails with the response that ‘No Mailbox Selected’ so logs out again. I tried 
> putting the setting between the two calls. Same issue. 
> 
> There are 1639 messages on the server but only 110 seem to show even with 
> ‘UID SEARCH ALL’. 
> 
> Sean Cole
> 

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: IMAP Search skipping results

[Pi Digital via use-livecode]

Thanks Matthias

I will give it a go. I tried some of the other settings but not that one as it 
happens. Does that not upset the fact that you have set the inbox and also 
download each message body one at a time? That’s why I didn’t try it. 

In fact, just gave it a go but by the time the second call is requested it 
fails with the response that ‘No Mailbox Selected’ so logs out again. I tried 
putting the setting between the two calls. Same issue. 

There are 1639 messages on the server but only 110 seem to show even with ‘UID 
SEARCH ALL’. 

Sean Cole


> On 20 Mar 2020, at 13:00, matthias rebbe via use-livecode 
>  wrote:
> 
> Hi Sean,
> 
> just a guess.
> 
> Did you already try to set  the NO_REUSE  element in the tSettings array to 
> TRUE?
> I am using that setting always, because the default one caused very often 
> problems here.
> 
> 
> -
> Matthias Rebbe
> Life Is Too Short For Boring Code
> 
>> Am 20.03.2020 um 03:16 schrieb Sean Cole (Pi) via use-livecode 
>> :
>> 
>> Hey all,
>> 
>> I'm using TSNet lib to fetch email from IMAP.
>> 
>> The Script:
>>  put "MyName" into tSettings["username"]
>>  put "MyPass" into tSettings["password"]
>>  put empty into xHeaders
>>  tsnetinit
>>  put tsNetCustomSync("imaps://mail.myserver.co.uk", \
>>"SELECT INBOX",xHeaders,retHeaders,retResult, \
>>retBytes,tSettings) into tStat
>>  put tsNetCustomSync("imaps://mail.myserver.co.uk", \
>>"UID SEARCH NOT SEEN",xHeaders,retHeaders, \
>>retResult,retBytes,tSettings) into tStat
>>  put word 3 to -1 of tStat into tUIDs
>>  put the number of words in tUIDs into sMsgCount
>>  put retHeaders & tStat & sMsgCount
>> 
>> My results are so inconsistent and it keeps skipping or omitting details.
>> Heres what I see:
>> 
>> 1ST RUN:::
>> * SEARCH 11646 11647 11648 11649 11650 13018 13019 13020 13021 13022 13023
>> 13024
>> 13025 13026 13027 13028 13029 13030 13031 13032 13033 13034 13035 13036
>> 13037
>> 13038 13039 13040 13041 13042 13043 13044 13045 13046 13047 13048 13049
>> 13050
>> 13051 13052 13053 13054 13055 13056 13057 13058 13059 13060 13061 13062
>> 13063
>> 13064 13065 13066 13067 13068 13069 13070 13071 13072 13073 13074 13075
>> 13076
>> 13077 13078 13079 13080 13081 13082 13083 13084 13085 13086 13087 13088
>> 13089
>> 13090 13091 13092 13093 13094 13095 13096 13097 13098 13099 13100 13101
>> 13102
>> 13103 13104 13105 13106 13107 13108 13109 13110 13111 13112 13113 13114
>> 13115
>> 13116 13117 13118 13119 13120 13121 13122 13123 13124 13125 13126 13127
>> 13128
>> 13129 13130 13131 13132 13133 13134 13135 13136 13137 13138 13139 13140
>> 13141
>> 13142 13143 13144 13145 13146 13147 13148 13149 13150 13151 13152 13153
>> 13154
>> 13155 13156 13157 13158 13159 13160 13161 13162 13163 13164 13165 13166
>> 13167
>> 13168 13169 13170 13171 13172 13173 13174 13175 13176 13177 13178 13179
>> 13180
>> 13181 13182 13183 13184 13185 13186 13187 13188 13189 13190 13191 13192
>> 13193
>> 13194
>> H160 OK Search completed (0.004 + 0.000 + 0.003 secs).
>> 
>> 2ND RUN:::
>> * SEARCH 11646 11647 11648 11649 11650 1
>> H162 OK Search completed (0.003 + 0.000 + 0.002 secs).
>> 
>> 3RD RUN:::
>> * SEARCH 11646 11647 11648 11649 11650 13170 13171 13172 13173 13174 13175
>> 13176
>> 13177 13178 13179 13180 13181 13182 13183 13184 13185 13186 13187 13188
>> 13189
>> 13190 13191 13192 13193 13194
>> H164 OK Search completed (0.003 + 0.000 + 0.002 secs).
>> 
>> 4TH RUN:::
>> * SEARCH 11646 11647 11648 11649 11650 1
>> H166 OK Search completed (0.003 + 0.000 + 0.002 secs).
>> 
>> 5TH RUN:::
>> * SEARCH 11646 11647 11648 11649 11650 1 13057 13058 13059 13060 13061
>> 13062
>> 13063 13064 13065 13066 13067 13068 13069 13070 13071 13072 13073 13074
>> 13075
>> 13076 13077 13078 13079 13080 13081 13082 13083 13084 13085 13086 13087
>> 13088
>> 13089 13090 13091 13092 13093 13094 13095 13096 13097 13098 13099 13100
>> 13101
>> 13102 13103 13104 13105 13106 13107 13108 13109 13110 13111 13112 13113
>> 13114
>> 13115 13116 13117 13118 13119 13120 13121 13122 13123 13124 13125 13126
>> 13127
>> 13128 13129 13130 13131 13132 13133 13134 13135 13136 13137 13138 13139
>> 13140
>> 13141 13142 13143 13144 13145 13146 13147 13148 13149 13150 13151 13152
>> 13153
>> 13154 13155 13156 13157 13158 13159 13160 13161 13162 13163 13164 13165
>> 13166
>> 13167 13168 13169 13170 13171 13172 13173 13174 13175 13176 13177 13178
>> 13179
>> 13180 13181 13182 13183 13184 13185 13186 13187 13188 13189 13190 13191
>> 13192
>> 13193 13194
>> H170 OK Search completed (0.003 + 0.000 + 0.002 secs).
>> 
>> What setting am I missing to make sure it is always consistent? Or is this
>> a bug with TSNet?
>> 
>> I can't think of any workarounds for this one. And it is so key to the
>> whole app.
>> 
>> Sean Cole
>> *Pi Digital*
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription 
>> preferences:

Re: IMAP Search skipping results

[matthias rebbe via use-livecode]

Hi Sean,

just a guess.

Did you already try to set  the NO_REUSE  element in the tSettings array to 
TRUE?
I am using that setting always, because the default one caused very often 
problems here.


-
Matthias Rebbe
Life Is Too Short For Boring Code

> Am 20.03.2020 um 03:16 schrieb Sean Cole (Pi) via use-livecode 
> :
> 
> Hey all,
> 
> I'm using TSNet lib to fetch email from IMAP.
> 
> The Script:
>   put "MyName" into tSettings["username"]
>   put "MyPass" into tSettings["password"]
>   put empty into xHeaders
>   tsnetinit
>   put tsNetCustomSync("imaps://mail.myserver.co.uk", \
> "SELECT INBOX",xHeaders,retHeaders,retResult, \
> retBytes,tSettings) into tStat
>   put tsNetCustomSync("imaps://mail.myserver.co.uk", \
> "UID SEARCH NOT SEEN",xHeaders,retHeaders, \
> retResult,retBytes,tSettings) into tStat
>   put word 3 to -1 of tStat into tUIDs
>   put the number of words in tUIDs into sMsgCount
>   put retHeaders & tStat & sMsgCount
> 
> My results are so inconsistent and it keeps skipping or omitting details.
> Heres what I see:
> 
> 1ST RUN:::
> * SEARCH 11646 11647 11648 11649 11650 13018 13019 13020 13021 13022 13023
> 13024
> 13025 13026 13027 13028 13029 13030 13031 13032 13033 13034 13035 13036
> 13037
> 13038 13039 13040 13041 13042 13043 13044 13045 13046 13047 13048 13049
> 13050
> 13051 13052 13053 13054 13055 13056 13057 13058 13059 13060 13061 13062
> 13063
> 13064 13065 13066 13067 13068 13069 13070 13071 13072 13073 13074 13075
> 13076
> 13077 13078 13079 13080 13081 13082 13083 13084 13085 13086 13087 13088
> 13089
> 13090 13091 13092 13093 13094 13095 13096 13097 13098 13099 13100 13101
> 13102
> 13103 13104 13105 13106 13107 13108 13109 13110 13111 13112 13113 13114
> 13115
> 13116 13117 13118 13119 13120 13121 13122 13123 13124 13125 13126 13127
> 13128
> 13129 13130 13131 13132 13133 13134 13135 13136 13137 13138 13139 13140
> 13141
> 13142 13143 13144 13145 13146 13147 13148 13149 13150 13151 13152 13153
> 13154
> 13155 13156 13157 13158 13159 13160 13161 13162 13163 13164 13165 13166
> 13167
> 13168 13169 13170 13171 13172 13173 13174 13175 13176 13177 13178 13179
> 13180
> 13181 13182 13183 13184 13185 13186 13187 13188 13189 13190 13191 13192
> 13193
> 13194
> H160 OK Search completed (0.004 + 0.000 + 0.003 secs).
> 
> 2ND RUN:::
> * SEARCH 11646 11647 11648 11649 11650 1
> H162 OK Search completed (0.003 + 0.000 + 0.002 secs).
> 
> 3RD RUN:::
> * SEARCH 11646 11647 11648 11649 11650 13170 13171 13172 13173 13174 13175
> 13176
> 13177 13178 13179 13180 13181 13182 13183 13184 13185 13186 13187 13188
> 13189
> 13190 13191 13192 13193 13194
> H164 OK Search completed (0.003 + 0.000 + 0.002 secs).
> 
> 4TH RUN:::
> * SEARCH 11646 11647 11648 11649 11650 1
> H166 OK Search completed (0.003 + 0.000 + 0.002 secs).
> 
> 5TH RUN:::
> * SEARCH 11646 11647 11648 11649 11650 1 13057 13058 13059 13060 13061
> 13062
> 13063 13064 13065 13066 13067 13068 13069 13070 13071 13072 13073 13074
> 13075
> 13076 13077 13078 13079 13080 13081 13082 13083 13084 13085 13086 13087
> 13088
> 13089 13090 13091 13092 13093 13094 13095 13096 13097 13098 13099 13100
> 13101
> 13102 13103 13104 13105 13106 13107 13108 13109 13110 13111 13112 13113
> 13114
> 13115 13116 13117 13118 13119 13120 13121 13122 13123 13124 13125 13126
> 13127
> 13128 13129 13130 13131 13132 13133 13134 13135 13136 13137 13138 13139
> 13140
> 13141 13142 13143 13144 13145 13146 13147 13148 13149 13150 13151 13152
> 13153
> 13154 13155 13156 13157 13158 13159 13160 13161 13162 13163 13164 13165
> 13166
> 13167 13168 13169 13170 13171 13172 13173 13174 13175 13176 13177 13178
> 13179
> 13180 13181 13182 13183 13184 13185 13186 13187 13188 13189 13190 13191
> 13192
> 13193 13194
> H170 OK Search completed (0.003 + 0.000 + 0.002 secs).
> 
> What setting am I missing to make sure it is always consistent? Or is this
> a bug with TSNet?
> 
> I can't think of any workarounds for this one. And it is so key to the
> whole app.
> 
> Sean Cole
> *Pi Digital*
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

IMAP Search skipping results

[Sean Cole (Pi) via use-livecode]

Hey all,

I'm using TSNet lib to fetch email from IMAP.

The Script:
   put "MyName" into tSettings["username"]
   put "MyPass" into tSettings["password"]
   put empty into xHeaders
   tsnetinit
   put tsNetCustomSync("imaps://mail.myserver.co.uk", \
 "SELECT INBOX",xHeaders,retHeaders,retResult, \
 retBytes,tSettings) into tStat
   put tsNetCustomSync("imaps://mail.myserver.co.uk", \
 "UID SEARCH NOT SEEN",xHeaders,retHeaders, \
 retResult,retBytes,tSettings) into tStat
   put word 3 to -1 of tStat into tUIDs
   put the number of words in tUIDs into sMsgCount
   put retHeaders & tStat & sMsgCount

My results are so inconsistent and it keeps skipping or omitting details.
Heres what I see:

1ST RUN:::
* SEARCH 11646 11647 11648 11649 11650 13018 13019 13020 13021 13022 13023
13024
 13025 13026 13027 13028 13029 13030 13031 13032 13033 13034 13035 13036
13037
 13038 13039 13040 13041 13042 13043 13044 13045 13046 13047 13048 13049
13050
 13051 13052 13053 13054 13055 13056 13057 13058 13059 13060 13061 13062
13063
 13064 13065 13066 13067 13068 13069 13070 13071 13072 13073 13074 13075
13076
 13077 13078 13079 13080 13081 13082 13083 13084 13085 13086 13087 13088
13089
 13090 13091 13092 13093 13094 13095 13096 13097 13098 13099 13100 13101
13102
 13103 13104 13105 13106 13107 13108 13109 13110 13111 13112 13113 13114
13115
 13116 13117 13118 13119 13120 13121 13122 13123 13124 13125 13126 13127
13128
 13129 13130 13131 13132 13133 13134 13135 13136 13137 13138 13139 13140
13141
 13142 13143 13144 13145 13146 13147 13148 13149 13150 13151 13152 13153
13154
 13155 13156 13157 13158 13159 13160 13161 13162 13163 13164 13165 13166
13167
 13168 13169 13170 13171 13172 13173 13174 13175 13176 13177 13178 13179
13180
 13181 13182 13183 13184 13185 13186 13187 13188 13189 13190 13191 13192
13193
 13194
H160 OK Search completed (0.004 + 0.000 + 0.003 secs).

2ND RUN:::
* SEARCH 11646 11647 11648 11649 11650 1
H162 OK Search completed (0.003 + 0.000 + 0.002 secs).

3RD RUN:::
* SEARCH 11646 11647 11648 11649 11650 13170 13171 13172 13173 13174 13175
13176
13177 13178 13179 13180 13181 13182 13183 13184 13185 13186 13187 13188
13189
13190 13191 13192 13193 13194
H164 OK Search completed (0.003 + 0.000 + 0.002 secs).

4TH RUN:::
* SEARCH 11646 11647 11648 11649 11650 1
H166 OK Search completed (0.003 + 0.000 + 0.002 secs).

5TH RUN:::
* SEARCH 11646 11647 11648 11649 11650 1 13057 13058 13059 13060 13061
13062
13063 13064 13065 13066 13067 13068 13069 13070 13071 13072 13073 13074
13075
13076 13077 13078 13079 13080 13081 13082 13083 13084 13085 13086 13087
13088
13089 13090 13091 13092 13093 13094 13095 13096 13097 13098 13099 13100
13101
13102 13103 13104 13105 13106 13107 13108 13109 13110 13111 13112 13113
13114
13115 13116 13117 13118 13119 13120 13121 13122 13123 13124 13125 13126
13127
13128 13129 13130 13131 13132 13133 13134 13135 13136 13137 13138 13139
13140
13141 13142 13143 13144 13145 13146 13147 13148 13149 13150 13151 13152
13153
13154 13155 13156 13157 13158 13159 13160 13161 13162 13163 13164 13165
13166
13167 13168 13169 13170 13171 13172 13173 13174 13175 13176 13177 13178
13179
13180 13181 13182 13183 13184 13185 13186 13187 13188 13189 13190 13191
13192
13193 13194
H170 OK Search completed (0.003 + 0.000 + 0.002 secs).

What setting am I missing to make sure it is always consistent? Or is this
a bug with TSNet?

I can't think of any workarounds for this one. And it is so key to the
whole app.

Sean Cole
*Pi Digital*
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode