Re: POP3 Email Processing

[David Weinberg via 4D_Tech]

Hi Alex,

I am adding a function to a legacy system that still need to support v12
customers. We are reading incoming emails via POP3 and IMAP and your code
and Keisuke Miyako's plug in sound very helpful.

Am I able to take up the offer you made to Dougie and obtain your starter
code as parsing out all the headers is looking like a real head ache.

Thanks,
David Weinberg



--
Sent from: http://4d.1045681.n5.nabble.com/4D-Tech-f1376241.html
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: POP3 Email Processing (and ramblings by Tim Nevels on 4D)

[Roger Reed – Illus House via 4D_Tech]

Tim’s note is reminding me that this year is the THIRTIETH I’ve been using 4D. 
Kinda stunning.

Roger

> On Oct 28, 2017, at 00:21, Tim Nevels via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> On Oct 27, 2017, at 8:11 PM, David Adams wrote:
> 
>> Plug-ins can yield time (in two flavors) to the scheduler, so they don't
>> *have* to be completely blocking. The original PDM code was not re-entrant
>> and that's legacy seems to have been haunting us for the 25 years since 4D
>> took it on.
> 
> Rumor has it that 4D is building a replacement for the 4D Internet Command 
> plugin that will be implemented as native 4D commands. It will be like the 
> new 4D Write Pro implementation. Not a plugin. (Heard it at a 4D Summit from 
> 4D people.) Unknown if it will be thread-safe, but I would guess it will be. 
> For sure it will not be blocking like the current plugin. 
> 
> They only patch the current plugin so that we can “get by” until this new 
> work is done. They know the current plugin is not a long term solution. But 
> to implement a good, long term solution takes time. 
> 
> Why do you think they bought the PDM plugin many years ago and did not just 
> write their own or build it from scratch inside 4D? Because it is not easy. 
> It is hard work to implement all the capabilities in that plugin and do a 
> great job. 
> 
> Remember that 4D builds things that will last for decades. Think about that. 
> We are using work that they released decades ago. So any new work needs to be 
> done with the same care and forward thinking. It has to last for decades 
> without the need for constant maintenance or rewrites. That has got to be 
> hard to accomplish. 
> 
> 4D, Inc. does not have an unlimited budget. There are only so many engineers 
> that can get work done. Decisions have to be made. Priorities have to be set. 
> We may not all agree with their decisions or priorities, but we can 
> understand them. 
> 
> And I forgot to mention that all these decisions have to result in a profit 
> year after year after year. So far that has happened. For decades. They are 
> still here and going strong. 
> 
> None of us are C++ programmers that have built a code base that runs on macOS 
> and Windows — which is a constant moving target — and that must have backward 
> compatibility with 4D code we have written decades ago. An interpreter for a 
> language and a compiler that makes binaries for both operating systems. It 
> has got to be a very hard task. Much harder than the kind of work we do with 
> the results of all that work. Just to give it some perspective. 
> 
> We have a great platform here folks. Of course it can always get better. And 
> I’m sure it will. Nobody can have everything immediately. It all takes time 
> and money. 
> 
> I started making my living using 4D in 1990.  27 years with this great 
> product. Literally millions of dollars in billable hours over all these 
> years. I’m glad I got on board this ship and it’s really been a pretty good 
> ride when I look back. 
> 
> I’m looking forward to 4D Summit in 2018 and all the new stuff they will 
> announce. Looking forward to another 20 years. 
> 
> Thanks to Laurent Ribardière for making this all possible. I think we all owe 
> him, at least I certainly do. 
> 
> Here is something fun. Google “4D creator”. See what the first item is. :)
> 
> Tim
> 
> Sent from my iPad
> 
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: POP3 Email Processing (and ramblings by Tim Nevels on 4D)

[David Adams via 4D_Tech]

On Sat, Oct 28, 2017 at 3:21 PM, Tim Nevels via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> On Oct 27, 2017, at 8:11 PM, David Adams wrote:
>
>
Tim, I wasn't really making a broad comment of any kind about 4D or
criticizing - just citing a bit of the history I was there for. The
original programmer was Phil Weiss, nice guy. But I do remember how shocked
4D was when they got the code and discovered that it was not re-entrant.
Then there was ITK, which was really kind of great (again, a single
programmer  - Christian Quest?- the same person that did some of the
network layer pieces back when TCP/IP was just an option). But that
disappeared...or sort of withered on the vine. And then NTK, which I like a
lot.

All three of these tools were written by individual programmers and they
didn't spend years on them. Rob could say something about how long the
4DIC-like features of NTK took to write, not that I expect him to. (No
reason he should.). And he's got a cross-platform C++ framework as well, I
believe. Paul Carnine wrote a multi-platform framewor too - but with
support for even more platforms - for his award-winning video software.
(That's where he went, if I remember correctly, it had nothing to do with
4D.)

If 4D comes up with a native replacement 4DIC, great! For example, I'm
really happy that they've got HTTP Get and HTTP Request in the language,
very helpful.

But waiting for a future feature that hasn't been announced, shown, put
into a beta or released? Something that might or might not happen, and who
knows when? And who knows how it will work or what it will do when/if it
arrives? That's not something useful to me today or something that I can
plan around. If I need a feature, I already need it. So I have to work with
what we've got. Once there's a native and non-blocking way to do email in
4D, great! If they've got it and I need it, I'll check it out.

Until then, there are stacks of could-based mail services with nice APIs.
That's where I'd be looking, particularly since HTTP Get and HTTP Request
seem to do a decent job with no real downsides.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: POP3 Email Processing (and ramblings by Tim Nevels on 4D)

[Tim Nevels via 4D_Tech]

On Oct 27, 2017, at 8:11 PM, David Adams wrote:

> Plug-ins can yield time (in two flavors) to the scheduler, so they don't
> *have* to be completely blocking. The original PDM code was not re-entrant
> and that's legacy seems to have been haunting us for the 25 years since 4D
> took it on.

Rumor has it that 4D is building a replacement for the 4D Internet Command 
plugin that will be implemented as native 4D commands. It will be like the new 
4D Write Pro implementation. Not a plugin. (Heard it at a 4D Summit from 4D 
people.) Unknown if it will be thread-safe, but I would guess it will be. For 
sure it will not be blocking like the current plugin. 

They only patch the current plugin so that we can “get by” until this new work 
is done. They know the current plugin is not a long term solution. But to 
implement a good, long term solution takes time. 

Why do you think they bought the PDM plugin many years ago and did not just 
write their own or build it from scratch inside 4D? Because it is not easy. It 
is hard work to implement all the capabilities in that plugin and do a great 
job. 

Remember that 4D builds things that will last for decades. Think about that. We 
are using work that they released decades ago. So any new work needs to be done 
with the same care and forward thinking. It has to last for decades without the 
need for constant maintenance or rewrites. That has got to be hard to 
accomplish. 

4D, Inc. does not have an unlimited budget. There are only so many engineers 
that can get work done. Decisions have to be made. Priorities have to be set. 
We may not all agree with their decisions or priorities, but we can understand 
them. 

And I forgot to mention that all these decisions have to result in a profit 
year after year after year. So far that has happened. For decades. They are 
still here and going strong. 

None of us are C++ programmers that have built a code base that runs on macOS 
and Windows — which is a constant moving target — and that must have backward 
compatibility with 4D code we have written decades ago. An interpreter for a 
language and a compiler that makes binaries for both operating systems. It has 
got to be a very hard task. Much harder than the kind of work we do with the 
results of all that work. Just to give it some perspective. 

We have a great platform here folks. Of course it can always get better. And 
I’m sure it will. Nobody can have everything immediately. It all takes time and 
money. 

I started making my living using 4D in 1990.  27 years with this great product. 
Literally millions of dollars in billable hours over all these years. I’m glad 
I got on board this ship and it’s really been a pretty good ride when I look 
back. 

I’m looking forward to 4D Summit in 2018 and all the new stuff they will 
announce. Looking forward to another 20 years. 

Thanks to Laurent Ribardière for making this all possible. I think we all owe 
him, at least I certainly do. 

Here is something fun. Google “4D creator”. See what the first item is. :)

Tim

Sent from my iPad

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: POP3 Email Processing

[David Adams via 4D_Tech]

> It blocks the whole application while waiting for and downloading
messages from the server. This is because the plugin interface of 4D is
designed
> to be in cooperative mode. Even switching to v16 and using preemptive
threads will not help, the plugin interface isn't thread-safe.

Plug-ins can yield time (in two flavors) to the scheduler, so they don't
*have* to be completely blocking. The original PDM code was not re-entrant
and that's legacy seems to have been haunting us for the 25 years since 4D
took it on.

Plug-ins are not thread-safe in 4D, period. Will that change? *Can* that
even change? That's above my pay grade...not a clue. Maybe Rob, Miyako or
someone from 4D could comment. So far, I've heard nothing on this important
question.

What Kirk says. He's using a watch folder system which, now that I think
about it, is a fantastic idea in his setup.

You can use HTTP Get/Request in a non-blocking way, so that's *always* a
solid possibility for external and internal communications.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: POP3 Email Processing

[Kirk Brooks via 4D_Tech]

On Fri, Oct 27, 2017 at 7:05 AM, Epperlein, Lutz (agendo) via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> The main problem with 4D Internet Commands is:
> It blocks the whole application while waiting for and downloading messages
> from the server.


​I moved downloading emails into a separate 4D database for this reason.
It's not as clunky a solution as you might think. The app doesn't require
any users to be logged in. I run it as a separate 4D server instance and it
simply wakes up periodically, checks some email accounts it monitors,
downloads any messages, deals with attachments and writes the data to a set
of watched folders which the parent database checks.

In our case we receive a lot of image files via email which was a real
performance hit​ for all the reasons you list. Separating it out made even
more sense when I began storing all those images on AWS and there was no
need to have them in the parent db at all.

I'm starting to think it might be worthwhile to add the email sending tasks
as well. The parent db drops out-going emails into a watched folder, the
email db sends them.

-- 
Kirk Brooks
San Francisco, CA
===

*The only thing necessary for the triumph of evil is for good men to do
nothing.*

*- Edmund Burke*
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

RE: POP3 Email Processing

[Epperlein, Lutz (agendo) via 4D_Tech]

The main problem with 4D Internet Commands is: 
It blocks the whole application while waiting for and downloading messages from 
the server. This is because the plugin interface of 4D is designed to be in 
cooperative mode. Even switching to v16 and using preemptive threads will not 
help, the plugin interface isn't thread-safe.

So if you want to receive e-mails the best thing is to use a third party tool 
and develop a non-blocking interface to this tool. Developing a plugin will not 
help.

We are challenged with the same problem, so we decided to get the e-mails using 
a job in the night, so blocking of the server doesn't disturb anyone. For the 
requirements of our customers this is ok.

HTH
Lutz Epperlein


**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: POP3 Email Processing

[Herr Alexander Heintz via 4D_Tech]

Yep, parsing complex mails is a pain in the you know where…

Keisuke helped somewhat with his GMime plugin:

https://github.com/miyako/4d-plugin-gmime 


I intended to write a component making it easier to use, but did not get around 
to it yet.
Either you wait for me to get around to it, or PM me ( awheint(at)gmail.com 
 ) and i will drop you a bunch of code that you can use as 
starting point.

Cheers

Alex

> Am 27.10.2017 um 15:35 schrieb Douglas Cryer via 4D_Tech 
> <4d_tech@lists.4d.com>:
> 
> Hi folks,
> 
> After many years using OTRS for our help ticketing and change control we have 
> recently decided to roll our own 4D based solution.  This is partly a cost 
> thing; partly a support/functionality thing and partly to get our teeth into 
> a project where we can get used to the latest version of 4D while we wait for 
> v17 R3 :-).
> 
> The core of a system like this is reading in Email messages.  I have done 
> simple parsing of emails in the past (long past) but looking at it now I was 
> surprised at how little the internet commands actually do for you before you 
> have to get down and dirty with loads of parsing.
> 
> Essentially we need to create a HTML file with the email body and inline 
> images.  Plus all the attachments; recipients; subject; dates and other info.
> 
> We are looking at using a 3rd party product called "EAGetMail POP3/IMAP4 
> Component” - https://www.emailarchitect.net/eagetmail 
>  to do all the heavy lifting.  
> Before I go ahead and purchase this is does anyone else have any other ideas 
> or options?
> 
> Regards,  Dougie
> 
> 
> telekinetix Limited- J. Douglas Cryer
> Phone : 01234 761759  Mobile : 07973 675 218
> 2nd Floor Broadway House, 4-6 The Broadway, Bedford MK40 2TE
> Email : jdcr...@telekinetix.com   Web : 
> http://www.telekinetix.com 
> 
> 
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**