Jim,
Your observations and questions are helpful.
re: Installing the Mac OS Server App.
I hadn't considered that but if it is the only way to tame the power
management problems, they might have to try that. There was a time when we
ran this system on Mac OS X Server, but when that OS option was dropped, I
saw no value in running Apple's Server App. Even under OS X Server, I
always ran Apache manually. I found that the Mac OS X Server's UI for
managing web sites would often overwrite our custom configurations, e.g.,
rewrite rules. I wasn't clear earlier but the 4D web server (Active4D)
sits behind Apache and serves on 127.0.0.1 on a high port. All requests go
through Apache.
re: Is Apache still serving when 4D Remote Stops working?
I believe it is, but haven't actually gotten up and checked at 3am. If I
see that we're down in the morning. I'll check their static site first.
re: You¹ve only mentioned being notified by the scripts on the machine
itself. Does the external service monitor also notify you that the 4D web
server has stopped serving?
Yes, The external monitor would tell me if the web server was down, or
if
the machine isn't reachable (ping). We've seen some of those, but not as
many as the on-machine script notifications. However, I have the external
monitor to stop sending notifications after a few failures. Actually,
given that the machine is sending mail every 3 minutes from 3:01 am until
I restart 4D it isn't totally asleep.
re: Which part of #3 is failing? Is the script to kill 4D being run and
failing? Is it running ³kill -9 process_id²? Is the auto-launch failing?
Is the heartbeat file writing stopping while the service is actually
running? There are numerous reports of problems with file writing and
delayed processes. If only the heartbeat file writing is stopping, does
the file actually exist and is still writable by that process? Where is
the file located?
I think that this is a big part of the problem. Granted this has run
reliably on past OS versions but I need to thoroughly retest this script.
- It runs "killall 4D", but could be modified to kill the process. I just
verified that "killall 4D" worked at this time. After doing so, the
auto-launch kicked in and restarted 4D.
- I suspect that the problem might be with the heartbeat file. I'll have
to check if the file is being deleted. We have code in 4D that notifies us
if the heartbeat log can't be created or is not writable when the
process starts. If that occurs, the client doesn't enable 4D's web server
and it is quickly evident that we have a problem. I have seen no heartbeat
file problem notifications since the OS upgrade, except on 1st 4D launch.
I had to create and set permissions on /var/log/local/4D/, which is where
the heartbeat file is located.
re: Does the external service monitor have an API? If so, could you
periodically check the external service monitor for your service, and
bounce 4D when there is actually a problem with the service?
I don't believe it does, but if it did I couldn't leverage it. The
monitor unfortunately runs on a machine on a different network that is
behind the institution's firewall. The server in question is on the 'open'
network. The firewalled app can see 'out' but can't be seen from the
server. Moving the monitor isn't really an option. Getting the firewall
open isn't even a consideration.
Beyond all of the above, I would like to find out why this problem
repeatedly occurs at 3am. If you've followed my earlier posts I do have a
launchd job that runs at that time, but rescheduling it earlier did not
move the problem to the earlier time. I even turned that job off for one
night and that didn't prevent the 3am sleep. I grep'd all of my logs and
could not find anything notable happening between 2:57 and 3:01.
Thanks,
Brad Perkins
>
>------------------------------
>
>Message: 5
>Date: Sun, 5 Nov 2017 12:07:43 -0500
>From: Jim Crate <j...@quevivadev.com>
>To: 4D iNug Technical <4d_tech@lists.4d.com>
>Subject: Re: 4D Remote hanging at 3am after upgrading to Hi Sierra.
> Need logging advice.
>Message-ID: <0f3680b7-1b6c-4e79-b052-da4139867...@quevivadev.com>
>Content-Type: text/plain; charset=utf-8
>
>On Nov 4, 2017, at 3:32 PM, Perkins, Bradley D via 4D_Tech
><4d_tech@lists.4d.com> wrote:
>
>> I believe I've done everything I can using the pmset command to prevent
>>sleep, standby etc. It isn't clear if you can totally turn power
>>management off everywhere though. In researching this problem I also
>>discovered the 'caffeinate' command. This is a command-line system
>>utility that is supposed to prevent the computer from sleeping. I
>>haven't tried it yet. Has anyone here used it? If so, can you describe
>>how?
>
>I don¹t know if this is possible in your situation, but for the Mac Mini
>running 4D Server on 10.12 for my client I installed Apple¹s Server app
>after having a couple issues suspected to be related to the power
>management. We had already purchased Server for other machines. If you go
>this route, you will probably have to configure your Apache web serving
>through the Server app. If you leave the FTP services turned off it
>shouldn¹t interfere with whatever you¹re using to run the SFTP server.
>
>> I first deployed this system in 1995 under 3.x and it has provided
>>continuous near 24/7/365 service since then. It has a small, but global
>>user base which is primarily web-based. The machine in question is the
>>web 'client', but it also runs a Apache for static web content and an
>>SFTP server. This system has remained largely unchanged
>>functionality-wise in the last 10 years. The main structure changes have
>>been those required to upgrade major 4D versions. I maintain this
>>system, but haven't implemented many newer 4D features. I've tried to
>>stay aware of how 4D has changed.
>
>Is Apache still serving static content when 4D Remote stops working?
>
>> 2. Continuously monitor the 4D web server. Every few minutes an
>>off-site monitoring agent requests a simple dynamically generated page.
>>If the request times out, or if the response contains unexpected
>>content, I get notified. These requests provide an added benefit of
>>keeping 4D busy during light or no usage periods.
>
>You¹ve only mentioned being notified by the scripts on the machine
>itself. Does the external service monitor also notify you that the 4D web
>server has stopped serving?
>
>> 3. There is not an easy way to externally determine if 4D, or any Mac
>>OS desktop app has hung. You can easily detect if 4D is running, but not
>>when it is non-responsive. For us this is when 4D has encountered a
>>low-level error that can't be gracefully handled with an error handler.
>>It doesn't help that any displayed dialogs provide little useful
>>information. One solution is to have 4D run a process that writes a
>>'heartbeat' file every minute and have a system process check for
>>changes to this file. If the file is present and hasn't changed in more
>>than a few minutes, we'll assume that 4D has hung and run a script to
>>kill it. At this point the auto-launcher from (1) above takes over. This
>>is where we seem to be failing currently.
>>
>> If others have done (3) how do you implement your heartbeat? I write a
>>current timestamp into the file every minute. That assumes the file
>>exists and is writable. It usually is. Is there a better way that is
>>more robust?
>
>Which part of #3 is failing? Is the script to kill 4D being run and
>failing? Is it running ³kill -9 process_id²? Is the auto-launch failing?
>Is the heartbeat file writing stopping while the service is actually
>running? There are numerous reports of problems with file writing and
>delayed processes. If only the heartbeat file writing is stopping, does
>the file actually exist and is still writable by that process? Where is
>the file located?
>
>Does the external service monitor have an API? If so, could you
>periodically check the external service monitor for your service, and
>bounce 4D when there is actually a problem with the service?
>
>
>Jim Crate
>
>
>
>------------------------------
>
>Subject: Digest Footer
>
>**********************************************************************
>4D Internet Users Group (4D iNUG)
>FAQ: http://lists.4d.com/faqnug.html
>Archive: http://lists.4d.com/archives.html
>Options: http://lists.4d.com/mailman/options/4d_tech
>Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
>**********************************************************************
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
**********************************************************************
