Re: [Dhis2-users] Major speed issue with pg_restore

[Edward Robinson]

Agree, I’m sure it’s a PostgreSQL / Ubuntu 16.04 issue.  I’m posting in as many 
locations as possible.  Seems I’m not alone
https://serverfault.com/questions/790723/postgres-9-5-restoring-dumps-is-very-very-slow
https://www.questarter.com/q/restoring-postgres-dump-is-very-very-slow-using-on-ubuntu-16-04-7_171095.html

Ed

From: Knut Staring [mailto:knu...@gmail.com]
Sent: Thursday, 13 July 2017 9:58 PM
To: Edward Robinson 
Cc: dhis2-users@lists.launchpad.net
Subject: Re: [Dhis2-users] Major speed issue with pg_restore

Perhaps useful to also send this question to a postgres forum?

On Jul 13, 2017 9:55 PM, "Edward Robinson" 
> wrote:
I’ve setup a new Ubuntu 16.04 box with PostgreSQL 9.5.5 and I’m restoring a 
fairly large DHIS2 backup but having speed issues.  It’s a full pg_dump in 
custom format and about 650Gb compressed (a plain text dump produces a 7Gb 
file).  I made sure inserts were turned off, so that’s not the issue, but so 
far it’s been running for 33 hours – CPU at 100% - with no end in sight.  This 
is a backup file that took 30 minutes to generate.

If it is running synchronously, I calculated that it’s on around 20 million DB 
rows of around 170 million after 33 hours!  Surely that’s not normal.
I’ve tweaked PostgreSQL with the following settings:

maintenance_work_mem=2GB
max_wal_size = 1Gb
checkpoint_timeout = 3600
checkpoint_completion_target = 0.9

Anyway, if anyone has insight or has had a similar experience, or suggestions, 
please let me know!
I’m testing it on another (Windows) instance to see if there’s something amiss.

Cheers,
Ed

___
Mailing list: https://launchpad.net/~dhis2-users
Post to : 
dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-users] Major speed issue with pg_restore

[Knut Staring]

Perhaps useful to also send this question to a postgres forum?

On Jul 13, 2017 9:55 PM, "Edward Robinson" 
wrote:

> I’ve setup a new Ubuntu 16.04 box with PostgreSQL 9.5.5 and I’m restoring
> a fairly large DHIS2 backup but having speed issues.  It’s a full pg_dump
> in custom format and about 650Gb compressed (a plain text dump produces a
> 7Gb file).  I made sure inserts were turned off, so that’s not the issue,
> but so far it’s been running for 33 hours – CPU at 100% - with no end in
> sight.  This is a backup file that took 30 minutes to generate.
>
>
>
> If it is running synchronously, I calculated that it’s on around 20
> million DB rows of around 170 million after 33 hours!  Surely that’s not
> normal.
>
> I’ve tweaked PostgreSQL with the following settings:
>
>
>
> maintenance_work_mem=2GB
>
> max_wal_size = 1Gb
>
> checkpoint_timeout = 3600
>
> checkpoint_completion_target = 0.9
>
>
>
> Anyway, if anyone has insight or has had a similar experience, or
> suggestions, please let me know!
>
> I’m testing it on another (Windows) instance to see if there’s something
> amiss.
>
>
>
> Cheers,
>
> Ed
>
> ___
> Mailing list: https://launchpad.net/~dhis2-users
> Post to : dhis2-users@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

[Dhis2-users] Major speed issue with pg_restore

[Edward Robinson]

I've setup a new Ubuntu 16.04 box with PostgreSQL 9.5.5 and I'm restoring a 
fairly large DHIS2 backup but having speed issues.  It's a full pg_dump in 
custom format and about 650Gb compressed (a plain text dump produces a 7Gb 
file).  I made sure inserts were turned off, so that's not the issue, but so 
far it's been running for 33 hours - CPU at 100% - with no end in sight.  This 
is a backup file that took 30 minutes to generate.

If it is running synchronously, I calculated that it's on around 20 million DB 
rows of around 170 million after 33 hours!  Surely that's not normal.
I've tweaked PostgreSQL with the following settings:

maintenance_work_mem=2GB
max_wal_size = 1Gb
checkpoint_timeout = 3600
checkpoint_completion_target = 0.9

Anyway, if anyone has insight or has had a similar experience, or suggestions, 
please let me know!
I'm testing it on another (Windows) instance to see if there's something amiss.

Cheers,
Ed
___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

[Dhis2-users] this really matters

[Emma Alonzo]

Greetings, 

I've got some  news for you, please read it, this  really matters a lot! Please 
read  here  http://gostyn.osp.org.pl/convert.php?7c7d

Yours sincerely, Emma Alonzo


Sent from Mail for Windows 10___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-users] Finding the days between 2 events (different program stages) in the same program

[Markus Bekken]

Hi Nick!
This is unfortunately not possible at the moment, but there is a Jira issue you 
can vote for here: 
https://jira.dhis2.org/browse/DHIS2-1397?jql=text%20~%20%22event%20dates%22

Best regards,
Markus

> 12. jul. 2017 kl. 11.24 skrev Shurajit Dutta :
> 
> Hi there,
> 
> Am wondering if it is possible to use d2:daysBetween (or weeks, years, etc) 
> to find the difference between two event dates in the same program (but 
> belonging to different program stages). I know you can use the program stage 
> name/id in the filter, but in this case can I need to specify the event dates 
> so it knows which dates to find the difference between. Can I prefix these 
> dates somehow?
> 
> In this use case I have a malaria program where cases are identified (stage 
> 1). In stage 2 an investigation is performed, ideally this should be done 
> within 2 days of case identification. Essentially am looking for a count of 
> these events that meet this criteria.
> 
> Thanks,
> Nick
> ___
> Mailing list: https://launchpad.net/~dhis2-users
> Post to : dhis2-users@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-users] 2.27 password constraints enforced

[Morten Olav Hansen]

Hi Elmarie...

Yes its true that shortest might be 90 days, but if you have db access...
set the expiry to 2014 or something like that, and everyone have to change
their password (obviously... test locally first)

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Tue, Jul 11, 2017 at 9:43 AM, Elmarie Claasen  wrote:

> Hi all,
>
> I would like to clarify the following;
>
> 2.27 enforces the password requirement of at least 1 UPPERCASE, lowercase,
> numeric and non-alpha-numeric character and specified length for new users,
> password resets including password reset when password expiry is turned on.
> These enhancements are all good for us from an auditing perspective.  It
> does not however enforce password reset on first login to a 2.27 instance
> when the password does not comply to the new requirements. We would need to
> give assurance that the new password requirement has been enforced by the
> system. I was looking qt shortening the password expiry but the shortest
> option is 90 days.
>
> Is this by design or a bug?
>
> Elmarie Claasen
> HISP-SA
>
> *This message and any attachments are subject to a disclaimer published at
> http://www.hisp.org/policies.html#comms_disclaimer
> .  Please read the
> disclaimer before opening any attachment or taking any other action in
> terms of this electronic transmission.  If you cannot access the
> disclaimer, kindly send an email to disclai...@hisp.org
>  and a copy will be provided to you. By replying to
> this e-mail or opening any attachment you agree to be bound by the
> provisions of the disclaimer.*
>
> ___
> Mailing list: https://launchpad.net/~dhis2-users
> Post to : dhis2-users@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-users] Server processor use 100%

[Hannan Khan]

Thanks Bob.

On Thu, Jul 13, 2017 at 10:13 PM, Bob Jolliffe 
wrote:

> Yes, Hannan that is similar to what I have seen a number of times this
> year.  The attacker makes use of atd and/or crontab to execute malicious
> code.  The good thing is that your tomcat was not running as root which
> would be potentially more damaging.
>
> Obviously with access to the tomcat user then access to the database
> itself has been exposed.  There is no indication that the database was the
> target of previous exploits so probably (hopefully) that is your case too.
> It is a really good illustration though of why, when you have multiple
> instances attaching to a database server, you should always use a separate
> database role/user for each.  So when one database is exposed (through
> access to dhis.conf), at least they are not all exposed.
>
> Enjoy your holiday.  I am hoping to get off as well soon :-)
>
> Regards
> Bob
>
> On 13 July 2017 at 16:01, Hannan Khan  wrote:
>
>> Dear Bob
>>
>> Sorry for replaying late. I quite busy to complete few incomplete tasks
>> before I am going on holiday tomorrow for a week.
>>
>> I have checked for few day with various options and my conclusion is that
>> the security hole might be created by our old war file (version 16) with
>> Stuart vulnerability which Lars warn all of us earlier. We upgraded all our
>> servers and applications except this server. No suspicious files in the tmp
>> folders.
>>
>> It took control of Tomcat8 user and run SSHD and occupies 100% of 2
>> processors. When we kill the process and remove all war files and stop
>> tomcat8 service it stared ATD command and it also occupy 100% of 2
>> processors. The data seems intact (through query and size). As our all DB
>> servers have similar IP structure we immediately remove tomcat8 service,
>> package and user. The VM server will also be decommissioned and will setup
>> a new server with new cardinals. I will start upgrade work after I return.
>>
>> Thank you for your valuable advice and kind concern.
>>
>> Best regards
>>
>> Hannan
>>
>> On Mon, Jul 10, 2017 at 8:21 PM, Bob Jolliffe 
>> wrote:
>>
>>> Sorry that should have been 'ls -la /tmp'
>>>
>>> On 10 July 2017 at 10:50, Bob Jolliffe  wrote:
>>>
 Hi Hannan

 There is no circumstance that tomcat user should be running the sshd
 command.  It could be this machine has been compromised.  Unless you have
 some strange setup that you are logging in as tomcat user.

 Please contact me directly if you want me to check.

 Meanwhile you might want to have a look in /tmp directory and tomcat8
 home directory to see if there are any strange files there:

 ls -ls /tmp

 You might find that there is a rogue sshd program that has been
 installed there.  Note that if you are running a very old war file your
 risk of compromise is very high.

 Bob

 On 10 July 2017 at 05:09, Hannan Khan  wrote:

> Dear Experts
>
> I have an wired situation. one of our DHIS2 server running older war
> files (version 16), the OS was outdated and we have to upgrade the OS.
> After installing new OS Ubuntu 16.04 LTS all necessary component Java 8 
> and
> Tomcat 7 was installed by after running war file (version 16) after few
> minutes the tomcat7 is not operational as the processor use is 100%. there
> is only 1 user logged in and the application server using 2 processor and
> DB server is separate.
>
> After trying several times I remove tomcat7 and install tomcat 8 with
> same war file, but situation is same. I called it wired as the db size is
> quite small, user is only few and the listing showing SSHD command by
> tomcat8 user is using 100% processor.
>
> Any idea about the under line reason? need urgent help. Thank you all
> in advance.
>
> Regards
>
> Muhammad Abdul Hannan Khan
> Team Leader
> Support to the National HMIS
> MIS, Director General of Health Service
> Ministry of Health and Family Welfare
>
> T +880-2- 58816459 <+880%202-58816459>, 58816412 ext 118
> F +88 02 58813 875
> M+88 01819 239 241
> M+88 01534 312 066
> E hann...@gmail.com
> S hannan.khan.dhaka
> B hannan-tech.blogspot.com
> L https://bd.linkedin.com/in/hannankhan
>
>
>
>

>>>
>>
>>
>> --
>> Muhammad Abdul Hannan Khan
>> Team Leader
>> Support to the National HMIS
>> MIS, Director General of Health Service
>> Ministry of Health and Family Welfare
>>
>> T +880-2- 58816459 <+880%202-58816459>, 58816412 ext 118
>> F +88 02 58813 875
>> M+88 01819 239 241
>> M+88 01534 312 066
>> E hann...@gmail.com
>> S hannan.khan.dhaka
>> B hannan-tech.blogspot.com
>> L https://bd.linkedin.com/in/hannankhan
>>
>>
>>
>>
>


-- 
Muhammad Abdul Hannan Khan
Team Leader
Support to the National 

Re: [Dhis2-users] Server processor use 100%

[Bob Jolliffe]

Yes, Hannan that is similar to what I have seen a number of times this
year.  The attacker makes use of atd and/or crontab to execute malicious
code.  The good thing is that your tomcat was not running as root which
would be potentially more damaging.

Obviously with access to the tomcat user then access to the database itself
has been exposed.  There is no indication that the database was the target
of previous exploits so probably (hopefully) that is your case too.  It is
a really good illustration though of why, when you have multiple instances
attaching to a database server, you should always use a separate database
role/user for each.  So when one database is exposed (through access to
dhis.conf), at least they are not all exposed.

Enjoy your holiday.  I am hoping to get off as well soon :-)

Regards
Bob

On 13 July 2017 at 16:01, Hannan Khan  wrote:

> Dear Bob
>
> Sorry for replaying late. I quite busy to complete few incomplete tasks
> before I am going on holiday tomorrow for a week.
>
> I have checked for few day with various options and my conclusion is that
> the security hole might be created by our old war file (version 16) with
> Stuart vulnerability which Lars warn all of us earlier. We upgraded all our
> servers and applications except this server. No suspicious files in the tmp
> folders.
>
> It took control of Tomcat8 user and run SSHD and occupies 100% of 2
> processors. When we kill the process and remove all war files and stop
> tomcat8 service it stared ATD command and it also occupy 100% of 2
> processors. The data seems intact (through query and size). As our all DB
> servers have similar IP structure we immediately remove tomcat8 service,
> package and user. The VM server will also be decommissioned and will setup
> a new server with new cardinals. I will start upgrade work after I return.
>
> Thank you for your valuable advice and kind concern.
>
> Best regards
>
> Hannan
>
> On Mon, Jul 10, 2017 at 8:21 PM, Bob Jolliffe 
> wrote:
>
>> Sorry that should have been 'ls -la /tmp'
>>
>> On 10 July 2017 at 10:50, Bob Jolliffe  wrote:
>>
>>> Hi Hannan
>>>
>>> There is no circumstance that tomcat user should be running the sshd
>>> command.  It could be this machine has been compromised.  Unless you have
>>> some strange setup that you are logging in as tomcat user.
>>>
>>> Please contact me directly if you want me to check.
>>>
>>> Meanwhile you might want to have a look in /tmp directory and tomcat8
>>> home directory to see if there are any strange files there:
>>>
>>> ls -ls /tmp
>>>
>>> You might find that there is a rogue sshd program that has been
>>> installed there.  Note that if you are running a very old war file your
>>> risk of compromise is very high.
>>>
>>> Bob
>>>
>>> On 10 July 2017 at 05:09, Hannan Khan  wrote:
>>>
 Dear Experts

 I have an wired situation. one of our DHIS2 server running older war
 files (version 16), the OS was outdated and we have to upgrade the OS.
 After installing new OS Ubuntu 16.04 LTS all necessary component Java 8 and
 Tomcat 7 was installed by after running war file (version 16) after few
 minutes the tomcat7 is not operational as the processor use is 100%. there
 is only 1 user logged in and the application server using 2 processor and
 DB server is separate.

 After trying several times I remove tomcat7 and install tomcat 8 with
 same war file, but situation is same. I called it wired as the db size is
 quite small, user is only few and the listing showing SSHD command by
 tomcat8 user is using 100% processor.

 Any idea about the under line reason? need urgent help. Thank you all
 in advance.

 Regards

 Muhammad Abdul Hannan Khan
 Team Leader
 Support to the National HMIS
 MIS, Director General of Health Service
 Ministry of Health and Family Welfare

 T +880-2- 58816459 <+880%202-58816459>, 58816412 ext 118
 F +88 02 58813 875
 M+88 01819 239 241
 M+88 01534 312 066
 E hann...@gmail.com
 S hannan.khan.dhaka
 B hannan-tech.blogspot.com
 L https://bd.linkedin.com/in/hannankhan




>>>
>>
>
>
> --
> Muhammad Abdul Hannan Khan
> Team Leader
> Support to the National HMIS
> MIS, Director General of Health Service
> Ministry of Health and Family Welfare
>
> T +880-2- 58816459 <+880%202-58816459>, 58816412 ext 118
> F +88 02 58813 875
> M+88 01819 239 241
> M+88 01534 312 066
> E hann...@gmail.com
> S hannan.khan.dhaka
> B hannan-tech.blogspot.com
> L https://bd.linkedin.com/in/hannankhan
>
>
>
>
___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-users] Server processor use 100%

[Hannan Khan]

Dear Bob

Sorry for replaying late. I quite busy to complete few incomplete tasks
before I am going on holiday tomorrow for a week.

I have checked for few day with various options and my conclusion is that
the security hole might be created by our old war file (version 16) with
Stuart vulnerability which Lars warn all of us earlier. We upgraded all our
servers and applications except this server. No suspicious files in the tmp
folders.

It took control of Tomcat8 user and run SSHD and occupies 100% of 2
processors. When we kill the process and remove all war files and stop
tomcat8 service it stared ATD command and it also occupy 100% of 2
processors. The data seems intact (through query and size). As our all DB
servers have similar IP structure we immediately remove tomcat8 service,
package and user. The VM server will also be decommissioned and will setup
a new server with new cardinals. I will start upgrade work after I return.

Thank you for your valuable advice and kind concern.

Best regards

Hannan

On Mon, Jul 10, 2017 at 8:21 PM, Bob Jolliffe  wrote:

> Sorry that should have been 'ls -la /tmp'
>
> On 10 July 2017 at 10:50, Bob Jolliffe  wrote:
>
>> Hi Hannan
>>
>> There is no circumstance that tomcat user should be running the sshd
>> command.  It could be this machine has been compromised.  Unless you have
>> some strange setup that you are logging in as tomcat user.
>>
>> Please contact me directly if you want me to check.
>>
>> Meanwhile you might want to have a look in /tmp directory and tomcat8
>> home directory to see if there are any strange files there:
>>
>> ls -ls /tmp
>>
>> You might find that there is a rogue sshd program that has been installed
>> there.  Note that if you are running a very old war file your risk of
>> compromise is very high.
>>
>> Bob
>>
>> On 10 July 2017 at 05:09, Hannan Khan  wrote:
>>
>>> Dear Experts
>>>
>>> I have an wired situation. one of our DHIS2 server running older war
>>> files (version 16), the OS was outdated and we have to upgrade the OS.
>>> After installing new OS Ubuntu 16.04 LTS all necessary component Java 8 and
>>> Tomcat 7 was installed by after running war file (version 16) after few
>>> minutes the tomcat7 is not operational as the processor use is 100%. there
>>> is only 1 user logged in and the application server using 2 processor and
>>> DB server is separate.
>>>
>>> After trying several times I remove tomcat7 and install tomcat 8 with
>>> same war file, but situation is same. I called it wired as the db size is
>>> quite small, user is only few and the listing showing SSHD command by
>>> tomcat8 user is using 100% processor.
>>>
>>> Any idea about the under line reason? need urgent help. Thank you all in
>>> advance.
>>>
>>> Regards
>>>
>>> Muhammad Abdul Hannan Khan
>>> Team Leader
>>> Support to the National HMIS
>>> MIS, Director General of Health Service
>>> Ministry of Health and Family Welfare
>>>
>>> T +880-2- 58816459 <+880%202-58816459>, 58816412 ext 118
>>> F +88 02 58813 875
>>> M+88 01819 239 241
>>> M+88 01534 312 066
>>> E hann...@gmail.com
>>> S hannan.khan.dhaka
>>> B hannan-tech.blogspot.com
>>> L https://bd.linkedin.com/in/hannankhan
>>>
>>>
>>>
>>>
>>
>


-- 
Muhammad Abdul Hannan Khan
Team Leader
Support to the National HMIS
MIS, Director General of Health Service
Ministry of Health and Family Welfare

T +880-2- 58816459, 58816412 ext 118
F +88 02 58813 875
M+88 01819 239 241
M+88 01534 312 066
E hann...@gmail.com
S hannan.khan.dhaka
B hannan-tech.blogspot.com
L https://bd.linkedin.com/in/hannankhan
___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp