Re: [Spacewalk-list] Spacewalk 2.6 Debian repo sync file handle leaking

[Jay McCanta]

I'm still learning how to do a pull request.  I'll get that in place, too.  In 
the meantime, for those with satellite-repo-sync erroring out after a while, 
these two patches will correct that.  The first fixes an issue where the string 
/dists occurs more than once in the URL:  
http://dists.example.com/vol/dists/ubuntun/dists/xenial/
The second really handles the file leak.

--- deb_src.py  2017-11-01 10:18:39.944630552 -0700
+++ backend/satellite_tools/repo_plugins/deb_src.py 2017-11-01 
09:14:15.473989697 -0700
@@ -32,7 +32,7 @@
 # url example - 
http://ftp.debian.org/debian/dists/jessie/main/binary-amd64/
 def __init__(self, url, cache_dir):
 self.url = url
-parts = url.split('/dists')
+parts = url.rsplit('/dists/', 1)
 self.base_url = parts[0]
 self.sslclientcert = self.sslclientkey = self.sslcacert = None
 self.basecachedir = cache_dir
diff -ruN spacewalk.orig/common/rhn_deb.py spacewalk/common/rhn_deb.py
--- spacewalk.orig/common/rhn_deb.py2017-05-19 08:11:17.0 +
+++ spacewalk/common/rhn_deb.py 2017-10-31 17:05:52.438187084 +
@@ -88,6 +88,7 @@
 except Exception:
 e = sys.exc_info()[1]
 raise_with_tb(InvalidPackageError(e), sys.exc_info()[2])
+self.deb = None
 
 @staticmethod
 def checksum_type():
@@ -138,3 +139,4 @@
 if output_stream:
 self.payload_stream = output_stream
 self.payload_size = output_stream.tell() - output_start
+self.header_data.close()



-Original Message-
From: Robert Paschedag [mailto:robert.pasche...@web.de]
Sent: Tuesday, October 31, 2017 12:16 AM
To: spacewalk-list@redhat.com; Jay McCanta <j.mcca...@f5.com>; 
spacewalk-list@redhat.com
Subject: Re: [Spacewalk-list] Spacewalk 2.6 Debian repo sync file handle leaking

EXTERNAL MAIL: robert.pasche...@web.de

Am 31. Oktober 2017 00:17:56 MEZ schrieb Jay McCanta <j.mcca...@f5.com>:
>I am running spacewalk 2.6 on CentOS7 trying to an sync Ubuntu 
>repository.  I have discovered that somewhere in 
>/usr/bin/spacewalk-sync-repo file handles are leaked.  It seems to leak
>2 handles for every file it needs to import.  At some point, I run out 
>of handles.  The files are all temp files '/tmp/tmpXX.  Once all 
>the open handles are exhausted, I get 'ERROR:
>requests.exceptions.RequestException occurred'  over and over and over.
>
>I have tried looking in
>/usr/lib/python2.7/site-packages/spacewalk/common/rhn_deb.py
>/usr/lib/python2.7/site-packages/spacewalk/satellite_tools/repo_plugins
>/deb_src.py
>
>To see if I can find it, but I cannot.  The repo (xenial-universe) has
>43567 packages in it.  Has anyone else encountered (and fixed) this?
>
>Jay McCanta
>F5 Networks, Inc.
>Seattle,  WA 98119
>
>We Make Apps GO

Hi,

because I just upgraded from SW 2.4 to 2.7 I'm still using Steve Meier's 
"spacewalk-debian-sync" script to import Debian packages .I also tried to 
import Debian repos with spacewalk-repo-sync but that often had errors 
importing many packages. Someone then suggested, to use the script from Steve 
for the first, large import and then switch to SW default script. That should 
work, but I did not yet test it.

Robert

___
Spacewalk-list mailing list
Spacewalk-list@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list

Re: [Spacewalk-list] Spacewalk 2.6 Debian repo sync file handle leaking

[Jay McCanta]

Crap.  This is embarrassing. Ignore this patch.  It doesn't work and wasn’t 
meant to be posted.  Got ahead of myself and clicked 'ok' to send, when my 
laptop was rebooting.  Egg on face. Since it's Halloween, let's call it my 
costume. 

Jay


-Original Message-
From: Jay McCanta 
Sent: Tuesday, October 31, 2017 11:49 AM
To: 'Robert Paschedag' <robert.pasche...@web.de>; spacewalk-list@redhat.com; 
spacewalk-list@redhat.com
Subject: RE: [Spacewalk-list] Spacewalk 2.6 Debian repo sync file handle leaking

I think I found the issue.  Here's the patch I applied to get 
spacewalk-repo-sync working. It closes the files opened when they are no longer 
used.  This prevents the file-handle leakage. 

diff -ruN spacewalk.orig/common/rhn_deb.py spacewalk/common/rhn_deb.py
--- spacewalk.orig/common/rhn_deb.py2017-05-19 08:11:17.0 +
+++ spacewalk/common/rhn_deb.py 2017-10-31 17:05:52.438187084 +
@@ -88,6 +88,7 @@
 except Exception:
 e = sys.exc_info()[1]
 raise_with_tb(InvalidPackageError(e), sys.exc_info()[2])
+self.deb = None
 
 @staticmethod
 def checksum_type():
@@ -138,3 +139,4 @@
 if output_stream:
 self.payload_stream = output_stream
 self.payload_size = output_stream.tell() - output_start
+self.header_data.close()



-Original Message-
From: Robert Paschedag [mailto:robert.pasche...@web.de]
Sent: Tuesday, October 31, 2017 12:16 AM
To: spacewalk-list@redhat.com; Jay McCanta <j.mcca...@f5.com>; 
spacewalk-list@redhat.com
Subject: Re: [Spacewalk-list] Spacewalk 2.6 Debian repo sync file handle leaking

EXTERNAL MAIL: robert.pasche...@web.de

Am 31. Oktober 2017 00:17:56 MEZ schrieb Jay McCanta <j.mcca...@f5.com>:
>I am running spacewalk 2.6 on CentOS7 trying to an sync Ubuntu 
>repository.  I have discovered that somewhere in 
>/usr/bin/spacewalk-sync-repo file handles are leaked.  It seems to leak
>2 handles for every file it needs to import.  At some point, I run out 
>of handles.  The files are all temp files '/tmp/tmpXX.  Once all 
>the open handles are exhausted, I get 'ERROR:
>requests.exceptions.RequestException occurred'  over and over and over.
>
>I have tried looking in
>/usr/lib/python2.7/site-packages/spacewalk/common/rhn_deb.py
>/usr/lib/python2.7/site-packages/spacewalk/satellite_tools/repo_plugins
>/deb_src.py
>
>To see if I can find it, but I cannot.  The repo (xenial-universe) has
>43567 packages in it.  Has anyone else encountered (and fixed) this?
>
>Jay McCanta
>F5 Networks, Inc.
>Seattle,  WA 98119
>
>We Make Apps GO

Hi,

because I just upgraded from SW 2.4 to 2.7 I'm still using Steve Meier's 
"spacewalk-debian-sync" script to import Debian packages .I also tried to 
import Debian repos with spacewalk-repo-sync but that often had errors 
importing many packages. Someone then suggested, to use the script from Steve 
for the first, large import and then switch to SW default script. That should 
work, but I did not yet test it.

Robert

___
Spacewalk-list mailing list
Spacewalk-list@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list

Re: [Spacewalk-list] Spacewalk 2.6 Debian repo sync file handle leaking

[Robert Paschedag]

Am 31. Oktober 2017 19:49:11 MEZ schrieb Jay McCanta <j.mcca...@f5.com>:
>I think I found the issue.  Here's the patch I applied to get
>spacewalk-repo-sync working. It closes the files opened when they are
>no longer used.  This prevents the file-handle leakage. 
>
>diff -ruN spacewalk.orig/common/rhn_deb.py spacewalk/common/rhn_deb.py
>--- spacewalk.orig/common/rhn_deb.py   2017-05-19 08:11:17.0
>+
>+++ spacewalk/common/rhn_deb.py2017-10-31 17:05:52.438187084 +
>@@ -88,6 +88,7 @@
> except Exception:
> e = sys.exc_info()[1]
> raise_with_tb(InvalidPackageError(e), sys.exc_info()[2])
>+self.deb = None
> 
> @staticmethod
> def checksum_type():
>@@ -138,3 +139,4 @@
> if output_stream:
> self.payload_stream = output_stream
> self.payload_size = output_stream.tell() - output_start
>+self.header_data.close()
>
>
>
>-Original Message-
>From: Robert Paschedag [mailto:robert.pasche...@web.de] 
>Sent: Tuesday, October 31, 2017 12:16 AM
>To: spacewalk-list@redhat.com; Jay McCanta <j.mcca...@f5.com>;
>spacewalk-list@redhat.com
>Subject: Re: [Spacewalk-list] Spacewalk 2.6 Debian repo sync file
>handle leaking
>
>EXTERNAL MAIL: robert.pasche...@web.de
>
>Am 31. Oktober 2017 00:17:56 MEZ schrieb Jay McCanta
><j.mcca...@f5.com>:
>>I am running spacewalk 2.6 on CentOS7 trying to an sync Ubuntu 
>>repository.  I have discovered that somewhere in 
>>/usr/bin/spacewalk-sync-repo file handles are leaked.  It seems to
>leak
>>2 handles for every file it needs to import.  At some point, I run out
>
>>of handles.  The files are all temp files '/tmp/tmpXX.  Once all 
>>the open handles are exhausted, I get 'ERROR:
>>requests.exceptions.RequestException occurred'  over and over and
>over.
>>
>>I have tried looking in
>>/usr/lib/python2.7/site-packages/spacewalk/common/rhn_deb.py
>>/usr/lib/python2.7/site-packages/spacewalk/satellite_tools/repo_plugins
>>/deb_src.py
>>
>>To see if I can find it, but I cannot.  The repo (xenial-universe) has
>>43567 packages in it.  Has anyone else encountered (and fixed) this?
>>
>>Jay McCanta
>>F5 Networks, Inc.
>>Seattle,  WA 98119
>>
>>We Make Apps GO
>
>Hi,
>
>because I just upgraded from SW 2.4 to 2.7 I'm still using Steve
>Meier's "spacewalk-debian-sync" script to import Debian packages .I
>also tried to import Debian repos with spacewalk-repo-sync but that
>often had errors importing many packages. Someone then suggested, to
>use the script from Steve for the first, large import and then switch
>to SW default script. That should work, but I did not yet test it.
>
>Robert

Hi Jay,

If the import works now, you should open a pull request on GitHub so this patch 
finds its way into the master branch.

Good work.

Robert

___
Spacewalk-list mailing list
Spacewalk-list@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list

Re: [Spacewalk-list] Spacewalk 2.6 Debian repo sync file handle leaking

[Jay McCanta]

I think I found the issue.  Here's the patch I applied to get 
spacewalk-repo-sync working. It closes the files opened when they are no longer 
used.  This prevents the file-handle leakage. 

diff -ruN spacewalk.orig/common/rhn_deb.py spacewalk/common/rhn_deb.py
--- spacewalk.orig/common/rhn_deb.py2017-05-19 08:11:17.0 +
+++ spacewalk/common/rhn_deb.py 2017-10-31 17:05:52.438187084 +
@@ -88,6 +88,7 @@
 except Exception:
 e = sys.exc_info()[1]
 raise_with_tb(InvalidPackageError(e), sys.exc_info()[2])
+self.deb = None
 
 @staticmethod
 def checksum_type():
@@ -138,3 +139,4 @@
 if output_stream:
 self.payload_stream = output_stream
 self.payload_size = output_stream.tell() - output_start
+self.header_data.close()



-Original Message-
From: Robert Paschedag [mailto:robert.pasche...@web.de] 
Sent: Tuesday, October 31, 2017 12:16 AM
To: spacewalk-list@redhat.com; Jay McCanta <j.mcca...@f5.com>; 
spacewalk-list@redhat.com
Subject: Re: [Spacewalk-list] Spacewalk 2.6 Debian repo sync file handle leaking

EXTERNAL MAIL: robert.pasche...@web.de

Am 31. Oktober 2017 00:17:56 MEZ schrieb Jay McCanta <j.mcca...@f5.com>:
>I am running spacewalk 2.6 on CentOS7 trying to an sync Ubuntu 
>repository.  I have discovered that somewhere in 
>/usr/bin/spacewalk-sync-repo file handles are leaked.  It seems to leak
>2 handles for every file it needs to import.  At some point, I run out 
>of handles.  The files are all temp files '/tmp/tmpXX.  Once all 
>the open handles are exhausted, I get 'ERROR:
>requests.exceptions.RequestException occurred'  over and over and over.
>
>I have tried looking in
>/usr/lib/python2.7/site-packages/spacewalk/common/rhn_deb.py
>/usr/lib/python2.7/site-packages/spacewalk/satellite_tools/repo_plugins
>/deb_src.py
>
>To see if I can find it, but I cannot.  The repo (xenial-universe) has
>43567 packages in it.  Has anyone else encountered (and fixed) this?
>
>Jay McCanta
>F5 Networks, Inc.
>Seattle,  WA 98119
>
>We Make Apps GO

Hi,

because I just upgraded from SW 2.4 to 2.7 I'm still using Steve Meier's 
"spacewalk-debian-sync" script to import Debian packages .I also tried to 
import Debian repos with spacewalk-repo-sync but that often had errors 
importing many packages. Someone then suggested, to use the script from Steve 
for the first, large import and then switch to SW default script. That should 
work, but I did not yet test it.

Robert

___
Spacewalk-list mailing list
Spacewalk-list@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list

Re: [Spacewalk-list] Spacewalk 2.6 Debian repo sync file handle leaking

[Robert Paschedag]

Am 31. Oktober 2017 00:17:56 MEZ schrieb Jay McCanta :
>I am running spacewalk 2.6 on CentOS7 trying to an sync Ubuntu
>repository.  I have discovered that somewhere in
>/usr/bin/spacewalk-sync-repo file handles are leaked.  It seems to leak
>2 handles for every file it needs to import.  At some point, I run out
>of handles.  The files are all temp files '/tmp/tmpXX.  Once all
>the open handles are exhausted, I get 'ERROR:
>requests.exceptions.RequestException occurred'  over and over and over.
>
>I have tried looking in
>/usr/lib/python2.7/site-packages/spacewalk/common/rhn_deb.py
>/usr/lib/python2.7/site-packages/spacewalk/satellite_tools/repo_plugins/deb_src.py
>
>To see if I can find it, but I cannot.  The repo (xenial-universe) has
>43567 packages in it.  Has anyone else encountered (and fixed) this?
>
>Jay McCanta
>F5 Networks, Inc.
>Seattle,  WA 98119
>
>We Make Apps GO

Hi,

because I just upgraded from SW 2.4 to 2.7 I'm still using Steve Meier's 
"spacewalk-debian-sync" script to import Debian packages .I also tried to 
import Debian repos with spacewalk-repo-sync but that often had errors 
importing many packages. Someone then suggested, to use the script from Steve 
for the first, large import and then switch to SW default script. That should 
work, but I did not yet test it.

Robert

___
Spacewalk-list mailing list
Spacewalk-list@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list