Re: [OSM-dev] osm2pgsql deltas
On Sat, Jan 16, 2010 at 11:27 PM, Steve Hill wrote: > I've just realised that the hourly and minutely planet deltas have been > replaced by the new transaction based formats, but I can't seem to find > any documentation regarding their use with osm2pgsql. > > Can anyone provide any info on this? for the things i'm using them for they seem to work fine with osm2pgsql without any changes. if you're using osmosis to grab the changes then you'll need to change the rcii/rci tasks to rrii/rri tasks [1], though. cheers, matt [1] http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#Replication_Tasks ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Sun, 17 Jan 2010, Matt Amos wrote: > for the things i'm using them for they seem to work fine with > osm2pgsql without any changes. if you're using osmosis to grab the > changes then you'll need to change the rcii/rci tasks to rrii/rri > tasks [1], though. Ok. The old hourly and minutely deltas had filenames based on their dates; the new ones don't. Is there a sensible way to figuring out which deltas you need without downloading several hundred *.state.txt files? The best I can come up with is trying to do a binary cut to reduce the number of downloads needed to find the right delta, but that's messy as hell compared to how trivial it was when we had date based names. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
Hi, Steve Hill wrote: > The old hourly and minutely deltas had filenames based on their dates; the > new ones don't. Is there a sensible way to figuring out which deltas you > need without downloading several hundred *.state.txt files? Yes, use Osmosis to do it automatically. Initialise with osmosis --rrii then modify the configuration file manually, and find the proper state.txt file that approximately matches your version of the data (explained approx. 100 times on this list, google for --rri), then in regular intervals run osmosis --rri --simc --wxc my-change-file.osc and apply the my-change-file.osc to your database using osm2pgsql. > The best I can come up with is trying to do a binary cut to reduce the > number of downloads needed to find the right delta, but that's messy as > hell compared to how trivial it was when we had date based names. The new method takes a moment to set up but after that it's even easier than the old. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Sun, Jan 17, 2010 at 11:32 PM, Frederik Ramm wrote: > > number of downloads needed to find the right delta, but that's messy as > > hell compared to how trivial it was when we had date based names. > > > The best I can come up with is trying to do a binary cut to reduce the > The new method takes a moment to set up but after that it's even easier > than the old. > If somebody is prepared to write a patch to the --rrii task to search for the right state.txt file to initialise with, then go for it ;-) While the old date based names were easy to identify, unfortunately they don't work with replication files which are only approximately aligned with timestamps. Initialising the replication is a little annoying due to the disconnect between timestamp and sequence number, but as Frederik points out from that point on the replication mechanism is simpler. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Mon, 18 Jan 2010, Brett Henderson wrote: > While the old date based names were easy to identify, unfortunately they > don't work with replication files which are only approximately aligned with > timestamps. Initialising the replication is a little annoying due to the > disconnect between timestamp and sequence number, but as Frederik points out > from that point on the replication mechanism is simpler. With the old system, I used the daily deltas if my database was lagging a day or more behind, hourly deltas if I was more than an hour behind and minutely deltas under normal operation. As I understand it, this is no longer possible and I'll end up having to use minutely deltas all the time. It seems that this new system is going to make the already very time-intensive process of applying deltas even less efficient... Or have I misunderstood how this works? -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
Hi, Steve Hill wrote: > As I understand it, this is > no longer possible and I'll end up having to use minutely deltas all the > time. If you use the --rri task then Osmsois will do all this for you - find out which minutely deltas are required, load them, and out drops a nice and shiny single custom-built .osc file that has exactly the changes you need. You are right in so far as Osmosis cannot mix daily/hourly/minutely diffs, if will always get the minutely if that's what you configure, thus incurring a slight overhead because of the larger number of HTTP requests, but that's all. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
2010/1/18 Frederik Ramm : > You are right in so far as Osmosis cannot mix daily/hourly/minutely > diffs, if will always get the minutely if that's what you configure, > thus incurring a slight overhead because of the larger number of HTTP > requests, but that's all. He meant he can't use daily diffs to catch up more quickly, he has to use individual diff files which is much slower to process thousands of diff files compared to a single daily file. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
Hi, John Smith wrote: > 2010/1/18 Frederik Ramm : >> You are right in so far as Osmosis cannot mix daily/hourly/minutely >> diffs, if will always get the minutely if that's what you configure, >> thus incurring a slight overhead because of the larger number of HTTP >> requests, but that's all. > > He meant he can't use daily diffs to catch up more quickly, he has to > use individual diff files which is much slower to process thousands of > diff files compared to a single daily file. Yes, and I have explained about 1.000 times that what comes out of Osmosis WILL be one single (daily or 1.73daily or 2.918daily or 0.002daily depending on how often you call it) file. If you run Osmosis --rri --simc exactly once per day, then what you will get is exactly what you had in your daily diff before; the only difference being that Osmosis will assemble that from hourly or minutely diffs (depending on your config) and thus will load marginally more data from the server. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Mon, 18 Jan 2010, Frederik Ramm wrote: > Yes, and I have explained about 1.000 times that what comes out of Osmosis You may have explained all this on this list 1,000 times (which I very much doubt), but I couldn't find any of it properly documented anywhere on the Wiki... > WILL be one single (daily or 1.73daily or 2.918daily or 0.002daily depending > on how often you call it) file. I'd wager that having Osmosis download 1440 minutely diffs and assemble them into a daily diff is going to take significantly more time than just wgetting a single daily diff. However, I can't verify my assumption at the moment because I'm having to spend my time battling (and currently failing) to get a honking great Java application (Osmosis) to work to do all this. I'm afraid this seems like a retrograde step to me, compared with the relative simplicity of stuffing diffs directly into osm2pgsql. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
2010/1/18 Steve Hill : > I'm afraid this seems like a retrograde step to me, compared with the > relative simplicity of stuffing diffs directly into osm2pgsql. You can stuff the new diff files straight into osm2pgsql, it's all I'm doing, I coded my own shell script to keep track of where my system is up to and to pull new ones when they are available. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Mon, 18 Jan 2010, John Smith wrote: > You can stuff the new diff files straight into osm2pgsql, it's all I'm > doing, I coded my own shell script to keep track of where my system is > up to and to pull new ones when they are available. I think thats what I'm going to have to do. I can't figure out how to get Osmosis to work - it just sits there doing nothing when I fire it up with "--rri --simc --wxc blah.osc". Even with "-v 9" on the commandline all I get is it sitting there saying "FINE: Reading current server state." and doing nothing more. :( I'm just exceptionally dubious about stuffing minutely deltas directly into osm2pgsql - my system struggles to keep up at times anyway, forcing it to catch up using only minutely deltas seems a recipe for disaster because my experience is that applying minutely deltas has a significant overhead when compared to daily deltas. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
2010/1/18 Steve Hill : > catch up using only minutely deltas seems a recipe for disaster because my > experience is that applying minutely deltas has a significant overhead when > compared to daily deltas. Which is why you keep your system 2-5 minutes behind the DB system and you don't have to catch up ;) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
Hi, Steve Hill wrote: > I'm just exceptionally dubious about stuffing minutely deltas directly > into osm2pgsql I would not recommend that either. The Osmosis process is *really* very elegant and takes all the responsibility of finding out where you are, what you need to get etc. away from you, it may be worth trying to get it to work. The process produces one single diff that you can then apply with osm2pgsql. > it to catch up using only minutely deltas seems a recipe for disaster > because my experience is that applying minutely deltas has a significant > overhead when compared to daily deltas. Yes, it surely has, because after the diff has been applied, the node geometries for all affected ways will be rebuilt. If you have a long way and someone takes his time refining it over the course of 30 minutes, you may incur 30 geometry updates when working with minutely diffs, vs. only one when working with one. So even if you manually download a daily and a few hourly diffs to get you where you want to be, it makes sense to combine them into one *before* applying them. Osmosis does that for you with osmosis --rxc 1.osc --rxc 2.osc --mc --wxc merged.osc But note that the new replication diffs may contain the same object multiple times, so it is always good to add a --simc which will then drop everything but the latest version. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Mon, 18 Jan 2010, John Smith wrote: > Which is why you keep your system 2-5 minutes behind the DB system and > you don't have to catch up ;) Feel free to buy me a server that is capable of doing this. :) My server isn't exactly underpowered, but: 1. From a blank database, it takes several days to import the planet file in slim mode. This means that even if I start importing the planet file the moment it is available, by the time it has finished my server will have several days of deltas to catch up with. 2. At times, there is enough new data in the deltas to cause my server to fall behind. My existing system automatically switched to using hourly or daily deltas as appropriate when it has got far enough behind for them to make sense - this helps it catch up quicker. 3. Any outage is going to end up causing me to have to play "catch up" again anyway, so the faster this can be done the better. Most of this is down to I/O load, and there is only so much that can be done to boost database performance. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
2010/1/18 Steve Hill : > 1. From a blank database, it takes several days to import the planet file in > slim mode. This means that even if I start importing the planet file the > moment it is available, by the time it has finished my server will have > several days of deltas to catch up with. This should almost never have to be done now, unless there is some bug in the diffs that people haven't noticed, previous diffs didn't have all the data, new diffs do. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
Hi, John Smith wrote: > 2010/1/18 Steve Hill : >> 1. From a blank database, it takes several days to import the planet file in >> slim mode. This means that even if I start importing the planet file the >> moment it is available, by the time it has finished my server will have >> several days of deltas to catch up with. > > This should almost never have to be done now, unless there is some bug > in the diffs that people haven't noticed, previous diffs didn't have > all the data, new diffs do. It will still be required if you change your osm2pgsql import rules. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
2010/1/18 Frederik Ramm : > It will still be required if you change your osm2pgsql import rules. I did say almost never Unless you don't care about updating existing data and are happy for the results to filter through ie a new tag you invented. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Mon, Jan 18, 2010 at 9:23 PM, Steve Hill wrote: > On Mon, 18 Jan 2010, Frederik Ramm wrote: > > > Yes, and I have explained about 1.000 times that what comes out of > Osmosis > > You may have explained all this on this list 1,000 times (which I very > much doubt), but I couldn't find any of it properly documented anywhere on > the Wiki... > If it helps, all Osmosis tasks are documented here: http://wiki.openstreetmap.org/wiki/Osmosis/DetailedUsage The documentation may not be comprehensive and does assume some knowledge around how Osmosis pipelines are constructed, but it is accurate and up to date to the best of my knowledge. > > > WILL be one single (daily or 1.73daily or 2.918daily or 0.002daily > depending > > on how often you call it) file. > > I'd wager that having Osmosis download 1440 minutely diffs and assemble > them into a daily diff is going to take significantly more time than just > wgetting a single daily diff. However, I can't verify my assumption at > the moment because I'm having to spend my time battling (and currently > failing) to get a honking great Java application (Osmosis) to work to do > all this. I'm afraid this seems like a retrograde step to me, compared > with the relative simplicity of stuffing diffs directly into osm2pgsql. > Yes, of course it will take longer to download a large number of diffs files than to download one. However I suspect the main overhead will be actually importing the files into PostgreSQL itself. Note that there are in fact hourly files that can be used but like all new replication files they aren't exactly time aligned which makes it harder to switch between them. http://planet.openstreetmap.org/hour-replicate/ "Honking great Java application" is a little excessive, we're hardly talking about installing a JEE server here. Does this refer to the download size, the fact that it runs on Java, or that fact that it seems hard to install. The binary distribution is approximately 8MB, it consumes a very reasonable amount of RAM regardless of dataset size (very approximately 50MB), and I've gone to a lot of effort to make installation as simple as possible. Assuming you have a working Java 1.6 runtime (which should be available out of the box on any modern Linux distribution, and available as a simple installer for Windows), then installing Osmosis is as simple as downloading the latest zip file, and extracting it. Yes, there is some setup required to establish replication, but again I've attempted to keep all locally maintained state simple and all exposed in text files. I agree that it would be nice from a user perspective to have a single application (ie. osm2pgsql) just do everything automatically but that has downsides. In particular it requires implementing the replication mechanism in every tool that wishes to support it. The Osmosis --rri has been written to be re-usable by any tool that can consume an *.osc file. It is well debugged and shown itself to be reliable. Replication is not as simple as it might seem, there are a lot of edge cases once you start taking long running transactions, unreliable network connections, zero server-side state, and a large consumer base into account. I guess a way to resolve this would be to create a shell script that wraps all of this together, but I guess nobody has bothered up to this point. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Mon, Jan 18, 2010 at 11:05 PM, Steve Hill wrote: > On Mon, 18 Jan 2010, John Smith wrote: > > > You can stuff the new diff files straight into osm2pgsql, it's all I'm > > doing, I coded my own shell script to keep track of where my system is > > up to and to pull new ones when they are available. > > I think thats what I'm going to have to do. I can't figure out how to get > Osmosis to work - it just sits there doing nothing when I fire it up with > "--rri --simc --wxc blah.osc". Even with "-v 9" on the commandline all I > get is it sitting there saying "FINE: Reading current server state." and > doing nothing more. :( > That seems odd. Are you able to point it to a debugging proxy to figure out what's going on? Does it eventually time out, or does it hang forever? On Windows I use Fiddler, but I'm not sure what else is available. http://www.fiddlertool.com/ I guess a packet sniffer such as Wireshark would also be able to help but could be harder to diagnose. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Mon, Jan 18, 2010 at 12:05 PM, Steve Hill wrote: > On Mon, 18 Jan 2010, John Smith wrote: > >> You can stuff the new diff files straight into osm2pgsql, it's all I'm >> doing, I coded my own shell script to keep track of where my system is >> up to and to pull new ones when they are available. > > I think thats what I'm going to have to do. I can't figure out how to get > Osmosis to work - it just sits there doing nothing when I fire it up with > "--rri --simc --wxc blah.osc". Even with "-v 9" on the commandline all I > get is it sitting there saying "FINE: Reading current server state." and > doing nothing more. :( this is very odd... for me, it completes in about 6s after downloading one hour's worth of changes and merging them. (i didn't change maxInterval in configuration.txt, and that's the default). if you're running under linux it would be very helpful if could you run osmosis via strace and let us know what syscall it's in when it's hanging. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Mon, Jan 18, 2010 at 1:16 PM, Steve Hill wrote: > My server isn't exactly underpowered, but: > > 1. From a blank database, it takes several days to import the planet file > in slim mode. This means that even if I start importing the planet file > the moment it is available, by the time it has finished my server will > have several days of deltas to catch up with. That seems awfully long. You did tune postgresql as described in the wiki before you started, right? You did make sure you actually had enough memory to make sure you aren't swapping in the process? On a system with only 1GB of memory I could get it to complete in a day (though that was a while back).. Have a nice day, -- Martijn van Oosterhout http://svana.org/kleptog/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Tue, 19 Jan 2010, Brett Henderson wrote: > That seems odd. Are you able to point it to a debugging proxy to figure out > what's going on? Does it eventually time out, or does it hang forever? I've got it working, but it seems to hang for about 3 minutes before doing anything - doesn't contact the server in that time, it just appears to sit there. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Tue, 19 Jan 2010, Brett Henderson wrote: > The documentation may not be comprehensive and does assume some knowledge > around how Osmosis pipelines are constructed, but it is accurate and up to > date to the best of my knowledge. It does document each specific commandline parameter, but it is rather lacking in the "how to" sense - i.e. its great if you know exactly what commandline parameters you need and just need to understand how to use them, but less great if you're working out how to use Osmosis to download deltas for the first time. > "Honking great Java application" is a little excessive, we're hardly talking > about installing a JEE server here. Does this refer to the download size, > the fact that it runs on Java, or that fact that it seems hard to install. Well, I was having a bad day and was utterly failing to get Osmosis to work at all so I may have gone off on one here somewhat unnecessarilly... sorry. :) However, my personal opinion of Java is that it is generally a Bad Thing - it has certain specific uses but is massively overused in inappropriate situations. That probably coloured my judgement somewhat. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Tue, 19 Jan 2010, Martijn van Oosterhout wrote: > That seems awfully long. It seems to match what other people have reported here. > You did tune postgresql as described in the > wiki before you started, right? Yep. > You did make sure you actually had > enough memory to make sure you aren't swapping in the process? 8GB of RAM. > On a > system with only 1GB of memory I could get it to complete in a day > (though that was a while back).. Were you using slim mode? I can import in under a day when not using slim mode, however you have to use slim mode if you want to be able to apply deltas and this adds a huge overhead. -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
> 8GB of RAM. > Were you using slim mode? I can import in under a day when not using slim > mode, however you have to use slim mode if you want to be able to apply > deltas and this adds a huge overhead. Did you increase the cached size to allow osm2pgsql to use more memory (--cache)? The default is only 800 MB and with your machine you should be easily able to allocate 4000 MB or so - I found this had a massive effect and is easily missed. -- Brian ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Tue, Jan 19, 2010 at 7:25 PM, Steve Hill wrote: > On Tue, 19 Jan 2010, Brett Henderson wrote: > > The documentation may not be comprehensive and does assume some knowledge >> around how Osmosis pipelines are constructed, but it is accurate and up to >> date to the best of my knowledge. >> > > It does document each specific commandline parameter, but it is rather > lacking in the "how to" sense - i.e. its great if you know exactly what > commandline parameters you need and just need to understand how to use them, > but less great if you're working out how to use Osmosis to download deltas > for the first time. Yep, that's fair enough. There are some generic Osmosis examples here: http://wiki.openstreetmap.org/index.php/Osmosis#Example_Usage and here: http://wiki.openstreetmap.org/wiki/Osmosis/Examples but neither of those cover replication. There's the minutely mapnik page itself: http://wiki.openstreetmap.org/wiki/Minutely_Mapnik The docs could certainly do with better walk throughs. So far I've left the creation of examples to others as they find them focusing on the reference docs myself. As with everything the challenge is finding time to create additional documentation, and more importantly to keep it all up to date. > "Honking great Java application" is a little excessive, we're hardly >> talking >> about installing a JEE server here. Does this refer to the download size, >> the fact that it runs on Java, or that fact that it seems hard to install. >> > > Well, I was having a bad day and was utterly failing to get Osmosis to work > at all so I may have gone off on one here somewhat unnecessarilly... sorry. > :) > > However, my personal opinion of Java is that it is generally a Bad Thing - > it has certain specific uses but is massively overused in inappropriate > situations. That probably coloured my judgement somewhat. Also fair enough. If there was a simple alternative I'd do it. But there's a few things that make Java the right approach for Osmosis: 1. It's already written in Java ;-) 2. It's a language I'm very comfortable in. 3. It has very good performance for a non-native language which is important due to the large data sets involved. 4. It is robust and so far the only bugs I've found in the Java runtime are limitations on gzipped file sizes (fixed in modern JVMs), and problems with some multi-byte UTF-8 sequences (fixed by using Xerces XML libs). I've often thought about re-writing it in C++, but the reality is that I'll never do it and reaching feature parity would be a huge amount of work for limited gains. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Tue, Jan 19, 2010 at 7:19 PM, Steve Hill wrote: > On Tue, 19 Jan 2010, Brett Henderson wrote: > > That seems odd. Are you able to point it to a debugging proxy to figure >> out >> what's going on? Does it eventually time out, or does it hang forever? >> > > I've got it working, but it seems to hang for about 3 minutes before doing > anything - doesn't contact the server in that time, it just appears to sit > there. Matt's suggestion of running under strace might be the way to go here. I've never seen anything like this before. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Wed, 20 Jan 2010, Brett Henderson wrote: > Matt's suggestion of running under strace might be the way to go here. I've > never seen anything like this before. Got it! For some reason, one of the threads creates a PF_INET6 socket on an ephemeral port, listenning on all network interfaces. The same thread then creates a connection to that socket. Due to a misconfiguration on my server, ip6tables was set to drop looped back IPv6 traffic, so it was sitting there waiting for the connection to time out. Fixing the ip6tables config to allow looped bak IPv6 traffic has resolved the problem. Some odd notes: 1. This connection can't be especially important since it was able to carry on working just fine without the connection present. 2. Since it is on an ephemeral port, the socket is presumably only be used for internal communication between threads - as such, using a socket that is listenning for connections on all network interfaces seems like a really bad idea. This sort of stuff should be done with unix domain sockets, not public network sockets. 3. Of course, I have no idea if Osmosis or the JVM itself is responsible for this. :) -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Thu, Jan 21, 2010 at 3:21 AM, Steve Hill wrote: > On Wed, 20 Jan 2010, Brett Henderson wrote: > > Matt's suggestion of running under strace might be the way to go here. >> I've >> never seen anything like this before. >> > > Got it! > > For some reason, one of the threads creates a PF_INET6 socket on an > ephemeral port, listenning on all network interfaces. The same thread then > creates a connection to that socket. Due to a misconfiguration on my > server, ip6tables was set to drop looped back IPv6 traffic, so it was > sitting there waiting for the connection to time out. > > Fixing the ip6tables config to allow looped bak IPv6 traffic has resolved > the problem. > Nice work! I'll try to remember this. I remember somebody on another list having similar problems with osm2pgsql and IPv6 the other day ... > > Some odd notes: > > 1. This connection can't be especially important since it was able to carry > on working just fine without the connection present. > 2. Since it is on an ephemeral port, the socket is presumably only be used > for internal communication between threads - as such, using a socket that is > listenning for connections on all network interfaces seems like a really bad > idea. This sort of stuff should be done with unix domain sockets, not > public network sockets. > 3. Of course, I have no idea if Osmosis or the JVM itself is responsible > for this. :) It must be the JVM ... The only socket stuff Osmosis does is making HTTP requests, it never listens on sockets. I'm very surprised it's listening on all interfaces though. I know there's some debug and profiling type ports that Java 6 opens but I thought they were only available locally and I (perhaps wrongly) assumed they were secured in some way (eg. You can connect using the jconsole tool and get stats on running JVMs). When I get a chance I'll take a look at it, but I suspect there's no much I can do about it. I'd be surprised if there was a security risk created by this, but who knows ... Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
2010/1/21 Brett Henderson : > Nice work! I'll try to remember this. I remember somebody on another list > having similar problems with osm2pgsql and IPv6 the other day ... A similar problem with debian and JOSM exist: http://www.openstreetmap.org/user/JohnSmith/diary/9124 Maybe the same fix would have fixed this problem too? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Thu, 2010-01-21 at 07:26 +1000, John Smith wrote: > 2010/1/21 Brett Henderson : > > Nice work! I'll try to remember this. I remember somebody on another list > > having similar problems with osm2pgsql and IPv6 the other day ... > > A similar problem with debian and JOSM exist: > > http://www.openstreetmap.org/user/JohnSmith/diary/9124 > > Maybe the same fix would have fixed this problem too? yes. I did fix my josm that way. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Thu, 21 Jan 2010, Brett Henderson wrote: > It must be the JVM ... The only socket stuff Osmosis does is making HTTP > requests, it never listens on sockets. I'm very surprised it's listening on > all interfaces though. I know there's some debug and profiling type ports > that Java 6 opens but I thought they were only available locally and I > (perhaps wrongly) assumed they were secured in some way (eg. You can connect > using the jconsole tool and get stats on running JVMs). It is possible that the JVM will check the source address and drop remote connections... seems a bit silly though. For what its worth, my Java version info is: java version "1.6.0" OpenJDK Runtime Environment (build 1.6.0-b09) OpenJDK 64-Bit Server VM (build 1.6.0-b09, mixed mode) -- - Steve xmpp:st...@nexusuk.org sip:st...@nexusuk.org http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
