Does Google actually have LIDAR or are they just running the images that they collect from multiple angles through an algorithm to guess at the obstructions?
----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Adam Moffett" <dmmoff...@gmail.com> To: af@af.afmug.com Sent: Wednesday, December 18, 2019 1:53:10 PM Subject: Re: [AFMUG] clutter data and drones Looks like no 3D building/trees in the profile. On 12/18/2019 1:57 PM, Brian Webster wrote: Adam, If you draw a line between the AP and the address point and then right click to show elevation profile, does it look like it’s including the 3D building/tree as part of the profile or just terrain. We are lucky in NY that the state has mapped every 911 address to the rooftop and they make it available for download. Not everyone has data that accurate. 911 address collection quality varies county by county. Not every county has put it in the public domain for download either. Thank You, Brian Webster www.wirelessmapping.com www.Broadband-Mapping.com From: AF [ mailto:af-boun...@af.afmug.com ] On Behalf Of Adam Moffett Sent: Wednesday, December 18, 2019 1:09 PM To: af@af.afmug.com Subject: Re: [AFMUG] clutter data and drones NY State makes E911 address points available online. Imported a subset of that, turned on the "3D Buildings" layer in Google Earth, and then adjust the view until I'm looking from where the hypothetical AP would be. I can click on each point to get the street address, but I'm not keen to sit here and do that one point at a time. Seems like a job for a computer. On 12/18/2019 12:33 PM, Steve Jones wrote: <blockquote> Thats really cool, how did you get that? On Wed, Dec 18, 2019 at 11:13 AM Adam Moffett < dmmoff...@gmail.com > wrote: <blockquote> Is the USGS LIDAR/clutter data available in a format that you could open in Manifold GIS? See the Google Earth screenshot below. Wouldn't it be nice to just have one more command to select only the address points that are "visible" based on the 3D clutter? I'm betting I could make that work in Manifold. Not an RF projection of any sort, just filter out the address points that can't be "seen" due to 3D clutter. It's an imperfect approach for a lot of reasons, but it would give me hundreds of almost definite LOS households that I could sell 100mbps to if I wanted to. Or Brian, if that's something you could do as a service that would be something I'd be willing to talk about offlist. On 12/18/2019 11:40 AM, Brian Webster wrote: <blockquote> I don’t have that level of detail and haven’t personally done any of the processing, I just know that on the Google Fiber project they would drive a bunch of streets in a city and then the dedicated server would run for two or three days to create a usable point cloud data set. Then it had to be hosted on a separate server due to space requirements. This process was done by a company who specialized in these LIDAR projects so they had the tools to do it the most efficient way. I would guess the overall project file sizes were a couple of terabytes. If you think about it, you have to have a data record for everything you can see/reflect from because that data is used to recreate that object in 3D, so the latitude, longitude and elevation needs to be built for something as simple as a road sign, all the points on the post and all the points to be able to recreate the sign and on both sides. Thank You, Brian Webster www.wirelessmapping.com www.Broadband-Mapping.com From: AF [ mailto:af-boun...@af.afmug.com ] On Behalf Of Steve Jones Sent: Tuesday, December 17, 2019 4:49 PM To: AnimalFarm Microwave Users Group Subject: Re: [AFMUG] clutter data and drones Can either of you describe the data size, and processing time? say clutter per uban square unit (mile kilometers, etc) at this res takes x time to collect with this particular method, it takes roughly y time to process it, and it requires z storage. Generalization is all im looking for here like , current drone/blimp, sensor tied to an eagle can only collect an x meter wide path per sweep so it would take y number of sweeps to cover a square km, the average speed of collection is x meters per minute and it would take so many fuel stops to gain that coverage assuming you have that detail of information On Tue, Dec 17, 2019 at 3:29 PM Brian Webster < i...@wirelessmapping.com > wrote: <blockquote> Yes Chuck the post processing time it takes to create the 3 dimensional/axis point cloud data (points with a lat, long and height value) is massive. When I worked the Google Fiber projects in California we in cities like San Jose and the post processing took days on dedicated high end servers for the limited geographic areas we needed. If you have clutter data at 2 meter resolution you get a much better result of treating the clutter as a solid object than if doing that with 30 meter resolution data. The 2 meter resolution will have such high accuracy of being able to see each building and any single tree that might block a path. In the case of 30 meter data, the clutter gets classified as only one type. So in many non-dense urban areas, the 30 meter square gets classified as low density urban, but then you don’t get any information for a tree lined street or even the rural plains areas where it’s all open crop land with the exception of the single tree line planted to block wind on typical rural farm homes. So averaging the clutter classes becomes more necessary and not treating it all like solid objects. When I worked for EarthLink and we were designing the outdoor Wi-Fi network, we did have the 2 meter resolution tree and building data in Philadelphia. It made a big difference but as I recall we also paid $250,000 just for that single city area clutter at that resolution. As you state there have been business models made on creating this type of data for years. It’s not cheap to create, so the cost justification vs. the added resolution accuracy of your intended project is a key consideration. I do know that NYC has LIDAR data for the whole city in the public domain, worked nice on the WISP propagations I did for the NY State broadband map when they had me produce the WISP coverage areas. In the end it’s all about who is paying to have that high resolution data created. If the government eventually pays for it, then it should be released in the public domain because the public funded it (just like their existing map data is today). It’s still going to take a lot of computing horsepower to digest and use that data in any RF propagation tool however. Thank You, Brian Webster www.wirelessmapping.com www.Broadband-Mapping.com From: AF [mailto: af-boun...@af.afmug.com ] On Behalf Of Chuck Hogg Sent: Tuesday, December 17, 2019 12:46 PM To: AnimalFarm Microwave Users Group Subject: Re: [AFMUG] clutter data and drones I thought I would chime in here a bit. Not disputing Brian or anyone else here, as many accurate statements have been made. I've done some LIDAR propagations at 2M vs 30M DEM data. I found in areas around mountains and hills (consistent in KY/VA markets) it is very helpful to treat "clutter" as obstructions. I have reviewed areas where 30M DEM data shows 400 homes serviced, and 2M LIDAR data shows 17. Going back to the original question, I know 2 people here in KY that fly drones for Engineering, Architectural, and Construction firms and I talked to some of the people at Common Networks, who use some version of Drone Photogrammetry to create their own datasets. In the construction industry it is being used to track building things like bridges, tunnels, commercial buildings, etc. These photogrammetry drone setups are $2-25k. The Drone Lidar setups are $5-300k and require much larger drones. Also, it takes a long time to take this data and compile it. I know they take anywhere from 3-18 months to do this in KY depending on how large the area and how high of a resolution it is. Here's a pretty good video and some tech in the beginning of one in use showing 5cm accuracy: https://www.youtube.com/watch?v=C8piSF40StQ&feature=emb_title Just an FYI, the opportunity to create a business out of this has been going on for many years. They have used planes, blimps, and balloons for years. I could see a move to drones. Best Regards, Chuck Hogg | SVP/Director of Acquisitions ALL POINTS BROADBAND | Live Connected. On Fri, Dec 6, 2019 at 12:14 PM Steve Jones < thatoneguyst...@gmail.com > wrote: <blockquote> Brian, Assuming the software allows you to input your own clutter data, at high resolution, what impact on processing the models is there as the clutter data gets higher in resolution? Are we talking massive percentage? I think im maybe overestimating clutter datas usefulness. I would first need to have accurate topo data that knows what is ground, and what is treetop/building roof. And that data really would need to be at the same, or better resolution than my clutter data. (if my topo data is 30 meter, and my clutter data is 3 meter, my output will be best guess on top of the 30 meter average that may or may not have already included the clutter, depending on when the sampling was done?) Am I misunderstanding clutter data? I had thought it was plotted elevations of clutter, but is it more just regionated averages? if that question makes any sense When the topo data was/is collected, are there mechanisms in play to differentiate terrain from structure/clutter? Say chicago was collected, would it show ground elevation or would it show the rooftop elevations as the average ground elevation? Back to the original query, assuming a guy had a drone with the capability of carrying the equipment and the battery life to not have to constantly recharge. Would a person be able to collect both topo and clutter data, that can differentiate it, and at a fine detail. What kind of data size is that information? I know that the data available to radio mobile in the day could be downloaded over dialup given some time, so it didnt seem to be overly massive. On Fri, Dec 6, 2019 at 9:07 AM Brian Webster < i...@wirelessmapping.com > wrote: <blockquote> Clutter data in the public domain is mostly 30 meter square resolution. Cameron has talked about a lot of the issues with the data. Radio Mobile (and TowerCoverage since it runs on that) has the ability to tune the cluster classifications a bit. I worked with Roger in implementing that clutter model. It is not actually part of the Longley Rice propagation model, what he did at my begging was allow a user to manually edit the height and density for each clutter class and then the tool assigns a loss factor per pixel/30 meter square of clutter and then subtracts the sum total of the clutter loss for the ray being propagated. This is not perfect but when the cell companies use their expensive propagation tools, they tune their clutter models for each market by drive testing a known transmitter with a roving unit and run those drive test results against what the software thinks the signals should be. In this process they compare the know clutter classes that were propagated through and it self-tweaks the loss factors is applies for each clutter class. In radio mobile you do basically the same thing but without automation. To get it right you have to go out and measure a lot of your real world signal levels and manually run propagations until the two match (minus your fade margins built in to your plots). This works well if you spend the time, the bigger issue is that the 30 meter square is assigned just one clutter class code. In general it works well for free stuff. The reality of knowing about specific tree lines alongside a house or in urban environments with tree lined streets or in back years, those individual trees to not get factored in to your propagation, just the building losses if that building clutter is set to a height to show as an obstruction(in WISP cases most are not if you are mounting your antenna on the roof for average suburban clutter). The answer to this is to have higher resolution clutter. The terrain data used is 10 meter resolution, meaning there have been hard data points gathered at least every 10 meters horizontally and interpolated. Some terrain data is available at 3 meters but that is not as widely available. So the issue remains how do you get better resolution clutter data. LIDAR can indeed be used and the best versions are actually driven on the streets and not flown from the air. As Cameron mentioned however that data still only gives you the height/size/area where the clutter is. It does not tell you what type of class that it is and/or what type of RF losses each pixel of that data should be assigned, plus you are typically only getting the clutter data from the street facing side. Think of the old movie sets and only seeing the building face. Another method of increasing clutter accuracy is to resample the data from 30 meter pixels down to smaller sized pixels. This has limited benefit. Mostly this can allow you to take things like tree clutter and trim out the highway areas and or possibly cut out the trees with specific building data footprints and assign a different clutter class by pixel. This is very tedious to do on a large scale and you first have to have other good data sources to trim or reclassify these smaller pixels properly to a new clutter class. While all of this gives you a better physical map of what and where you have clutter down to a more realistic reality, you would then have to go back and manually recalibrate the tuning because tuning over larger pixels is an averaging process using the single clutter class. As you might guess all of this takes time and money. At some point there will likely be some cool efforts done by others where we can integrate this. For instance Microsoft released building outline GIS data for the whole country that they machine learned from aerial imagery. That could be used over resampled data although if the buildings had tree cover they didn’t get captured in the first place because they are not visible in the images. There are other open source projects for things like spectrum sensing on a Raspberry Pi and software defined radio that if you put enough sensors out there they might help tune the clutter loss models. https://electrosense.org/ This is probably way more than you wanted to read about clutter data and RF propagations but hey I am a geek like that. Thank You, Brian Webster www.wirelessmapping.com www.Broadband-Mapping.com From: AF [mailto: af-boun...@af.afmug.com ] On Behalf Of castarritt . Sent: Thursday, December 05, 2019 4:47 PM To: AnimalFarm Microwave Users Group Subject: Re: [AFMUG] clutter data and drones Google maps uses some of the 1M resolution LIDAR data. Check out Austin, TX (maybe most other metro areas as well?) in google, enable "globe view", and then turn on 3D. Now use left ctrl and drag with the mouse to move your view angle. This is the data cnHeat and the Google CBRS SAS solution supposedly use. OT: I wonder if any of the usual suspects are making PC flight simulators that use this data. On Thu, Dec 5, 2019 at 3:30 PM Steve Jones < thatoneguyst...@gmail.com > wrote: <blockquote> The issue with publicly available clutter data is it seems old, poor resolution or inaccurate. If heat is using the same data as linkplanner, its definitely bunk. On Thu, Dec 5, 2019 at 3:26 PM Adam Moffett < dmmoff...@gmail.com > wrote: <blockquote> Have you looked at CnHeat? We're about to do some testing with it here. They mentioned USGS LIDAR as one of the data sources. Presumably that's blended with other imaging somehow. On 12/5/2019 4:02 PM, Cameron Crum wrote: <blockquote> LIDAR is not clutter specific, it just can't penetrate clutter (it's light) so clutter ends up looking like terrain. The benefit is that you get an elevation, the drawback is that you don't know the type of clutter or how high it is above the terrain. I suppose if you compare the lidar data against a terrain only DEM, you could extract the clutter height. Here is the thing... some propagation does penetrate vegetation to some degree, so if you are talking about frequencies that do, then lidar is not necessarily a good thing to use as everything ends up looking like an obstruction. You also need a model that can actually account for clutter (vegetation) density when talking about how much it will affect the signal. Obviously leaf types and things like that can have other effects, but I'm unaware of any model that goes to that depth. While some account for clutter heights to use diffraction losses and some lump-sum type losses for a given clutter category, none of the models that are in use in the wisp industry account for clutter density and there are only a few in existence that do. You can get high res clutter data (types) from thermal satellite imaging from one of the geospatial data companies like Terrapin Geographic, or SPOT. It is surprisingly accurate and is what real prop tools like Planet use. The downside is no elevations, so you still have user input for that. Unless you are willing to shell out big bucks, don't bother looking. We are talking about 10's of thousands for a modestly sized area. The cellcos can afford it. On Thu, Dec 5, 2019 at 10:41 AM Adam Moffett < dmmoff...@gmail.com > wrote: <blockquote> Interesting. And unfortunately I don't know any more about LIDAR than a Google Search does. On 12/5/2019 11:27 AM, Steve Jones wrote: <blockquote> Just the SAS administrators will be competitive product. So garbage in garbage out will really apply. Basic SAS functionality is uniform, but feature sets will differ. More accurate propagation modeling every night will be something we benefit from and Im thinking that will be one of the things they compete against each other with. They didnt say that specifically, but the second iteration of SAS will be more bigger, potentially even bigly in its scope. I really thought it was all going to be modeled after cellco, with a bend toward cellcos overtaking CBRS with shady handshakes and involuntary roaming agreements, but it appears winnforum isnt just government lackeys, the people involved have actually put gear in the air or at least listen to those that have. I think cantgetright may have been a co-chair of a committee somewhere Where would a guy who doesnt know what LIDAR is go to find out more about that clutter data? On Thu, Dec 5, 2019 at 10:12 AM Adam Moffett < dmmoff...@gmail.com > wrote: <blockquote> I think the USGS is making 3D clutter maps with LIDAR. CnHeat is supposed to use that wherever it's available. I haven't heard how that relates to the SAS though. Is this something you learned from the "450 Lady"? Care to share? On 12/5/2019 10:25 AM, Steve Jones wrote: <blockquote> first question is if a guy collects accurate clutter data, can he use it in any of the propagation tools we use? second, and this is where you braniacs come in, what equipment would it take on a drone to collect this data? IIRC drone limit without FAA is something like 300 feet. would that even be tall enough to sweep a wide enough path that it wouldnt take 300 battery charges to do a square mile? I envision a course plotted drone trip that will fly over with a pilot car trailing to maintain the required operator LOS. If you think about how many miles youve put on verifying link paths over the years, its not really a prohibitive thing. CBRS and SAS is whats driving this query, but general propagation anomalies creates quite a pickle that better accuracy/resolution clutter accuracy would alleviate. Please tell me there is already a consortium thats built out a clutter standard with a clutter submission mechanism, that would completely tickle me silly. I also dont know the impact to the propagation back ends as you increase the resolution of the data. Im assuming the SAS administrators are running something a little beefier than Radio Mobile. I could see this being a lucrative niche market, if there were a way around the drone operator licensing requirements (though that cost is pretty minimal). Basically a company builds up a small fleet of drones, outfitted with the appropriate gear. You create an account, input your coverage area (or any region) that you want high resolution data for. they reprogram the course and ship it to you (after collecting the upfront payment, deposit, and massive liability release) they provide you with a road course to drive while the drone does its thing, anticipate points of retrieval for recharge, etc. when its all done, you stick it in the box and ship it back. would be cooler if the whole thing was transported back and forth by amazon drones. If I had a guarantee that the collected data would be useful to the company, into radio mobile, link planner, towercoverage, and SAS administrators, its something i could see a fair price tag of 3-10k on it for our coverage area, and no farmers blasted it out of the sky. we use clutter data now thats antiquated so it would come with the understanding that photosynthesis and bulldozers impact accuracy from the minute its collected. maybe this data is already out there and i dont know? -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com </blockquote> </blockquote> -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com </blockquote> </blockquote> -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com </blockquote> -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com </blockquote> -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com </blockquote> -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com </blockquote> -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com </blockquote> </blockquote> -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com </blockquote> </blockquote> </blockquote> -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
-- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
