Re: timers in threaded application

[dieter]

Skip Montanaro  writes:

> On Fri, Jan 13, 2017 at 3:23 AM, dieter  wrote:
>
>> If what you want to timeout are not I/O operations, you can have a
>> look at "threading.Timer". It uses a separate thread, lets it wait
>> a specified time and then starts a specified function, unless
>> "cancel"ed.
>>
>
> Ooh, hadn't considered that. That would seem to do the trick. I assume the
> function to be executed will be run in the timer's thread, so will have to
> suitably lock any shared data.

Yes.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Running Virtualenv with a custom distlib?

[dieter]

haraldnordg...@gmail.com writes:

> I want to do some development on `distlib`, and in the process run the code 
> via `virtualenv` which has distlib as a dependency.
>
> That is, not run the process inside a virtualenv, but run virtualenv's code 
> using a custom dependency. What are the steps I need to go through to achieve 
> this?

I would try to run "virtualenv" in a virtual environment with a "distlib"
seemingly to fulfill the "virtualenv" requirements but in fact be under
your control (e.g. set up via "python setup.py develop").

I do not know whether the virtual environment mentioned above can
be set up via "virtualenv" itself (that would be the easiest thing).
If not, I would install "python" from source and tweak its "site-packages"
as necessary.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python Web Scrapping : Within href readonly those value that have href in it

[Peter Otten]

shahs...@gmail.com wrote:

> I am trying to scrape a webpage just for learning. In that webpage there
> are multiple "a" tags. consider the below code
> 
>  Something 
> 
>  Something

These are probaly all forward slashes.

> Now i want to read only those href in which there is http. My Current code
> is
> 
> for link in soup.find_all("a"):
> print link.get("href")
> 
> i would like to change it to read only http links.

You mean href values that start with "http://";?
While you can do that with a callback

def check_scheme(href):
return href is not None and href.startswith("http://";)

for a in soup.find_all("a", href=check_scheme):
print(a["href"])

or a regular expression

import re

for a in soup.find_all("a", href=re.compile("^http://";)):
print(a["href"])

why not keep things simple and check before printing? Like

for a in soup.find_all("a"):
href = a.get("href", "") # empty string if href is missing
if href.startswith("http://";):
print(href)


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Announcement: TLSv1.2 will become mandatory in the future for Python.org Sites

[dieter]

oliver  writes:

> When I run this per email from my work laptop,
>
> python3 -c "import urllib.request,json;
> print(json.loads(urllib.request.urlopen('
> https://www.howsmyssl.com/a/check').read())['tls_version'])"
>
> I get the following traceback:
> ...
> File "c:\Python35\lib\ssl.py", line 633, in do_handshake
> self._sslobj.do_handshake()
> ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
> (_ssl.c:645)

I guess (!) that somehow the well known trusted CA (= "Certificate authority")
certificates are incomplete on your machine.

Certificate verification works as follows:
a certificate is always signed by a certificate authority ("CA");
for a certificate to be trusted, the signing CA must be trusted.
There may be several trust steps but finally, there must be
some "CA" that you are trusting "without further proof".
The certificates of those "CA"s are somewhere stored on your machine.

Apparently, the "https" servers you have problems with
are using a CA which is not declared trusted on your machine
(by installing the appropriate certificate at the correct place).

-- 
https://mail.python.org/mailman/listinfo/python-list

geopandas bug ?

[Xristos Xristoou]

i want  to create a simple spatial joing using geopandas but i thing so 
geopandas has bug ?



geopandas code :

from geopandas import gpd
import geopandas
points = geopandas.GeoDataFrame.from_file('points.shp') # or geojson etc
polys = geopandas.GeoDataFrame.from_file('polygons.shp')
pointInPoly = gpd.sjoin(points, polys, how='left',op='within')

error :

Traceback (most recent call last):
  File "/home/sarantis/testshapely/sumpointsinsidepolygon/testgeo.py", line 7, 
in 
pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
  File "/usr/local/lib/python2.7/dist-packages/geopandas/tools/sjoin.py", line 
57, in sjoin
r_idx = np.concatenate(idxmatch.values)
ValueError: need at least one array to concatenate

and if i change the imports with the some code :

import geopandas
import pandas as pd
import geopandas as gpd
from geopandas import GeoDataFrame, read_file
from geopandas.tools import sjoin
from shapely.geometry import Point, mapping,shape
import pandas as gpd

i take that error :

pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
AttributeError: 'module' object has no attribute 'sjoin'


any idea why ?
-- 
https://mail.python.org/mailman/listinfo/python-list

geopandas bug ?

[Xristos Xristoou]

i want  to create a simple spatial joing using geopandas but i thing so 
geopandas has bug ?



geopandas code :

from geopandas import gpd
import geopandas
points = geopandas.GeoDataFrame.from_file('points.shp') # or geojson etc
polys = geopandas.GeoDataFrame.from_file('polygons.shp')
pointInPoly = gpd.sjoin(points, polys, how='left',op='within')

error :

Traceback (most recent call last):
  File "/home/username/testshapely/sumpointsinsidepolygon/testgeo.py", line 7, 
in 
pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
  File "/usr/local/lib/python2.7/dist-packages/geopandas/tools/sjoin.py", line 
57, in sjoin
r_idx = np.concatenate(idxmatch.values)
ValueError: need at least one array to concatenate

and if i change the imports with the some code :

import geopandas
import pandas as pd
import geopandas as gpd
from geopandas import GeoDataFrame, read_file
from geopandas.tools import sjoin
from shapely.geometry import Point, mapping,shape
import pandas as gpd

i take that error :

pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
AttributeError: 'module' object has no attribute 'sjoin'


any idea why ?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[Peter Otten]

Xristos Xristoou wrote:

> i want  to create a simple spatial joing using geopandas but i thing so
> geopandas has bug ?

Have you tried the examples on ?
Do they work? If yes, inspect your data, does it have the same format?

> geopandas code :
> 
> from geopandas import gpd
> import geopandas
> points = geopandas.GeoDataFrame.from_file('points.shp') # or geojson etc
> polys = geopandas.GeoDataFrame.from_file('polygons.shp')
> pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> 
> error :
> 
> Traceback (most recent call last):
>   File "/home/sarantis/testshapely/sumpointsinsidepolygon/testgeo.py",
>   line 7, in 
> pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
>   File "/usr/local/lib/python2.7/dist-packages/geopandas/tools/sjoin.py",
>   line 57, in sjoin
> r_idx = np.concatenate(idxmatch.values)
> ValueError: need at least one array to concatenate
> 
> and if i change the imports with the some code :
> 
> import geopandas
> import pandas as pd
> import geopandas as gpd
> from geopandas import GeoDataFrame, read_file
> from geopandas.tools import sjoin
> from shapely.geometry import Point, mapping,shape
> import pandas as gpd
> 
> i take that error :
> 
> pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> AttributeError: 'module' object has no attribute 'sjoin'
> 
> 
> any idea why ?

My crystal ball says that either points or polys is empty ;)


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[Xristos Xristoou]

Τη Σάββατο, 14 Ιανουαρίου 2017 - 3:43:10 μ.μ. UTC+2, ο χρήστης Peter Otten 
έγραψε:
> Xristos Xristoou wrote:
> 
> > i want  to create a simple spatial joing using geopandas but i thing so
> > geopandas has bug ?
> 
> Have you tried the examples on ?
> Do they work? If yes, inspect your data, does it have the same format?
> 
> > geopandas code :
> > 
> > from geopandas import gpd
> > import geopandas
> > points = geopandas.GeoDataFrame.from_file('points.shp') # or geojson etc
> > polys = geopandas.GeoDataFrame.from_file('polygons.shp')
> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> > 
> > error :
> > 
> > Traceback (most recent call last):
> >   File "/home/sarantis/testshapely/sumpointsinsidepolygon/testgeo.py",
> >   line 7, in 
> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> >   File "/usr/local/lib/python2.7/dist-packages/geopandas/tools/sjoin.py",
> >   line 57, in sjoin
> > r_idx = np.concatenate(idxmatch.values)
> > ValueError: need at least one array to concatenate
> > 
> > and if i change the imports with the some code :
> > 
> > import geopandas
> > import pandas as pd
> > import geopandas as gpd
> > from geopandas import GeoDataFrame, read_file
> > from geopandas.tools import sjoin
> > from shapely.geometry import Point, mapping,shape
> > import pandas as gpd
> > 
> > i take that error :
> > 
> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> > AttributeError: 'module' object has no attribute 'sjoin'
> > 
> > 
> > any idea why ?
> 
> My crystal ball says that either points or polys is empty ;)

is not empty and yes i have two shapefiles from qgis.what is the error ?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[Peter Otten]

Xristos Xristoou wrote:

> Τη Σάββατο, 14 Ιανουαρίου 2017 - 3:43:10 μ.μ. UTC+2, ο χρήστης Peter Otten
> έγραψε:
>> Xristos Xristoou wrote:
>> 
>> > i want  to create a simple spatial joing using geopandas but i thing so
>> > geopandas has bug ?
>> 
>> Have you tried the examples on ?
>> Do they work? If yes, inspect your data, does it have the same format?

Looks like you chose to ignore the hard part.
 
>> > geopandas code :
>> > 
>> > from geopandas import gpd
>> > import geopandas
>> > points = geopandas.GeoDataFrame.from_file('points.shp') # or geojson
>> > etc polys = geopandas.GeoDataFrame.from_file('polygons.shp')
>> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
>> > 
>> > error :
>> > 
>> > Traceback (most recent call last):
>> >   File "/home/sarantis/testshapely/sumpointsinsidepolygon/testgeo.py",
>> >   line 7, in 
>> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
>> >   File
>> >   "/usr/local/lib/python2.7/dist-packages/geopandas/tools/sjoin.py",
>> >   line 57, in sjoin
>> > r_idx = np.concatenate(idxmatch.values)
>> > ValueError: need at least one array to concatenate

>> > any idea why ?
>> 
>> My crystal ball says that either points or polys is empty ;)
> 
> is not empty and yes i have two shapefiles from qgis.

Can I download those files somewhere?

> what is the error ?

No idea. Without those files I cannot run your code.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[Xristos Xristoou]

Τη Σάββατο, 14 Ιανουαρίου 2017 - 4:30:48 μ.μ. UTC+2, ο χρήστης Peter Otten 
έγραψε:
> Xristos Xristoou wrote:
> 
> > Τη Σάββατο, 14 Ιανουαρίου 2017 - 3:43:10 μ.μ. UTC+2, ο χρήστης Peter Otten
> > έγραψε:
> >> Xristos Xristoou wrote:
> >> 
> >> > i want  to create a simple spatial joing using geopandas but i thing so
> >> > geopandas has bug ?
> >> 
> >> Have you tried the examples on ?
> >> Do they work? If yes, inspect your data, does it have the same format?
> 
> Looks like you chose to ignore the hard part.
>  
> >> > geopandas code :
> >> > 
> >> > from geopandas import gpd
> >> > import geopandas
> >> > points = geopandas.GeoDataFrame.from_file('points.shp') # or geojson
> >> > etc polys = geopandas.GeoDataFrame.from_file('polygons.shp')
> >> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> >> > 
> >> > error :
> >> > 
> >> > Traceback (most recent call last):
> >> >   File "/home/sarantis/testshapely/sumpointsinsidepolygon/testgeo.py",
> >> >   line 7, in 
> >> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> >> >   File
> >> >   "/usr/local/lib/python2.7/dist-packages/geopandas/tools/sjoin.py",
> >> >   line 57, in sjoin
> >> > r_idx = np.concatenate(idxmatch.values)
> >> > ValueError: need at least one array to concatenate
> 
> >> > any idea why ?
> >> 
> >> My crystal ball says that either points or polys is empty ;)
> > 
> > is not empty and yes i have two shapefiles from qgis.
> 
> Can I download those files somewhere?
> 
> > what is the error ?
> 
> No idea. Without those files I cannot run your code.

yes i sent you
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[duncan smith]

On 14/01/17 11:18, Xristos Xristoou wrote:
> i want  to create a simple spatial joing using geopandas but i thing so 
> geopandas has bug ?
> 
> 
> 
> geopandas code :
> 
> from geopandas import gpd
> import geopandas
> points = geopandas.GeoDataFrame.from_file('points.shp') # or geojson etc
> polys = geopandas.GeoDataFrame.from_file('polygons.shp')
> pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> 
> error :
> 
> Traceback (most recent call last):
>   File "/home/sarantis/testshapely/sumpointsinsidepolygon/testgeo.py", line 
> 7, in 
> pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
>   File "/usr/local/lib/python2.7/dist-packages/geopandas/tools/sjoin.py", 
> line 57, in sjoin
> r_idx = np.concatenate(idxmatch.values)
> ValueError: need at least one array to concatenate
> 
> and if i change the imports with the some code :
> 
> import geopandas
> import pandas as pd
> import geopandas as gpd
> from geopandas import GeoDataFrame, read_file
> from geopandas.tools import sjoin
> from shapely.geometry import Point, mapping,shape
> import pandas as gpd
> 
> i take that error :
> 
> pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> AttributeError: 'module' object has no attribute 'sjoin'
> 
> 
> any idea why ?
> 


import geopandas as gpd
import pandas as gpd

Does pandas have an attribute 'sjoin'?

Duncan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[Xristos Xristoou]

Τη Σάββατο, 14 Ιανουαρίου 2017 - 4:30:48 μ.μ. UTC+2, ο χρήστης Peter Otten 
έγραψε:
> Xristos Xristoou wrote:
> 
> > Τη Σάββατο, 14 Ιανουαρίου 2017 - 3:43:10 μ.μ. UTC+2, ο χρήστης Peter Otten
> > έγραψε:
> >> Xristos Xristoou wrote:
> >> 
> >> > i want  to create a simple spatial joing using geopandas but i thing so
> >> > geopandas has bug ?
> >> 
> >> Have you tried the examples on ?
> >> Do they work? If yes, inspect your data, does it have the same format?
> 
> Looks like you chose to ignore the hard part.
>  
> >> > geopandas code :
> >> > 
> >> > from geopandas import gpd
> >> > import geopandas
> >> > points = geopandas.GeoDataFrame.from_file('points.shp') # or geojson
> >> > etc polys = geopandas.GeoDataFrame.from_file('polygons.shp')
> >> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> >> > 
> >> > error :
> >> > 
> >> > Traceback (most recent call last):
> >> >   File "/home/sarantis/testshapely/sumpointsinsidepolygon/testgeo.py",
> >> >   line 7, in 
> >> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> >> >   File
> >> >   "/usr/local/lib/python2.7/dist-packages/geopandas/tools/sjoin.py",
> >> >   line 57, in sjoin
> >> > r_idx = np.concatenate(idxmatch.values)
> >> > ValueError: need at least one array to concatenate
> 
> >> > any idea why ?
> >> 
> >> My crystal ball says that either points or polys is empty ;)
> > 
> > is not empty and yes i have two shapefiles from qgis.
> 
> Can I download those files somewhere?
> 
> > what is the error ?
> 
> No idea. Without those files I cannot run your code.

https://www.dropbox.com/s/2693nfi248z0y9q/files.zip?dl=0 with the shapefiles
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[Xristos Xristoou]

Τη Σάββατο, 14 Ιανουαρίου 2017 - 4:38:39 μ.μ. UTC+2, ο χρήστης duncan smith 
έγραψε:
> On 14/01/17 11:18, Xristos Xristoou wrote:
> > i want  to create a simple spatial joing using geopandas but i thing so 
> > geopandas has bug ?
> > 
> > 
> > 
> > geopandas code :
> > 
> > from geopandas import gpd
> > import geopandas
> > points = geopandas.GeoDataFrame.from_file('points.shp') # or geojson etc
> > polys = geopandas.GeoDataFrame.from_file('polygons.shp')
> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> > 
> > error :
> > 
> > Traceback (most recent call last):
> >   File "/home/sarantis/testshapely/sumpointsinsidepolygon/testgeo.py", line 
> > 7, in 
> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> >   File "/usr/local/lib/python2.7/dist-packages/geopandas/tools/sjoin.py", 
> > line 57, in sjoin
> > r_idx = np.concatenate(idxmatch.values)
> > ValueError: need at least one array to concatenate
> > 
> > and if i change the imports with the some code :
> > 
> > import geopandas
> > import pandas as pd
> > import geopandas as gpd
> > from geopandas import GeoDataFrame, read_file
> > from geopandas.tools import sjoin
> > from shapely.geometry import Point, mapping,shape
> > import pandas as gpd
> > 
> > i take that error :
> > 
> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> > AttributeError: 'module' object has no attribute 'sjoin'
> > 
> > 
> > any idea why ?
> > 
> 
> 
> import geopandas as gpd
> import pandas as gpd
> 
> Does pandas have an attribute 'sjoin'?
> 
> Duncan

i dont know i follow detais i am newbie
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[Peter Otten]

Xristos Xristoou wrote:

> Τη Σάββατο, 14 Ιανουαρίου 2017 - 4:30:48 μ.μ. UTC+2, ο χρήστης Peter Otten
> έγραψε:
>> Xristos Xristoou wrote:
>> 
>> > Τη Σάββατο, 14 Ιανουαρίου 2017 - 3:43:10 μ.μ. UTC+2, ο χρήστης Peter
>> > Otten έγραψε:
>> >> Xristos Xristoou wrote:
>> >> 
>> >> > i want  to create a simple spatial joing using geopandas but i thing
>> >> > so geopandas has bug ?
>> >> 
>> >> Have you tried the examples on
>> >> ? Do they work? If yes, inspect
>> >> your data, does it have the same format?
>> 
>> Looks like you chose to ignore the hard part.
>>  
>> >> > geopandas code :
>> >> > 
>> >> > from geopandas import gpd
>> >> > import geopandas
>> >> > points = geopandas.GeoDataFrame.from_file('points.shp') # or geojson
>> >> > etc polys = geopandas.GeoDataFrame.from_file('polygons.shp')
>> >> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
>> >> > 
>> >> > error :
>> >> > 
>> >> > Traceback (most recent call last):
>> >> >   File
>> >> >   "/home/sarantis/testshapely/sumpointsinsidepolygon/testgeo.py",
>> >> >   line 7, in 
>> >> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
>> >> >   File
>> >> >   "/usr/local/lib/python2.7/dist-packages/geopandas/tools/sjoin.py",
>> >> >   line 57, in sjoin
>> >> > r_idx = np.concatenate(idxmatch.values)
>> >> > ValueError: need at least one array to concatenate
>> 
>> >> > any idea why ?
>> >> 
>> >> My crystal ball says that either points or polys is empty ;)
>> > 
>> > is not empty and yes i have two shapefiles from qgis.
>> 
>> Can I download those files somewhere?
>> 
>> > what is the error ?
>> 
>> No idea. Without those files I cannot run your code.
> 
> https://www.dropbox.com/s/2693nfi248z0y9q/files.zip?dl=0 with the
> shapefiles

It looks like there are no intersections in your data.
With the proviso that I've learned about the library only today I think you 
should get an empty result set rather than the ValueError.
Here's a way to reproduce the error (?) with the data provided in the 
project:

import geopandas
from geopandas import gpd

world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
cities = gpd.read_file(gpd.datasets.get_path('naturalearth_cities'))
countries = world[['geometry', 'name']]

def find(items, name):
# didn't find the idomatic way quickly, so
for index, n in enumerate(items["name"]):
if n == name:
return items[index:index+1]
raise ValueError

berlin = find(cities, "Berlin")
paris = find(cities, "Paris")
germany = find(countries, "Germany")

print(gpd.sjoin(berlin, germany))
print(gpd.sjoin(paris, germany)) # ValueError
 
$ python demo.py
geometry name_left  index_right  \
175  POINT (13.39960276470055 52.52376452225116)Berlin   41   

name_right  
175Germany  

[1 rows x 4 columns]
Traceback (most recent call last):
  File "demo.py", line 20, in 
print(gpd.sjoin(paris, germany)) # ValueError
  File "/home/peter/virt/geopandas/lib/python3.4/site-
packages/geopandas/tools/sjoin.py", line 57, in sjoin
r_idx = np.concatenate(idxmatch.values)
ValueError: need at least one array to concatenate
$

I suggest that you file a bug report.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[Xristos Xristoou]

Τη Σάββατο, 14 Ιανουαρίου 2017 - 6:01:39 μ.μ. UTC+2, ο χρήστης Peter Otten 
έγραψε:
> Xristos Xristoou wrote:
> 
> > Τη Σάββατο, 14 Ιανουαρίου 2017 - 4:30:48 μ.μ. UTC+2, ο χρήστης Peter Otten
> > έγραψε:
> >> Xristos Xristoou wrote:
> >> 
> >> > Τη Σάββατο, 14 Ιανουαρίου 2017 - 3:43:10 μ.μ. UTC+2, ο χρήστης Peter
> >> > Otten έγραψε:
> >> >> Xristos Xristoou wrote:
> >> >> 
> >> >> > i want  to create a simple spatial joing using geopandas but i thing
> >> >> > so geopandas has bug ?
> >> >> 
> >> >> Have you tried the examples on
> >> >> ? Do they work? If yes, inspect
> >> >> your data, does it have the same format?
> >> 
> >> Looks like you chose to ignore the hard part.
> >>  
> >> >> > geopandas code :
> >> >> > 
> >> >> > from geopandas import gpd
> >> >> > import geopandas
> >> >> > points = geopandas.GeoDataFrame.from_file('points.shp') # or geojson
> >> >> > etc polys = geopandas.GeoDataFrame.from_file('polygons.shp')
> >> >> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> >> >> > 
> >> >> > error :
> >> >> > 
> >> >> > Traceback (most recent call last):
> >> >> >   File
> >> >> >   "/home/sarantis/testshapely/sumpointsinsidepolygon/testgeo.py",
> >> >> >   line 7, in 
> >> >> > pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> >> >> >   File
> >> >> >   "/usr/local/lib/python2.7/dist-packages/geopandas/tools/sjoin.py",
> >> >> >   line 57, in sjoin
> >> >> > r_idx = np.concatenate(idxmatch.values)
> >> >> > ValueError: need at least one array to concatenate
> >> 
> >> >> > any idea why ?
> >> >> 
> >> >> My crystal ball says that either points or polys is empty ;)
> >> > 
> >> > is not empty and yes i have two shapefiles from qgis.
> >> 
> >> Can I download those files somewhere?
> >> 
> >> > what is the error ?
> >> 
> >> No idea. Without those files I cannot run your code.
> > 
> > https://www.dropbox.com/s/2693nfi248z0y9q/files.zip?dl=0 with the
> > shapefiles
> 
> It looks like there are no intersections in your data.
> With the proviso that I've learned about the library only today I think you 
> should get an empty result set rather than the ValueError.
> Here's a way to reproduce the error (?) with the data provided in the 
> project:
> 
> import geopandas
> from geopandas import gpd
> 
> world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
> cities = gpd.read_file(gpd.datasets.get_path('naturalearth_cities'))
> countries = world[['geometry', 'name']]
> 
> def find(items, name):
> # didn't find the idomatic way quickly, so
> for index, n in enumerate(items["name"]):
> if n == name:
> return items[index:index+1]
> raise ValueError
> 
> berlin = find(cities, "Berlin")
> paris = find(cities, "Paris")
> germany = find(countries, "Germany")
> 
> print(gpd.sjoin(berlin, germany))
> print(gpd.sjoin(paris, germany)) # ValueError
>  
> $ python demo.py
> geometry name_left  index_right  \
> 175  POINT (13.39960276470055 52.52376452225116)Berlin   41   
> 
> name_right  
> 175Germany  
> 
> [1 rows x 4 columns]
> Traceback (most recent call last):
>   File "demo.py", line 20, in 
> print(gpd.sjoin(paris, germany)) # ValueError
>   File "/home/peter/virt/geopandas/lib/python3.4/site-
> packages/geopandas/tools/sjoin.py", line 57, in sjoin
> r_idx = np.concatenate(idxmatch.values)
> ValueError: need at least one array to concatenate
> $
> 
> I suggest that you file a bug report.

Mr.Peter Otten do you see my shapefiles ?have instersection 100 to 100 i use 
instersection on QGIS ad work fine
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[Peter Otten]

Xristos Xristoou wrote:

>> I suggest that you file a bug report.
> 
> Mr.Peter Otten do you see my shapefiles ?have instersection 100 to 100 i
> use instersection on QGIS ad work fine

Yes, I downloaded the zipfile at

> https://www.dropbox.com/s/2693nfi248z0y9q/files.zip?dl=0 with the

and when I ran your code I got the very error that you saw. There are many 
NaN values in your data, so if it works elsewhere perhaps the data is 
corrupted in some way. I'm sorry I cannot help you any further. 

Good luck!

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[Xristos Xristoou]

Τη Σάββατο, 14 Ιανουαρίου 2017 - 6:33:54 μ.μ. UTC+2, ο χρήστης Peter Otten 
έγραψε:
> Xristos Xristoou wrote:
> 
> >> I suggest that you file a bug report.
> > 
> > Mr.Peter Otten do you see my shapefiles ?have instersection 100 to 100 i
> > use instersection on QGIS ad work fine
> 
> Yes, I downloaded the zipfile at
> 
> > https://www.dropbox.com/s/2693nfi248z0y9q/files.zip?dl=0 with the
> 
> and when I ran your code I got the very error that you saw. There are many 
> NaN values in your data, so if it works elsewhere perhaps the data is 
> corrupted in some way. I'm sorry I cannot help you any further. 
> 
> Good luck!

one more question,i have a idea what is wrong,but if my code work how to export 
spatial join "pointInPoly" to new shapefile ?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[Peter Otten]

Xristos Xristoou wrote:

> Τη Σάββατο, 14 Ιανουαρίου 2017 - 6:33:54 μ.μ. UTC+2, ο χρήστης Peter Otten
> έγραψε:
>> Xristos Xristoou wrote:
>> 
>> >> I suggest that you file a bug report.
>> > 
>> > Mr.Peter Otten do you see my shapefiles ?have instersection 100 to 100
>> > i use instersection on QGIS ad work fine
>> 
>> Yes, I downloaded the zipfile at
>> 
>> > https://www.dropbox.com/s/2693nfi248z0y9q/files.zip?dl=0 with the
>> 
>> and when I ran your code I got the very error that you saw. There are
>> many NaN values in your data, so if it works elsewhere perhaps the data
>> is corrupted in some way. I'm sorry I cannot help you any further.
>> 
>> Good luck!
> 
> one more question,i have a idea what is wrong,but if my code work how to
> export spatial join "pointInPoly" to new shapefile ?

You can find an object's methods in the interactive interpreter with dir()

>>> dir(pointInPoly)
['T', '_AXIS_ALIASES', '_AXIS_IALIASES', '_AXIS_LEN', '_AXIS_NAMES', 
'_AXIS_NUMBERS', '_AXIS_ORDERS', '_AXIS_REVERSED', '_AXIS_SLICEMAP', 



'rmod', 'rmul', 'rotate', 'rpow', 'rsub', 'rtruediv', 'save', 'scale', 
'select', 'set_geometry', 'set_index', 'set_value', 'shape', 'shift', 
'simplify', 'sindex', 'skew', 'sort', 'sort_index', 'sortlevel', 'squeeze', 
'stack', 'std', 'sub', 'subtract', 'sum', 'swapaxes', 'swaplevel', 
'symmetric_difference', 'tail', 'take', 'to_clipboard', 'to_crs', 'to_csv', 
'to_dense', 'to_dict', 'to_excel', 'to_file', 'to_gbq', 'to_hdf', 'to_html', 
'to_json', 'to_latex', 'to_msgpack', 'to_panel', 'to_period', 'to_pickle', 
'to_records', 'to_sparse', 'to_sql', 'to_stata', 'to_string', 
'to_timestamp', 'to_wide', 'total_bounds', 'touches', 'translate', 
'transpose', 'truediv', 'truncate', 'tshift', 'type', 'tz_convert', 
'tz_localize', 'unary_union', 'union', 'unstack', 'update', 'values', 'var', 
'where', 'within', 'xs']

OK, that's quite a lot, but to_file() seems to be a good candidate. Let's 
see what it does:

>>> help(pointInPoly.to_file)
Help on method to_file in module geopandas.geodataframe:

to_file(filename, driver='ESRI Shapefile', schema=None, **kwargs) metod of 
geopandas.geodataframe.GeoDataFrame instance
Write this GeoDataFrame to an OGR data source

A dictionary of supported OGR providers is available via:
>>> import fiona
>>> fiona.supported_drivers

Parameters
--
filename : string
File path or file handle to write to.
driver : string, default 'ESRI Shapefile'
The OGR format driver used to write the vector file.
schema : dict, default None
If specified, the schema dictionary is passed to Fiona to
better control how the file is written.

The *kwargs* are passed to fiona.open and can be used to write
to multi-layer data, store data within archives (zip files), etc.

Looks good, run it:

>>> pointInPoly.to_file("point_in_poly")

No error. Does it round-trip?

>>> pointInPoly == gpd.GeoDataFrame.from_file("point_in_poly")
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3/dist-packages/pandas/core/ops.py", line 875, in f
return self._compare_frame(other, func, str_rep)
  File "/usr/lib/python3/dist-packages/pandas/core/frame.py", line 2860, in 
_compare_frame
raise ValueError('Can only compare identically-labeled '
ValueError: Can only compare identically-labeled DataFrame objects

Ouch, unfortunately not. Upon further inspection:

>>> pip.columns
Index(['geometry', 'index_righ', 'name_left', 'name_right'], dtype='object')
>>> pointInPoly.columns
Index(['geometry', 'name_left', 'index_right', 'name_right'], 
dtype='object')

Looks like column names are either corrupted or limited to 10 characters by 
default. Again I don't know how to overcome this, but as a special service 
here's the first hit for 'shp file column name limit' on a popular search 
engine:

http://gis.stackexchange.com/questions/15784/how-to-bypass-10-character-limit-of-field-name-in-shapefiles

I'm out of this now.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[duncan smith]

On 14/01/17 14:59, Xristos Xristoou wrote:
> Τη Σάββατο, 14 Ιανουαρίου 2017 - 4:38:39 μ.μ. UTC+2, ο χρήστης duncan smith 
> έγραψε:
>> On 14/01/17 11:18, Xristos Xristoou wrote:
>>> i want  to create a simple spatial joing using geopandas but i thing so 
>>> geopandas has bug ?
>>>
>>>
>>>
>>> geopandas code :
>>>
>>> from geopandas import gpd
>>> import geopandas
>>> points = geopandas.GeoDataFrame.from_file('points.shp') # or geojson etc
>>> polys = geopandas.GeoDataFrame.from_file('polygons.shp')
>>> pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
>>>
>>> error :
>>>
>>> Traceback (most recent call last):
>>>   File "/home/sarantis/testshapely/sumpointsinsidepolygon/testgeo.py", line 
>>> 7, in 
>>> pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
>>>   File "/usr/local/lib/python2.7/dist-packages/geopandas/tools/sjoin.py", 
>>> line 57, in sjoin
>>> r_idx = np.concatenate(idxmatch.values)
>>> ValueError: need at least one array to concatenate
>>>
>>> and if i change the imports with the some code :
>>>
>>> import geopandas
>>> import pandas as pd
>>> import geopandas as gpd
>>> from geopandas import GeoDataFrame, read_file
>>> from geopandas.tools import sjoin
>>> from shapely.geometry import Point, mapping,shape
>>> import pandas as gpd
>>>
>>> i take that error :
>>>
>>> pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
>>> AttributeError: 'module' object has no attribute 'sjoin'
>>>
>>>
>>> any idea why ?
>>>
>>
>>
>> import geopandas as gpd
>> import pandas as gpd
>>
>> Does pandas have an attribute 'sjoin'?
>>
>> Duncan
> 
> i dont know i follow detais i am newbie
> 

You import geopandas as gpd, then import pandas as gpd. So maybe you
think gpd refers to geopandas while it actually refers to pandas. I
don't know geopandas or pandas, but you should check your imports.

Duncan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: geopandas bug ?

[Xristos Xristoou]

Τη Σάββατο, 14 Ιανουαρίου 2017 - 8:09:53 μ.μ. UTC+2, ο χρήστης duncan smith 
έγραψε:
> On 14/01/17 14:59, Xristos Xristoou wrote:
> > Τη Σάββατο, 14 Ιανουαρίου 2017 - 4:38:39 μ.μ. UTC+2, ο χρήστης duncan smith 
> > έγραψε:
> >> On 14/01/17 11:18, Xristos Xristoou wrote:
> >>> i want  to create a simple spatial joing using geopandas but i thing so 
> >>> geopandas has bug ?
> >>>
> >>>
> >>>
> >>> geopandas code :
> >>>
> >>> from geopandas import gpd
> >>> import geopandas
> >>> points = geopandas.GeoDataFrame.from_file('points.shp') # or geojson etc
> >>> polys = geopandas.GeoDataFrame.from_file('polygons.shp')
> >>> pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> >>>
> >>> error :
> >>>
> >>> Traceback (most recent call last):
> >>>   File "/home/sarantis/testshapely/sumpointsinsidepolygon/testgeo.py", 
> >>> line 7, in 
> >>> pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> >>>   File "/usr/local/lib/python2.7/dist-packages/geopandas/tools/sjoin.py", 
> >>> line 57, in sjoin
> >>> r_idx = np.concatenate(idxmatch.values)
> >>> ValueError: need at least one array to concatenate
> >>>
> >>> and if i change the imports with the some code :
> >>>
> >>> import geopandas
> >>> import pandas as pd
> >>> import geopandas as gpd
> >>> from geopandas import GeoDataFrame, read_file
> >>> from geopandas.tools import sjoin
> >>> from shapely.geometry import Point, mapping,shape
> >>> import pandas as gpd
> >>>
> >>> i take that error :
> >>>
> >>> pointInPoly = gpd.sjoin(points, polys, how='left',op='within')
> >>> AttributeError: 'module' object has no attribute 'sjoin'
> >>>
> >>>
> >>> any idea why ?
> >>>
> >>
> >>
> >> import geopandas as gpd
> >> import pandas as gpd
> >>
> >> Does pandas have an attribute 'sjoin'?
> >>
> >> Duncan
> > 
> > i dont know i follow detais i am newbie
> > 
> 
> You import geopandas as gpd, then import pandas as gpd. So maybe you
> think gpd refers to geopandas while it actually refers to pandas. I
> don't know geopandas or pandas, but you should check your imports.
> 
> Duncan

@Duncan i see that and i remove line with pandas ans no change
-- 
https://mail.python.org/mailman/listinfo/python-list

multiprocessing.Process call blocks other processes from running

[Rodrick Brown]

I'm trying to implement a script that tracks how much space certain
applications are using and send a metric to a statsd service for realtime
analysis however when running this code it nots forking multiple processes
its still running sequential at some point the forking was working but I
can't seem to figure out where I went wrong please advise

I'm just trying to keep track of the growth rate of certain dirs these dirs
have hundreds of thousands of files and take sometime to run I use du as it
runs much faster than os.walk() and it also returns the correct compressed
size on the file system pythons getsize() does not.

Thanks.

from datadog import initialize
from datadog import api
from datadog import statsd
import os
import subprocess
from import Process, Queue
from datadog import ThreadStats
import time
import datetime
from hashlib import md5

options = {
  'api_key': 'xx',
  'app_key': 'xx'
}

def getWhispererLogsDirSize(clientcfg, queue):
  clientName, logPath = clientcfg.items()[0]
  totalSize = 0
  clientResult = {}
  for item in os.listdir(logPath):
logDir = logPath + "/" + item
try:
  totalSize = totalSize +
int(subprocess.check_output(["du","-s",logDir]).split('\t')[0])
except subprocess.CalledProcessError:
  print("Error processing {0} skipping.".format(logDir))
  continue
  clientResult[clientName] = [os.path.basename(logPath),totalSize]
  queue.put(clientResult)
  return

if __name__ == '__main__':

  title = 'Whisperer client marketdata usage'
  text = 'This simple utility sends whisperer logs into datadog based on
client usage on disk'
  tags = ['version:1']
  initialize(**options)
  #api.Event.create(title=title, text=text, tags=tags)
  #stats = ThreadStats()
  #stats.start()
  queue = Queue()
  jobs = []
  clients = [
{'xx1':'/mnt/auto/glusterfs/app/NYC01-xx-PROD-01'},
{'xx2':'/mnt/auto/glusterfs/app/NYC01-xx-PROD-01'},
{'xx3':'/mnt/auto/glusterfs/app/NYC01-xx-PROD-01'},
{'xx4':'/mnt/auto/glusterfs/app/NYC01-xx-PROD-01'}
  ]
  tags = []
  while True:
for client in clients:
  stats = ThreadStats()
  stats.start()
  p = Process(target=getWhispererLogsDirSize, args=(client,queue,))
  jobs.append(p)
  p.start()
  p.join()
  clientinfo = queue.get()

  clientName = clientinfo.values()[0][0]
  clientPath = clientinfo.keys()[0]
  clientLogSize = clientinfo.values()[0][1]
  tags = [clientName,clientPath]
  aggregation_key = md5(clientName).hexdigest()

  print(clientName, clientPath, clientLogSize)
  with open('/tmp/dogstatd_out.log', 'a+') as fp:
fp.write("{0} {1} {2}
{3}\n".format(str(datetime.datetime.now()),clientName, clientPath,
clientLogSize))

stats.gauge('whisperer.marketdata.clientlogsize',int(clientLogSize),tags=tags)
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: multiprocessing.Process call blocks other processes from running

[Joseph L. Casale]

>  while True:
>for client in clients:
>  stats = ThreadStats()
>  stats.start()
>  p = Process(target=getWhispererLogsDirSize, args=(client,queue,))
>  jobs.append(p)
>  p.start()
>  p.join()

You start one client then join before starting the next...

Start them all and push the pointer the resulting process object into a 
collection.
Then use whatever semantics you desire to wait them all...

jlc
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing.Process call blocks other processes from running

[MRAB]

On 2017-01-14 19:05, Joseph L. Casale wrote:

 while True:
   for client in clients:
 stats = ThreadStats()
 stats.start()
 p = Process(target=getWhispererLogsDirSize, args=(client,queue,))
 jobs.append(p)
 p.start()
 p.join()


You start one client then join before starting the next...

Start them all and push the pointer the resulting process object into a 
collection.
Then use whatever semantics you desire to wait them all...


To me it also looks slightly odd that you're creating a ThreadStats
instance each time around the loop, and the only other reference to it
is to the last one created, and that line's indentation puts it outside
the """if __name__ == '__main__':""" block.

--
https://mail.python.org/mailman/listinfo/python-list

Re: How can I make a sentinel value NOT be initialized in a class/method - OOP?

[Erik]

[Replying to direct email. David - please respond to the list so that 
you can receive other people's suggestions also]


Hi David,

On 14/01/17 15:30, David D wrote:
> 1)  No this is not homework, I am doing this all on my own.

In that case I apologise - I gave you the benefit of the doubt though as 
I was only half-suspicious. FWIW, listing specific constructs that must 
be used is often a thing that a tutor would write in an assignment 
(because the purpose of the assignment is to get you to learn about 
those particular constructs). That's what raised the flag for me. Anyway ...


> 2) I am changing the class to cars.
>
> 3) A second question arose that I thought would work, but I am getting
> this error :*can't assign to a function call*
>
> What I am trying to do is this pseudo code :

Firstly, there's little point in posting pseudo code if you are 
reporting a specific compiler or run-time error. The errors you get are 
very specific to your code and in some cases are caused by *subtle* 
things in your code. We generally need to see the actual code (or a 
cut-down example) that allows the subtleties to be described to you.


> count= 0
>
> class car
> initialize all of the values and put self so they are available to the
> entire class
> def __init__(self, model...)
>   self.mode=model etc
>
>
> while loop
> input model =what is the model
> input serial = what is the serial
> input doors = how many doors
> count = count + 1
> #create the instance/object
> mycar (count) = car(model, serial, doors)
>
> input do you want another car in the database?
> if yes
>   continue
> if no
>   break
>
> The issue is that when creating an object/instance, python won't let me
> use this syntax of having -- car(count) which would give me multiple
> objects (car1, car2, car3 etc) with the count variable.  I am not 
sure why.


The syntax "foo(spam)" will *call* the function "foo" and pass it the 
value "spam". It doesn't make any sense to _assign a value_ to a 
function call operation (you're effectively trying to assign a value - 
the thing after the '=' - to another value - the return value of the 
function call).


So, what is "mycar"? How do you create it?

In Python, the built-in structure for a group of objects which can be 
dynamically extended with new entries in the way you want is a list. 
Create an empty one with:


mycar = []

or

mycar = list()

Lists have an 'append()' method which will add the parameter it is 
called with to the end of the list object it is called on:


mycar.append(car(model, serial, doors))


You don't have to worry about keeping track of 'count'. Lists will grow 
dynamically as you add things. Use "len(mycar)" to see how many items it 
holds and "for vehicle in mycar:" to step through them all or "mycar[n]" 
to address as specific entry.


From the Python prompt, type "help(list)"

Hope that helps.
E.
--
https://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing.Process call blocks other processes from running

[Steve D'Aprano]

On Sun, 15 Jan 2017 05:46 am, Rodrick Brown wrote:

> at some point the forking was working

Then whatever you changed, you should change back to the way it was.

That's the most important lesson here: never make two or more unrelated
changes to a program unless you have a backup of the working file.

An excellent way to manage this process is by using a revision control
system like Mercurial (hg) or equivalent. But at the very least, whenever
you change a working program, you should confirm it is still working before
making the next change.





-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list