RE: [Arches] Arches 4: ElasticSearch

[Van Daele, Koen]

It might be good to point out that - as Alexei mentioned - opening up your 
Elasticsearch instance to the world is a security breach and can get you hit by 
ransomware attacks. As has happened recently with elasticsearch instances 
around the world: 
http://www.zdnet.com/article/elasticsearch-ransomware-attacks-now-number-in-the-thousands/
 So if you do change the default setup, you need to make sure only Arches can 
to talk to Elasticsearch (or you need to setup a good security system in front 
of Elasticsearch).

Cheers,

Koen

Van Daele Koen

Applicatiebeheerder


Agentschap Onroerend Erfgoed

T 02 553 16 82, M 0499 94 93 68


Van: archesproject@googlegroups.com <archesproject@googlegroups.com> namens 
Adam Cox <mr.adam...@gmail.com>
Verzonden: dinsdag 31 januari 2017 21:52
Aan: Vincent Meijer
CC: Arches Project
Onderwerp: Re: [Arches] Arches 4: ElasticSearch

Hey Vincent, glad you brought this up. That graphic does more to show the 
resulting system architecture from the commands in the dev installation than it 
does to show the possible ways Arches can be setup. So, as Alexei mentioned it 
is a little misleading: Arches need not be setup with elasticsearch inside, but 
certain install steps allow that to happen out of convenience. I'll fix some of 
the text to say that. As mentioned in the note above the graphic, final 
distribution architecture isn't set yet anyway.

Glad we can work these wrinkles while drafting the documentation!

Adam

On Tue, Jan 31, 2017 at 12:25 PM, Vincent Meijer 
<meijer.vinc...@gmail.com<mailto:meijer.vinc...@gmail.com>> wrote:
Got it, thanks!

On Tuesday, 31 January 2017 13:19:46 UTC-5, Alexei Peters wrote:
Does it mean that external apps can query ElasticSearch only through the Arches 
app?

Essentially, yes, although you can open up the port to access ES directly, but 
I would consider that a security breach.


Director of Web Development - Farallon Geographics, Inc. - 
971.227.3173<tel:(971)%20227-3173>

On Tue, Jan 31, 2017 at 10:14 AM, Vincent Meijer <meijer@gmail.com> wrote:
Thanks for explaining! Could you please tell a little bit more about what it 
means that the Arches app acts as a proxy to ElasticSearch?
Does it mean that external apps can query ElasticSearch only through the Arches 
app?

Vincent

On Tuesday, 31 January 2017 12:44:33 UTC-5, Alexei Peters wrote:
Hi Vincent,
Good question!  This graphic is actually a little misleading and should be 
changed to show Arches app acting as a proxy to to the Elasticsearch server.
Elasticsearch runs as a separate service and additional instances of 
Elasticsearch can be setup on additional machines if needed.
Cheers,
Alexei


Director of Web Development - Farallon Geographics, Inc. - 971.227.3173

On Tue, Jan 31, 2017 at 8:28 AM, Vincent Meijer <meijer@gmail.com> wrote:
In Arches 4 the ElasticSearch server is incorporated inside Arches itself:
https://github.com/archesproject/arches/wiki/Dev-Installation#system-architecture

I was wondering how this works and what the idea behind this is. It seems odd 
to me.
E.g. How would it work if you want to run a cluster of Arches instances?

Thanks!
Vincent

--
-- To post, send email to arches...@googlegroups.com. To unsubscribe, send 
email to archesprojec...@googlegroups.com. For more information, visit 
https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesprojec...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
-- To post, send email to arches...@googlegroups.com. To unsubscribe, send 
email to archesprojec...@googlegroups.com. For more information, visit 
https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesprojec...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
-- To post, send email to 
archesproject@googlegroups.com<mailto:archesproject@googlegroups.com>. To 
unsubscribe, send email to 
archesproject+unsubscr...@googlegroups.com<mailto:archesproject%2bunsubscr...@googlegroups.com>.
 For more information, visit 
https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
archesproject+unsubscr...@googlegroups.com<mailto:archesproject+unsubscr...@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout.


--
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, sen

RE: [Arches] How to integrate Built Heritage, Immoveable Collections, and Moveable Collections

[Van Daele, Koen]

Hello Allison,

I would certainly recommend looking further into Collective Access (CA) 
(http://www.collectiveaccess.org/) It’s a project that has been around for 
years, but still seems to be going strong. I’ve met some of the people behind 
the project. Very open and friendly and knowledgeable. What was nice was that 
they also had archivists and historians on staff, so they could approach a 
project from the technical and the business side. The software itself is very 
flexible and can be customised heavily. Which is an advantage and a 
disadvantage at the same time. Depending on who you are, what your technical 
skills are and how much you're willing to invest.

I believe Omeka (http://omeka.org/) is somewhat similar in some aspects, 
although it is supposed to be less flexible and less customisable. But much 
simpeler to get up and running. Finally there's also Collection Space 
(http://www.collectionspace.org/). I assume 
that one is a bit closer to Collective Access, but maybe that's just because of 
the names. I do think Collective Access is the most flexible of them all and 
based on my limited experience it has the bigger user base.
Cheers,
Koen

Van: archesproject@googlegroups.com [mailto:archesproject@googlegroups.com] 
Namens Allison Lee
Verzonden: maandag 14 november 2016 16:34
Aan: Arches Project 
Onderwerp: Re: [Arches] How to integrate Built Heritage, Immoveable 
Collections, and Moveable Collections

Hello David,

Thank you for the detailed response about bringing these documentation 
objectives together into one place.  We have reviewed a bit of Collective 
Access, and it does seem to be a good option for storing our collection data 
and linking it to the main Arches site.  At the moment, developing new modules 
may be too demanding for our limited resources but we will revisit this when v4 
is released.

If anyone has other suggestions for moveable collections, or any advice 
regarding Collective Access, please do let us know.  We don't have experience 
with Collective Access, but it does appear very capable for extensive component 
collections such as ours.

Best,
Allison
--
-- To post, send email to 
archesproject@googlegroups.com. To 
unsubscribe, send email to 
archesproject+unsubscr...@googlegroups.com.
 For more information, visit 
https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [Arches] Command "python setup.py egg_info" failed with error code 1

[Van Daele, Koen]

Hi,

I think that error might indicate you are not using python 2.7. It looks like a 
typical error when running Python 2 code on Python 3. Arches does NOT work with 
anything else than Python 2.7.x Please check the installation instructions and 
make sure you are following them to the letter.

To receive further help with your issue, please provide a more detailed error 
report. What system are you installing on (Looks like a *nix system), what 
version of pip are you using (see Adam's answer), what Python version are you 
using, what steps in the installation instruction have you followed and what 
output did you get along the way?

Cheers,

Koen


Van Daele Koen

Applicatiebeheerder


Agentschap Onroerend Erfgoed

T 02 553 16 82, M 0499 94 93 68


Van: archesproject@googlegroups.com <archesproject@googlegroups.com> namens 
Ajay Mahajan <ajay.maha...@enhancelabs.co.in>
Verzonden: zondag 3 april 2016 13:02
Aan: Arches Project
Onderwerp: Re: [Arches] Command "python setup.py egg_info" failed with error 
code 1

Thanks for reply, but i have tried the same before and the result is in this 
way as given below:


Collecting arches-hip
  Using cached arches_hip-1.0.4.tar.gz
Collecting arches>=3.1.0 (from arches-hip)
  Using cached arches-3.1.2.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
  File "", line 1, in 
  File "/tmp/pip-build-ubsl7_6s/arches/setup.py", line 14, in 
version = __import__('arches').__version__
  File "/tmp/pip-build-ubsl7_6s/arches/arches/__init__.py", line 1, in 

from arches.setup import get_version
  File "/tmp/pip-build-ubsl7_6s/arches/arches/setup.py", line 124
print "Downloading: %s Bytes: %s" % (file_name, file_size)
^
SyntaxError: invalid syntax


Command "python setup.py egg_info" failed with error code 1 in 
/tmp/pip-build-ubsl7_6s/arches/



so request you to please check where i am lacking in this code.


On Sunday, April 3, 2016 at 4:02:03 PM UTC+5:30, Koen Van Daele wrote:

According to https://pypi.python.org/pypi/arches/3.1.2 , Arches only runs on 
Python 2.7. So, you would first need to upgrade to python 2.7. It doesn't run 
on Python 3.x (yet), so don't go to a version higher than 2.7.

Bear in mind that Python 2.6 is very old, so if that's the default version on 
you computer system you probably have a really old system and upgrading that 
might be necessary.


Koen


Van Daele Koen

Applicatiebeheerder


Agentschap Onroerend Erfgoed

T 02 553 16 82, M 0499 94 93 68


Van: arches...@googlegroups.com <arches...@googlegroups.com> namens Ajay 
Mahajan <ajay.m...@enhancelabs.co.in>
Verzonden: zondag 3 april 2016 10:58
Aan: Arches Project
Onderwerp: Re: [Arches] Command "python setup.py egg_info" failed with error 
code 1

Actully  i have installed python 2.6 and now want to install arches. just 
confues how to place package and run arches app.

just need steps to install and run arches hip project

On Saturday, April 2, 2016 at 4:28:16 AM UTC+5:30, Adam Cox wrote:
Hi there, it would be easier to troubleshoot if you could provide some more 
information about your system environment.  Do you have a virtual environment 
active?  If so, I'd recommend that you try upgrading your version of pip, as I 
recently described in this 
post<https://groups.google.com/forum/#!topic/archesproject/0S8M8s7n4vo>. No 
idea if that's what your problem is, but it may be a good place to start.

Adam

On Fri, Apr 1, 2016 at 3:00 PM, <in...@tangytummy.in> wrote:
Hello there,

While installing the pacages facing this issue:

Collecting arches-hip
  Using cached arches_hip-1.0.4.tar.gz
Collecting arches>=3.1.0 (from arches-hip)
  Using cached arches-3.1.2.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
  File "", line 1, in 
  File "/tmp/pip-build-ubsl7_6s/arches/setup.py", line 14, in 
version = __import__('arches').__version__
  File "/tmp/pip-build-ubsl7_6s/arches/arches/__init__.py", line 1, in 

from arches.setup import get_version
  File "/tmp/pip-build-ubsl7_6s/arches/arches/setup.py", line 124
print "Downloading: %s Bytes: %s" % (file_name, file_size)
^
SyntaxError: invalid syntax


Command "python setup.py egg_info" failed with error code 1 in 
/tmp/pip-build-ubsl7_6s/arches/


is there any solution for this.

as i am first time using this arches. so please be clear with answer

--
-- To post, send email to arches...@googlegroups.com. To unsubscribe, send 
email to archesprojec

RE: [Arches] Command "python setup.py egg_info" failed with error code 1

[Van Daele, Koen]

According to https://pypi.python.org/pypi/arches/3.1.2 , Arches only runs on 
Python 2.7. So, you would first need to upgrade to python 2.7. It doesn't run 
on Python 3.x (yet), so don't go to a version higher than 2.7.

Bear in mind that Python 2.6 is very old, so if that's the default version on 
you computer system you probably have a really old system and upgrading that 
might be necessary.


Koen


Van Daele Koen

Applicatiebeheerder


Agentschap Onroerend Erfgoed

T 02 553 16 82, M 0499 94 93 68


Van: archesproject@googlegroups.com <archesproject@googlegroups.com> namens 
Ajay Mahajan <ajay.maha...@enhancelabs.co.in>
Verzonden: zondag 3 april 2016 10:58
Aan: Arches Project
Onderwerp: Re: [Arches] Command "python setup.py egg_info" failed with error 
code 1

Actully  i have installed python 2.6 and now want to install arches. just 
confues how to place package and run arches app.

just need steps to install and run arches hip project

On Saturday, April 2, 2016 at 4:28:16 AM UTC+5:30, Adam Cox wrote:
Hi there, it would be easier to troubleshoot if you could provide some more 
information about your system environment.  Do you have a virtual environment 
active?  If so, I'd recommend that you try upgrading your version of pip, as I 
recently described in this 
post<https://groups.google.com/forum/#!topic/archesproject/0S8M8s7n4vo>. No 
idea if that's what your problem is, but it may be a good place to start.

Adam

On Fri, Apr 1, 2016 at 3:00 PM, <in...@tangytummy.in> wrote:
Hello there,

While installing the pacages facing this issue:

Collecting arches-hip
  Using cached arches_hip-1.0.4.tar.gz
Collecting arches>=3.1.0 (from arches-hip)
  Using cached arches-3.1.2.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
  File "", line 1, in 
  File "/tmp/pip-build-ubsl7_6s/arches/setup.py", line 14, in 
version = __import__('arches').__version__
  File "/tmp/pip-build-ubsl7_6s/arches/arches/__init__.py", line 1, in 

from arches.setup import get_version
  File "/tmp/pip-build-ubsl7_6s/arches/arches/setup.py", line 124
print "Downloading: %s Bytes: %s" % (file_name, file_size)
^
SyntaxError: invalid syntax


Command "python setup.py egg_info" failed with error code 1 in 
/tmp/pip-build-ubsl7_6s/arches/


is there any solution for this.

as i am first time using this arches. so please be clear with answer

--
-- To post, send email to arches...@googlegroups.com. To unsubscribe, send 
email to archesprojec...@googlegroups.com. For more information, visit 
https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesprojec...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
archesproject+unsubscr...@googlegroups.com<mailto:archesproject+unsubscr...@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout.

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [Arches] help with long-term Ubuntu 14.04 server maintenance?

[Van Daele, Koen]

Hi Adam,

A few tips:

 * Stick with an Ubuntu LTS until the next one comes along.
 * Most of the updates that happen during an Ubuntu LTS cycle are bug-fixes and 
security fixes. I have never seen them deliver backwards incompatible features 
or fixes. So, running apt-get update and apt-get upgrade is fine. Think we 
generally use aptitude instead of apt-get but I forget why that is at the 
moment.
 * As long as you stay within a certain Ubuntu release, your dependencies (eg. 
python and postgres) will stay at the same major versions and only receive 
fixes. Going to a newer Ubuntu release runs the risk of getting you a newer 
Python version. I believe for Ubuntu 16.04 they want to make Python 3.5 the 
default Python. Last time I checked Arches wasn't py3 compatible (might have 
changed).
 * I would not run apt-get dist-upgrade on a production server. And certainly 
not to go from 14.04 to eg. 14.10. I have done this for desktops, but never for 
servers. Something always seems to break somewhere.
 * ​When there's a new LTS and we're satisfied it's stable (after a few 
months), we basically wipe the server and reinstall from scratch.
To make this an easy task (and because we load-balance), we script everything. 
We use fabric (http://www.fabfile.org/) for this, but you could do the same 
with something like Ansible, Chef, Puppet, ... Our fabfile has a set of tasks 
that update stuff, configure apache, build packages and configuration files and 
deploy the results. So, once we have wiped a server it's generally a matter of 
reinstalling Ubuntu, running fab  update_ubuntu and fab  deploy. We 
keeps these fab files in version control as well and they server as excellent 
documentation on how to set up a certain application.
 * Bear in mind that scripting makes it easy to reinstall, but it also makes it 
easy to blow things to bits. We have a development, test and production 
environment. Needless to say, we always test the scripts before executing them 
in production.
 * On more thing, we have quite a large datacenter at our disposal, with a lot 
of virtualised servers. We never run database servers on webservers. Upgrading 
a database is always a tricky thing and might require dumping your entire 
database and reload (generally when upgrading major postgres versions). Same 
thing for elasticsearch, we also run those as separate clusters. Setting up 
everything on different machine is a lot more work, but it does offer nice 
possibilities when scheduling maintenance.

Let me know if you have any further questions.

Cheers,
Koen





Van: archesproject@googlegroups.com  namens 
Adam Cox 
Verzonden: maandag 21 september 2015 16:59
Aan: Arches Project
Onderwerp: [Arches] help with long-term Ubuntu 14.04 server maintenance?

Hello all, I'm wondering if anyone could help suggest the best way to handle a 
server that is only being used for arches, when it comes to package 
updates/upgrades?

I only know the very basics of using apt-get update and apt-get upgrade, and 
apt-get dist-upgrade.  I don't want any of the Arches dependencies to be 
upgraded, but I do need to be able to get current security updates and make 
sure things keep running smoothly.

I'll look into these suggestions 
http://askubuntu.com/questions/194/how-can-i-install-just-security-updates-from-the-command-line,
 but would be interested to know if anyone has experience with maintaining a 
server over a long period of time.

Adam

--
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [Arches] ElasticSearch in Production

[Van Daele, Koen]

Hi Adam,

not sure about 1. Generally the ES development team gives ample warning when 
they make backwards incompatible fixes.

From what I can see, most of the settings in the elasticsearch.yml file make 
sure that ES is running locally and with only one leg. ES is a very powerful 
system, but to use it to the fullest of it's capabilities, you will need to 
learn how to configure it a bit. We eg. have several ES nodes running (on 
different machines, separated froom the webservers) as a high availability 
cluster. That does involve some more configuration work. One thing to be wary 
of, with the default ES config (unless that has been changed by now), the 
nodes have a tendency to go and look for each other. I remember when my ES 
instance started syncing with a colleagues (without us realising). Easiest way 
to prevent that is to give your cluster a dedicated name (cluster.name in the 
config).

You might also want to install the elasticsearch-head plugin that Arches also 
installs. I don't think any other plugins are needed at this moment.

Anyway, in my opinion, running ES from within Arches is fine for development 
purposes. But for production I'd prefer to separate the two.

Cheers,
Koen

Van: archesproject@googlegroups.com [archesproject@googlegroups.com] namens 
Adam Cox [mr.adam...@gmail.com]
Verzonden: maandag 11 mei 2015 18:39
Aan: archesproject@googlegroups.com
Onderwerp: [Arches] ElasticSearch in Production

Hello all.  I have successfully installed ElasticSearch on Ubuntu outside of 
the arches environment so it will run as a service, and it was far easier than 
I was expecting... a pleasant surprise that makes me wary.

To do so I began with a fresh Amazon EC2 instance, and I followed the normal 
arches3 installation steps.  However, instead of running

python manage.py packages -o setup_elasticsearch

(which installs elasticsearch within the app itself) I followed these 
stepshttp://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html
 from the ElasticSearch webpage to install the current APT distribution of 
ElasticSearch on Ubuntu.  After that installation, I was able to run ES as a 
service, and after finishing the arches-hip installation, was able to use the 
database and add/search resources (it works!).

I have two questions:
1. I see the current ES repo is for version 1.5 instead of 1.4 which is used by 
arches3.   Will this cause problems later?  I assume I could install 1.4 
instead of 1.5 if necessary.
2. I have not changed any of the default elasticsearch config settings.  I see 
that the arches3 development installation has its own custom elasticsearch.yml 
file.  Would it be advisable to change any default elasticsearch config 
settings on this installation to match those in the arches3 elasticsearch.yml 
file?


--
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups 
Arches Project group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
archesproject+unsubscr...@googlegroups.commailto:archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Arches Project group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [Arches] Re: Arches 3.0 released!

[Van Daele, Koen]

Hi Adam,

running ElasticSearch as a daemon is normally very straightforward. Just follow 
instructions like these: 
https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04
 and then you can control it with:

$ sudo service elasticsearch start
$ sudo service elasticsearch stop
$ sudo service elasticsearch restart

It will automatically run, even after a reboot of your server.

Cheers,
Koen

Van: archesproject@googlegroups.com [archesproject@googlegroups.com] namens 
Adam Cox [mr.adam...@gmail.com]
Verzonden: woensdag 6 mei 2015 16:36
Aan: David Lopes
CC: archesproject@googlegroups.com
Onderwerp: Re: [Arches] Re: Arches 3.0 released!

Hi David, I also first tried installing those as the root user.  Not sure if 
that's exactly what you mean by root privileges, or if you just mean with sudo, 
but when I did so as the root user (su root), the installation worked but I 
also had to run later commands from the root user account, and felt that that 
was not a great precedence to set.  It seemed more prudent to do everything as 
the ubuntu user, hence the addition of the sudo commands to the 
ubuntu_trusty_setup.sh script.

As for setting up with Apache, here are the steps that I recorded when I made a 
successful test of that process:

1. get apache2 and the mod
 -- $: sudo apt-get install apache2
 -- $: sudo apt-get install libapache2-mod-wsgi

2. edit main apache2 config file
 -- $: sudo nano /etc/apache2/apache2.conf

add the top arches app directory (crip is the name of my app):

Directory /home/ubuntu/Projects/crip/
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
/Directory

3. edit the sites-enabled conf file:
 -- $: sudo nano /etc/apache2/sites-enabled/000-default

under VirtualHost *:80 add:

#create daemon process: make path to app and to virtualenv python
WSGIDaemonProcess arches 
python-path=/home/ubuntu/Projects/crip:/home/ubuntu/Projects/ENV/lib/python2.7/site-packages

#make group for app
WSGIProcessGroup arches

#path to app wsgi.py file
WSGIScriptAlias / /home/ubuntu/Projects/crip/wsgi.py

4. restart apache
 -- $: sudo service apache2 restart

reference: 
https://www.digitalocean.com/community/tutorials/how-to-run-django-with-mod_wsgi-and-apache-with-a-virtualenv-python-environment-on-a-debian-vps

Hope that works for you!

I'm also hoping to set up elasticsearch as a daemon process so that it will run 
without having a terminal open, but have not attempted that yet.  This is where 
I'm going to start with that once I'm able to get to it: 
http://wrapper.tanukisoftware.com/doc/english/launch-nix.html




On Wed, May 6, 2015 at 3:02 AM, David Lopes 
cim...@gmail.commailto:cim...@gmail.com wrote:
Hi Adam,

Thank you for your quick answer. I will have an in depth review it in a moment, 
although I think some of the problems I had are now solved simply by 
uninstalling with synaptic the apps mentioned in the ubuntu_trusty_setup.sh 
script and then running that same script in a terminal with root privileges.

My main problem now, since http://localhost:8000 is running (though not loading 
all the resources of arches_hip - the command line refers them as truncated), 
is dealing with the setup of apache for production, namely with the wgsi.py 
file config. What do you have to say about that?

Thank you so much, best regards.

David Lopes



terça-feira, 5 de Maio de 2015 às 17:33:11 UTC+1, Adam Cox escreveu:
David, I was having the same issue for a little while.  It stemmed from not 
having postgres install correctly during the ubuntu installation (I also had 
the postgis default name set incorrectly, but I doubt that is an issue for you).

Correctly installed postgres:
Did you use the ubuntu_trusty_setup.sh script for setup?  I found an issue with 
that script and I'm not sure that it has been resolved yet.  There are two tar 
commands that are used to unpack the postgis installation, and they need to 
have sudo added to them.  Without sudo, postgis is not unpacked correctly 
(permissions issues), and so your postgres database does not have spatial 
abilities, e.g. no geometry tables.  As you see, you can see the database, but 
you'll  there are no spatial abilities.

Here's a version of the ubuntu_trusty_setup.sh with those sudo commands added, 
http://www.adamcfcox.com/ubuntu_trusty_setup.sh, or you can just modify the 
script that you have downloaded.  Rerun that install script and see if that 
solves your issue.

OR, if you are using AWS EC2 services, you can find the AMI that I put up with 
all of the dependencies already installed.  (If you do that, I'd like to know 
if it works for you!)

As for using Apache2, I have successfully deployed with Apache2, but that was 
after solving the issue above, so hopefully it'll get working after that.

I see your other question about the settings.py files.  No, those files don't 
have to have the same values, 

RE: [Arches] Re: Dealing with Audio and Visual entries in the database

[Van Daele, Koen]

Hi Adam,

I do believe ExtJs has been replaced by jQuery in v3. Nowadays, there's also 
the HTML5 video tag that should work for most browsers. I think that adding a 
JS library like this: http://www.videojs.com/ will also help you in catering to 
older browsers.

So, I think you should just be able to create a custom report that loads some 
extra javascript files and uses the video tag to point at your resource. Just 
how easy or hard that would be is a question best answered from somebody at 
Farallon. But I do believe they're currently working very hard at technical 
stuff, so it might take them a while to get back to you.

Cheers,
Koen

Van: archesproject@googlegroups.com [archesproject@googlegroups.com] namens 
Adam Cox [mr.adam...@gmail.com]
Verzonden: donderdag 19 februari 2015 23:53
Aan: archesproject@googlegroups.com
Onderwerp: [Arches] Re: Dealing with Audio and Visual entries in the database

Ok, been looking around a bit, and per the roadmap it seems like the adoption 
of jQuery and Bootstrap in Arches v3 would indicate a replacement of ExtJS.  I 
have found simple examples of videos embedded in bootstrap modals, so I feel 
that this whole concept is possible... Still, any feedback or suggestions would 
be welcome.

Adam

On Wednesday, February 18, 2015 at 11:50:40 AM UTC-6, Adam Cox wrote:
Hello all, I'm doing some planning would be interested for any input on this 
issue.

One thing that the Cane River National Heritage Area is very interested in is 
using Arches as a way to store and catalogue their collection of oral histories 
and other interviews/recordings from around this area.  One way I had 
envisioned this was uploading videos and audio files as Information Resources, 
similar to a pdf or photo.  I figured that they would at the very least be 
downloadable by anyone viewing the database, and most browsers seem to play 
.mp3 files without needing to download them so if someone found an audio file 
in the database, the report for that entry would just have a link to the file 
itself.

I've been looking at ExtJS and I haven't found there to be a good up-to-date 
extension that would support an embedded video in the report for a resource.  
So, I was thinking maybe all of the videos could be uploaded to YouTube 
instead, and then the Arches resource report for that video would have an 
embedded YouTube video (html iframe element).  I think the workflow could be 
along these lines

1. upload the video to YouTube and copy the video ID from the YouTube watch 
link (e.g. vAOVRkSCWmg)
2. Create new Arches resource for the video, and store the video ID somewhere 
within the resource graph
3. construct the form for video Information Resources pull the video ID and 
construct the YouTube embed link.
4. Optionally, you could go back to the YouTube video and add description with 
some way of linking to the Arches resource report (maybe)

This could be done with the .mp3 files as well, I suppose, but that would 
require some conversion and prior editing.

There are some good upsides to this, like not having to store and use your own 
server for people streaming video.  Also, people could find more videos on 
YouTube and then be referred to the Arches database that way.

Anyway, any feedback on this strategy would be much appreciated.  I haven't 
gotten my feet wet with ExtJS yet, so there may be some issues that would not 
make this possible...

Thanks,
Adam

--
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups 
Arches Project group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
archesproject+unsubscr...@googlegroups.commailto:archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Arches Project group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [Arches] Re: Installing GeoServer before installing Arches v3

[Van Daele, Koen]

Hi Adam,

Geoserver is not needed to install Arches. You only need it if you want to add 
extra basemaps or overlays to Arches. Basically, for any spatial layer that's 
not being managed by Arches itself you need some external source. This can be 
one of several formats: WMS (Web Map Service), WFS (Web Feature Service), TMS 
(Tile Map Service). Over here in Europe we nowadays have quite a lot of public 
servers delivering basemaps because of the INSPIRE regulations. If the basemaps 
you need are not available as a service, you need to set it up yourself. There 
are several possible solutions for that: Geoserver, Mapserver, ESRI software or 
anything else that can serve up a WMS.

At work we use Geoserver (https://geo.onroerenderfgoed.be/geoserver) for 
serving up our own WMS/WFS layers from our Postgis databases. It's easy to set 
up and maintain. We use it in our own Geoportal 
(https://geo.onroerenderfgoed.be) that was built with some of the same 
technologies as Arches, most notably OpenLayers for interacting with maps. 

Although from the outside world they appear to be on the same servers, our 
Geoportal and Geoserver are actually on different servers. We just serve them 
through some proxies on one host. There are several reasons for this:

 * Our Geoportal (as well as Arches) runs on Python. We serve it with apache 
and mod_wsgi. Geoserver is a Java application that runs in a Java webserver 
like Tomcat. Depending on the machine you're using they might come into 
conflict when it comes to system resources. If you run them on one machine, you 
need to make sure that each has enough resources available and is not able to 
cause problems for the other one.
 * If you're running both on the same machine, you would need to run them on 
different ports. For eg., you could run Apache/Arches on port 80 (the default 
for a webserver) and Tomcat/Geoserver on port 8080. Depending on where and how 
you're hosting that might be a problem. If it is, you could set up an apache 
with mod_proxy so that eg. traffic from / geoserver is proxied to port 8080.
 * Having different servers for each makes it easier to scale our 
infrastructure and maintain a high availability cluster.

So, you only need Geoserver if you want to use additional basemaps in Arches. 
You can run both Arches and Geoserver on one machine, provided your machine has 
enough resources and set them up on different ports.

Hope this helps a bit. Let me know if you have any further questions.

Cheers,
Koen

Van: archesproject@googlegroups.com [archesproject@googlegroups.com] namens 
Adam Cox [mr.adam...@gmail.com]
Verzonden: dinsdag 10 februari 2015 21:31
Aan: archesproject@googlegroups.com
Onderwerp: [Arches] Re: Installing GeoServer before installing Arches v3

Ah, looks like GeoServer more or less equals GEOS, which is mentioned as a 
dependency in the earlier documentation... still unclear on the exact 
relationship though.

On Tuesday, February 10, 2015 at 2:26:05 PM UTC-6, Adam Cox wrote:
Hello all.  I'm wondering if there would be any issue with putting GeoServer on 
my server before running the install for Arches v3.  I thought I recalled that 
Arches used GeoServer, but looking back through documentation, I don't actually 
see mention of it.  I have been developing a number of extra overlay/basemap 
layers that I plan to include in my Arches installation, and at this point I'm 
been staging them in GeoServer on an Amazon EC2 instance.  I have our future 
permanent server for Arches now in place, so I'm ready to begin with the Arches 
installation once v3 is released, but I'm wondering if in the meantime there 
would be any problem with setting up GeoServer.

Also wondering if the late Feb. release date is still on schedule.

Thanks,
Adam

--
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups 
Arches Project group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
archesproject+unsubscr...@googlegroups.commailto:archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Arches Project group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [Arches] Using Docker for deploying Arches

[Van Daele, Koen]

Hi Rita,

welcome to the Arches project.

I'm aware about what Docker is. It does seems like it might be the way of the 
future for easier installations. But I've never used it myself. Do you have a 
lot of experience with it? Suggestions for the team if they want to move in 
that direction? Or do you want to do contribute in making this happen?

Farallon are currently finishing up Arches 3, so it would probably be a thing 
for that release, not for the older ones.

Cheers,
Koen

Van: archesproject@googlegroups.com [archesproject@googlegroups.com] namens 
Rita Sinha [rita.sinh...@gmail.com]
Verzonden: donderdag 8 januari 2015 14:53
Aan: archesproject@googlegroups.com
Onderwerp: [Arches] Using Docker for deploying Arches

Hi,

 I have been recently lurking around open source projects when I came across  
The Arches Project and developed keen interest in contributing to the org.

After going through the project, I'm eager to improve the existing installation 
of Arches.

After going through the current installation guide, I would like to suggest 
using Docker to deploy Arches.

For those not familiar with docker, can take the reference from here-

http://en.wikipedia.org/wiki/Docker_(software)

Regards,
Rita Sinha

--
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups 
Arches Project group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
archesproject+unsubscr...@googlegroups.commailto:archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Arches Project group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [Arches] Arches documentation has a new home!

[Van Daele, Koen]

Hi Yannis,

I think somebody already beat you to that: http://johnmacfarlane.net/pandoc/

Cheers,
Koen

Koen Van Daele
Applicatiebeheerder
M +32 (0)499 94 93 68  |  T +32 (0)2 553 16 82

Van: archesproject@googlegroups.com [mailto:archesproject@googlegroups.com] 
Namens Yiannis Avramides
Verzonden: donderdag 31 juli 2014 16:22
Aan: archesproject@googlegroups.com
CC: kfo...@opentechstrategies.com
Onderwerp: Re: [Arches] Arches documentation has a new home!

We only have the Installation Guide up on ReadTheDocs, thanks to Alexei. One of 
these days I'll learn reStructuredText and I'll transfer the User Guide there 
too (or I'll learn to code and I'll create a Markdown to reStructuredText 
converter, but that might take a little longer). If anyone else would like to 
take that on, please reply to this thread!

Cheers,

Yiannis

On Wednesday, July 30, 2014 9:21:08 PM UTC-4, Karl Fogel wrote:
Alexei Peters apet...@fargeo.commailto:apet...@fargeo.com writes:
The Arches team has been busy over the last few days moving our
documentation online.

You can now access it at http://arches.readthedocs.org/

By the way, at http://archesproject.org/documentation/, the link to the
Installation Guide v2.0 is to ReadTheDocs, but below it the link to the
Arches User Guide is still to leanpub.comhttp://leanpub.com.  Is that 
intentional?

Best,
-Karl
--
-- To post, send email to 
archesproject@googlegroups.commailto:archesproject@googlegroups.com. To 
unsubscribe, send email to 
archesproject+unsubscr...@googlegroups.commailto:archesproject+unsubscr...@googlegroups.com.
 For more information, visit 
https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups 
Arches Project group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
archesproject+unsubscr...@googlegroups.commailto:archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Arches Project group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [Arches] uncertain information in Arches

[Van Daele, Koen]

Hi all,

I just wanted to get back at what Dennis said at the beginning of this thread. 
Im quite curious how you will get people to agree on (un)certainty. If feels 
like a very natural idea to talk and think about, but I haven't really seen it 
function properly in practice.

We once did an experiment where we had 10 people who were used to entering data 
in our archaeological inventory system enter the same site. We paired the 
archaeologists: one more more experienced data entry person (a few years 
experience) and one newbie (a few months), so they would be forced to really 
think things through and discuss. In our database we have a field for certain 
the data entry person is about the location of the site, ie. about the polygon 
they might have drawn on a map. This field only allowed 5 choices, ranging from 
1 (I'm sure it's exactly where it needs to be) to 5 (I have no idea whatsoever 
where the site is). We had a very detailed manual with examples of all these 
cases, what to use when, ...
Final result of our experiment: every group had entered the location with a 
different level of certainty. So, based on the exact same information they had 
all drawn totally different conclusions. And this was about something as simple 
as the location of the site.

So, I'm very curious about how you manage to prevent stuff like this from 
happening.

The other thing I wonder about: how does certainty affect searching? Should a 
search for 'churches' only return sites that have a certain certainty 
attached to the interpretation? Are you working with sliding scale of certainty 
(ie. we are 75% percent certain about this statement) or a binary one (we're 
certain or uncertain)?

Cheers,
Koen

Van: archesproject@googlegroups.com [archesproject@googlegroups.com] namens 
dwuthr...@fargeo.com [dwuthr...@fargeo.com]
Verzonden: donderdag 27 maart 2014 22:36
Aan: thomas.enge...@gmail.com
CC: archesproject@googlegroups.com
Onderwerp: Re: [Arches] uncertain information in Arches

Thomas,

Good question!  You are quite correct that we haven't tried to include 
uncertainty in Arches.

One reason is pretty basic: certainty is quite subjective from person to 
person.  For example: most people agree that the earth is spherical.  But a 
flat-earther may be very certain that the earth is not a sphere, but is 
instead a plane.  His certainty does not make him correct, it merely states the 
degree to which he believes in his interpretation.  Clearly, you can be very 
certain and very wrong at the same time.  I guess my point is that in many 
cases certainty says more about the person making the assertion than it does 
about the thing being described.

OK, all philosophy aside, one could easily extend any Arches graph to include a 
certainty node.  Such a node could point to a thesaurus (as many of the nodes 
in Arches already do), allowing a user to select from a list of uncertainty 
levels.  Really, any Arches graph could include a certainty node under any 
entity that you might want to qualify (for example, one certainty node for 
period and another certainty node for heritage type).

Really, the hard part is not in getting Arches to allow you to add an 
uncertainty level to your cultural heritage data.  Rather, the difficult 
thing is to decide how you'll get different people to agree on what constitutes 
certain vs. uncertain interpretations of heritage.

Sorry that I can't be any more helpful... However, I'm very interested to hear 
how you will model uncertainty and how you will get people to implement it 
consistently.  Please keep me posted!

Cheers,

Dennis


On Mar 27, 2014, at 2:41, 
thomas.enge...@gmail.commailto:thomas.enge...@gmail.com wrote:

I have a question about conceptual modeling in CIDOC CRM, maybe there is 
someone one the list who is able to provide some guidance.

As posted before, we are trying to integrate research data of neolithic sites 
into Arches. Now, naturally a significant part of this data has a level of 
certainty to which the information is correct. e.g. a site can consist of 
some features for certain (in this case modeled in the Archaeological Heritage 
(Site).E27 - Component.E18 relationship) but if others exist is uncertain. We 
believe this valuable information should not get lost (quite often theory 
construction is based on such information).

For example it could be uncertain if an archaeological feature is to be named 
pit or ditch - or if it exists at all. Another example could be the 
questionable relationship of a findspot to a certain archaeological period. To 
make it even more difficult, different authors could have different thoughts on 
that.

As far as we can see, the expression of such uncertainty is not covered by 
Arches yet. Is there a concept for the integration of such data in the future? 
We are currently thinking into potential solutions but are struggeling to find 
adequate expressions for uncertain information in CIDOC.

RE: [Arches] Test suite in Arches

[Van Daele, Koen]

Hi Alexei,

I will ask you the Karl question: Has a ticket been added to the issue tracker 
yet? I had a quick look and I couldn't find one.

I think the best way to try for py3 support is to first get a test suite 
running under python 2.7. Strive for a decent amount of code coverage (ideally 
100%, but that's never easy). When you have that running, you can use tox 
(http://tox.readthedocs.org/) to run your test suite in different environments 
(py27, py32, py33, even stranger ones like jython and pypy). This allows for 
developing in one main environment and immediately checking for compatibiltiy 
under another one. I'm guessing that when you add py3 to the mix, you'll have 
some cleaning to do, but once that's finished it's relatively easy to maintain 
compatibility with different versions.

Cheers,
Koen

Van: Alexei Peters [apet...@fargeo.com]
Verzonden: woensdag 19 maart 2014 17:50
Aan: Van Daele, Koen
CC: Tharindu Rusira; archesproject@googlegroups.com
Onderwerp: Re: [Arches] Test suite in Arches

HI Koen,
No plans on python 3 support yet.  That doesn't mean we wouldn't be thrilled if 
someone from the community took that on (hint hint...).  :)

As far as the unit testing stuff goes, I still don't think django officially 
supports postgresql schemas.

Cheers,
Alexei


Director of Web Development - Farallon Geographics, Inc. - 971.227.3173


On Wed, Mar 19, 2014 at 3:38 AM, Van Daele, Koen 
koen.vanda...@rwo.vlaanderen.bemailto:koen.vanda...@rwo.vlaanderen.be wrote:
Hi Alexei,

Just had a look at the pytest-django docs. They seem to take a very sensible 
approach. Seems like a good way to move forward.

I remember trying to get any kind of unit test running with Arches 0.x and the 
main issue were some of the changes to make Django use postgresql schemes. No 
idea how that's changed in 2.0 (since it's now Django 1.6), so hopefully that's 
no longer an issue.

Good choice on dropping 2.6. It's only ArcGIS that sticks with ouddated python 
versions ;-)

Any plans for python 3 support?

Cheers,
Koen

Koen Van Daele
Applicatiebeheerder
M +32 (0)499 94 93 68tel:%2B32%20%280%29499%2094%2093%2068  |  T +32 (0)2 553 
16 82tel:%2B32%20%280%292%20553%2016%2082

Van: Alexei Peters [mailto:apet...@fargeo.commailto:apet...@fargeo.com]
Verzonden: dinsdag 18 maart 2014 19:27
Aan: Van Daele, Koen
CC: Tharindu Rusira; 
archesproject@googlegroups.commailto:archesproject@googlegroups.com

Onderwerp: Re: [Arches] Test suite in Arches

Hi Koen,
That's the one thing that I noticed about pytest-dango, is that you don't need 
to recreate the db for every test run.
Also, I removed the ability to use python 2.6 in version Arches 2.0
Cheers,
Alexei


Director of Web Development - Farallon Geographics, Inc. - 
971.227.3173tel:971.227.3173

On Tue, Mar 18, 2014 at 1:42 AM, Van Daele, Koen 
koen.vanda...@rwo.vlaanderen.bemailto:koen.vanda...@rwo.vlaanderen.be wrote:
What versions of python are you aiming at? Is Arches still trying to maintain 
compatibility with 2.6? And is python 3 compatibility wanted? Might influence 
your choice of testing framework.

If you're trying to maintain compatibility with 2.6, look into the unittest2 
library. I believe it brings python 2.6's unittest module up to the level of 
python 2.7's library.

I've never used py.test but it looks rather nice. We normally use the standard 
unittest library, nose as a testrunner and coverage for the coverage report. I 
believe nose can also run py.test tests. And py.test seems to be able to run 
nose test suites. So, it all seems to be rather connected.

When constructing your testsuite, it's a good idea to try and write as many 
unit tests as possible without needing a database. These can run a whole lot 
faster than unit tests requiring a database.

Normally your testdatabase will be constructed and torn down for every single 
unit test. As soon as your database is anything non-trivial, this slows your 
testsuite down to a crawl. We currently solved this in one of our applications 
by only setting up and tearing down the database once per testsuite. This is 
using the Pyramid framework (not Django). What we did, was to setup the 
database at the beginning of the module, start a database transaction, run the 
unit test, rollback the transaction, start a new transaction, run the second 
test, rollback the transaction, etc... At the end of the module, the entire 
database is torn down. The next module does the same thing. The inspiration for 
this came from this blog post: 
http://sontek.net/blog/detail/writing-tests-for-pyramid-and-sqlalchemy. It's 
about Pyramid (and uses py.test), but I think the principal should be 
applicable to other frameworks as well. Django might have something for this as 
well.

Cheers,
Koen



Koen Van Daele
Applicatiebeheerder
M +32 (0)499 94 93 68tel:%2B32%20%280%29499%2094%2093%2068  |  T +32 (0)2 553 
16 82tel:%2B32%20%280%292%20553%2016%2082

Van: archesproject

RE: [Arches] Test suite in Arches

[Van Daele, Koen]

Hi Alexei,

I don't think you need python 3 support right now. I do feel that the winds are 
changing. When we switched to python in late 2012, a lot of libraries were not 
compatible with python 3. The past year however, I've seen a lot of projects 
switch to support both 2.7 and 3.x. I believe Django is now also compatible 
with 3.x.

Thing is, supporting both 2.7 and 3.x is fairly easy to do. The biggest issues 
have to do with handling of unicode and bytes. But there are some libraries out 
there that handle that fairly well.

Whenever we set up something, we strive for compatibilty with 3.x on condition 
that all libraries we're using have been ported themselves. And lately, that 
seems to be the case more often. But, as most people out there, we're still 
running our production apps in 2.7.

The nice thing is, that if you start coding 2.7 in a style compatible with 3.x, 
once 2.7 stops being supported, you're not facing a massive change to 3.x 
anymore.

The fact that Arches creates it's own virtualenv during installation is one of 
the stranger aspects of Arches deployment. Most other projects I know tend to 
not create their own virtual environment. I thought the main reason Arches did 
this was because two Django files needed to be overwritten after installation 
of Django in the virtual environment? Other than that, does the installation 
specifically create a 2.7 environment or does it just end up with what the 
system python creates?

Anyway, python 2.7 is not going away soon. Python 3.x (3.4 just came out this 
week) is gaining momentum, but it'll still take a while before it takes over. 
My 2 cents: supporting both 2.7 and 3.x at the same times is not that hard once 
you have a decent test environment set up. Without one, I really wouldn't 
recommend even trying it.

Cheers,
Koen



Van: Alexei Peters [apet...@fargeo.com]
Verzonden: donderdag 20 maart 2014 17:27
Aan: Van Daele, Koen
CC: Tharindu Rusira; archesproject@googlegroups.com
Onderwerp: Re: [Arches] Test suite in Arches

Hi Koen,
No, there is no ticket yet for python 3 support.  My question to you is, do you 
really think we need it?  Remembering that Arches runs in it's own virtualenv 
and that 2.7 will probably be supported for a long time, is this something that 
might turn people away?
I don't really have a sense of python 3 support/adoption in the community.
Cheers,
Alexei


Director of Web Development - Farallon Geographics, Inc. - 971.227.3173


On Thu, Mar 20, 2014 at 1:37 AM, Van Daele, Koen 
koen.vanda...@rwo.vlaanderen.bemailto:koen.vanda...@rwo.vlaanderen.be wrote:
Hi Alexei,

I will ask you the Karl question: Has a ticket been added to the issue tracker 
yet? I had a quick look and I couldn't find one.

I think the best way to try for py3 support is to first get a test suite 
running under python 2.7. Strive for a decent amount of code coverage (ideally 
100%, but that's never easy). When you have that running, you can use tox 
(http://tox.readthedocs.org/) to run your test suite in different environments 
(py27, py32, py33, even stranger ones like jython and pypy). This allows for 
developing in one main environment and immediately checking for compatibiltiy 
under another one. I'm guessing that when you add py3 to the mix, you'll have 
some cleaning to do, but once that's finished it's relatively easy to maintain 
compatibility with different versions.

Cheers,
Koen

Van: Alexei Peters [apet...@fargeo.commailto:apet...@fargeo.com]
Verzonden: woensdag 19 maart 2014 17:50
Aan: Van Daele, Koen
CC: Tharindu Rusira; 
archesproject@googlegroups.commailto:archesproject@googlegroups.com
Onderwerp: Re: [Arches] Test suite in Arches

HI Koen,
No plans on python 3 support yet.  That doesn't mean we wouldn't be thrilled if 
someone from the community took that on (hint hint...).  :)

As far as the unit testing stuff goes, I still don't think django officially 
supports postgresql schemas.

Cheers,
Alexei


Director of Web Development - Farallon Geographics, Inc. - 
971.227.3173tel:971.227.3173


On Wed, Mar 19, 2014 at 3:38 AM, Van Daele, Koen 
koen.vanda...@rwo.vlaanderen.bemailto:koen.vanda...@rwo.vlaanderen.bemailto:koen.vanda...@rwo.vlaanderen.bemailto:koen.vanda...@rwo.vlaanderen.be
 wrote:
Hi Alexei,

Just had a look at the pytest-django docs. They seem to take a very sensible 
approach. Seems like a good way to move forward.

I remember trying to get any kind of unit test running with Arches 0.x and the 
main issue were some of the changes to make Django use postgresql schemes. No 
idea how that's changed in 2.0 (since it's now Django 1.6), so hopefully that's 
no longer an issue.

Good choice on dropping 2.6. It's only ArcGIS that sticks with ouddated python 
versions ;-)

Any plans for python 3 support?

Cheers,
Koen

Koen Van Daele
Applicatiebeheerder
M +32 (0)499 94 93 
68tel:%2B32%20%280%29499%2094%2093%2068tel:%2B32%20%280%29499%2094%2093

RE: [Arches] Test suite in Arches

[Van Daele, Koen]

What versions of python are you aiming at? Is Arches still trying to maintain 
compatibility with 2.6? And is python 3 compatibility wanted? Might influence 
your choice of testing framework.

If you're trying to maintain compatibility with 2.6, look into the unittest2 
library. I believe it brings python 2.6's unittest module up to the level of 
python 2.7's library.

I've never used py.test but it looks rather nice. We normally use the standard 
unittest library, nose as a testrunner and coverage for the coverage report. I 
believe nose can also run py.test tests. And py.test seems to be able to run 
nose test suites. So, it all seems to be rather connected.

When constructing your testsuite, it's a good idea to try and write as many 
unit tests as possible without needing a database. These can run a whole lot 
faster than unit tests requiring a database.

Normally your testdatabase will be constructed and torn down for every single 
unit test. As soon as your database is anything non-trivial, this slows your 
testsuite down to a crawl. We currently solved this in one of our applications 
by only setting up and tearing down the database once per testsuite. This is 
using the Pyramid framework (not Django). What we did, was to setup the 
database at the beginning of the module, start a database transaction, run the 
unit test, rollback the transaction, start a new transaction, run the second 
test, rollback the transaction, etc... At the end of the module, the entire 
database is torn down. The next module does the same thing. The inspiration for 
this came from this blog post: 
http://sontek.net/blog/detail/writing-tests-for-pyramid-and-sqlalchemy. It's 
about Pyramid (and uses py.test), but I think the principal should be 
applicable to other frameworks as well. Django might have something for this as 
well.

Cheers,
Koen



Koen Van Daele
Applicatiebeheerder
M +32 (0)499 94 93 68  |  T +32 (0)2 553 16 82

Van: archesproject@googlegroups.com [mailto:archesproject@googlegroups.com] 
Namens Tharindu Rusira
Verzonden: dinsdag 18 maart 2014 8:46
Aan: Alexei Peters
CC: archesproject@googlegroups.com
Onderwerp: Re: [Arches] Test suite in Arches



On Tue, Mar 18, 2014 at 12:25 PM, Alexei Peters 
apet...@fargeo.commailto:apet...@fargeo.com wrote:
Hi Tharindu,
It looks like PyUnit is in fact the unittest module in python (from their docs, 
'Unless you're stuck in the year 2000, PyUnit is in your Python standard 
library as module unittest.')

Yes, they are both the same.


I also just saw this: https://pypi.python.org/pypi/pytest-django/
One thing caught my eye when they describe the advantages of pytest which was 
Database re-use: no need to re-create the test database for every test run.
That may be something we might have to look out for.

Interesting.
I have not worked with py.test yet. The documentation is pretty convincing but 
I think we better have a look at these[1][2] comparisons
as well. (Also note that these discussions are quite old and probably obsolete).
Also, I found[3][4][5] to be very comprehensive articles on Python testing 
frameworks. So this might be a good starting point to decide where we will be 
going with our testing work-flow.

[1] 
http://stackoverflow.com/questions/191673/preferred-python-unit-testing-framework
[2] http://bytes.com/topic/python/answers/43330-unittest-vs-py-test

[3] http://pythontesting.net/framework/pytest/pytest-introduction/
[4] http://halfcooked.com/presentations/pyconau2013/why_I_use_pytest.html
[5] http://pydanny.com/pytest-no-boilerplate-testing.html

Thanks,
-Tharindu.


Anyway, we have options.
Cheers,
Alexei


Director of Web Development - Farallon Geographics, Inc. - 971.227.3173

On Mon, Mar 17, 2014 at 11:38 PM, Tharindu Rusira 
tharindurus...@gmail.commailto:tharindurus...@gmail.com wrote:


On Tue, Mar 18, 2014 at 11:59 AM, Alexei Peters 
apet...@fargeo.commailto:apet...@fargeo.com wrote:
Hi Tharindu,
Django uses the builtin unittest module.  It would be nice to use that if 
possible.
Of course it's possible, Alexei. I think that would be a better approach to use 
this native unittest module.
Thanks,
-Tharindu


https://docs.djangoproject.com/en/dev/topics/testing/

Cheers,
Alexei


Director of Web Development - Farallon Geographics, Inc. - 
971.227.3173tel:971.227.3173

On Mon, Mar 17, 2014 at 10:05 PM, Tharindu Rusira 
tharindurus...@gmail.commailto:tharindurus...@gmail.com wrote:

Thanks Alexei,
In my gsoc project I'm planning to write unit tests using PyUnit[1]. So I 
wanted to know whether PyUnit framework is fine or we have a project specific 
test mechanism.

[1] http://pyunit.sourceforge.net

Regards,
On Mar 18, 2014 10:05 AM, Alexei Peters 
apet...@fargeo.commailto:apet...@fargeo.com wrote:
Hi Tharindu,
At the moment we don't have any automated test routines.
That's why we have project Integrate a build/ci server and automated unit 
tests into the Arches dev environment.
It would be lovely to have that!
Cheers,
Alexei


Director of Web 

RE: [Arches] Re: Ideas for GSoC Project: Improve Representation of Spatial-Temporal Relationships

[Van Daele, Koen]

Hi Daphne,

I think that anything that has to deal with vague temporal or spatial 
information will automatically turn out te be fairly complex.

The fuzzy endpoints of periods is probably the easier one. There the big hurdle 
is making a good UI that allows for people to enter vague temporal information 
in a simple and straightforward way that doesn't take to long. We've currently 
implemented this with a set of widget that allow choosing a start period and an 
end period whereby both can have different levels of fuzzyness. Eg. A period of 
time that runs from 'the 8th century BC' till the year 57 BC (Iron Age in 
Belgium). It's a very textual way of entering data, but that seems to match our 
users best. Although in actual fact we currently only use the Fuzzy Temporal 
Intervals for dates of Events (an excavation, a survey, aerial photography, 
...) For dating of sites and monuments we use an entirely different approach 
based solely on a thesaurus of periods.

I think the second direction: Allow periods and regions to be associated poses 
a lot more conceptual issues. I guess one could define a whole set of different 
Iron Ages, but I have a hard time seeing how it would all work on a practical 
level. Especially once you start to factor in bringing together data from 
different implementations and the fact that a region can be vague as well. I 
also think that another complicating factor can be researches who tend take 
their chronological system with them. For example, in Flanders, a part of the 
Iron Age archaeologists use the French division of the Iron Age and another 
group uses the Dutch division. One group tends to work closer to the French 
border, the other one closer to the Dutch border. But they also seem to be 
aligned with the universities the archaeologists graduated from. So I'm not 
even sure anyone would be capable of geographically defining those ares. 
Also, when someone then tries to search for Late Iron Age, should they get 
results from the French Late Iron Age, the Dutch one or both? Or should it be 
impossible to search for something like 'Late Iron Age' and only possible to 
search for a specific one? I'm also not sure if the definition of a certain 
period in a certain area has stayed constant through time. That probably 
affects things as well. Anyway, it's an absolutely facinating research topic, 
but one where I think you really need to define a clear scope for your project 
and resist the tempation of going to far away from it. And make sure that you 
can communicate to users of your system which choices you've made and why.

Cheers,
Koen

Van: archesproject@googlegroups.com [archesproject@googlegroups.com] namens 
Daphne Ippolito [daphne...@gmail.com]
Verzonden: zondag 16 maart 2014 20:08
Aan: archesproject@googlegroups.com
Onderwerp: [Arches] Re: Ideas for GSoC Project: Improve Representation of 
Spatial-Temporal Relationships

Hi.
Thanks for your responses. I don't know Dutch, but your abstract did a good job 
of explaining what the situation is and the terminology I should be looking 
into.

There seems to be  two separate directions this project could go in.

 1.  Improving the way Arches deals with endpoints of periods by allowing for 
the input of fuzziness. This task doesn't have a spatial component to it. The 
project would involve implementing a way for users to specify uncertainty and 
subjectivity in their period definitions using the algorithms you mention in 
your post.
 2.  Creating the ability for periods to be spatially referenced. This task 
only deals with the spatial, not the temporal, definition of periods. It has 
two tiers of complexity. First, allow periods to be associated with geospatial 
regions. Second, devise a system to represent the fuzziness of these geospatial 
regions.

I am not sure whether both can be tackled in one summer, or if I should pin my 
focus on one.

Thanks for all the input.

-Daphne

On Sunday, 16 March 2014 05:10:07 UTC-4, Koen Van Daele wrote:
Hi Daphne,

I did some work on representing vague/imperfect temporal information in 
relational databases. It's a very interesting, but quite complicated subject. I 
did a masterpaper on it, but that's in Dutch. I don't suppose you speak Dutch? 
It does also contain an English abstract. You can find the masterpaper here: 
http://lib.ugent.be/fulltxt/RUG01/001/418/820/RUG01-001418820_2010_0001_AC.pdf. 
I did a conference paper on this subject at CAA 2012. You can find the slids 
here: 
http://www.slideshare.net/koenedaele/imperfect-temporal-information-in-data-sets.
 The digital paper should have been published by now, but the CAA publication 
process seems to be quite slow at the moment.

The basic idea was to treat spans of time as fuzzy sets. I then implemented and 
tested a few different algorithms on actually reasoning with these fuzzy 
temporal intervals. The implementation consists of using postgis geometries for 
representing fuzy 

RE: [Arches] Error while installing Arches on Ubuntu

[Van Daele, Koen]

Hi cristina,

I’d suggest the following. I’ve created and slightly changed version of Arches 
that installs on my Ubuntu 12.04. Please follow these instructions and let me 
know what happens:

Anything that starts with a dollar sign should be entered through the command 
line. Please don’t use sudo here. It’s necessary for the postgres installation, 
but a bad idea for installing the Arches software itself.

Go to you home dir
$ cd /home/cris
Remove your current version (might need to use sudo here if your previous 
version was downloaded with sudo)
$ rm –rf ArchesWeb
Download a modified version
$ wget https://bitbucket.org/koenedaele/arches/get/default.tar.gz
Untar this
$ tar –xzf default.tar.gz
Rename folder
$ mv koenedaele-arches-ca52ad7c6f55 ArchesWeb
Install Arches
$ cd ArchesWeb/Install
$ ./install.sh

This should work.

Alexei, the errors that Cristina encounters have to do with problems with 
distribute and setuptools. The changes I made (see 
https://bitbucket.org/koenedaele/arches/commits/ca52ad7c6f55204f43916f0b25acb462b2d545ca)
 fix these by first forcing an upgrade of setuptools (so pip can do its job 
later on) and removing the hardcode dependency on distribute 6.something. There 
might be other fixes possible (maybe downgrading pip away from pip 1.5) and I 
have no idea if this works on a windows or mac.

Also, is there any reason for downloading and untarring Arches as superuser? 
I’d suggest changing that in the installation guide.

Cheers,
Koen

Koen Van Daele
Applicatiebeheerder
M +32 (0)499 94 93 68  |  T +32 (0)2 553 16 82

Van: archesproject@googlegroups.com [mailto:archesproject@googlegroups.com] 
Namens Van Daele, Koen
Verzonden: dinsdag 14 januari 2014 9:46
Aan: Tudor Cristina; archesproject@googlegroups.com
Onderwerp: RE: [Arches] Error while installing Arches on Ubuntu

Hi Cristina,

Please don’t reinstall Ubuntu. That’s unneeded and it will not solve anything. 
I’ll have a look at your current erros and see if I can figure out how to best 
help you.

Cheers,
Koen

Koen Van Daele
Applicatiebeheerder
M +32 (0)499 94 93 68  |  T +32 (0)2 553 16 82

Van: Tudor Cristina [mailto:tudor.cristina.miha...@gmail.com]
Verzonden: maandag 13 januari 2014 21:30
Aan: archesproject@googlegroups.commailto:archesproject@googlegroups.com
CC: Tudor Cristina; Van Daele, Koen
Onderwerp: Re: [Arches] Error while installing Arches on Ubuntu

It in strange that in './virtualenv/ENV' I don't have a tmp folder. I will 
reinstall Ubuntu and everything else tomorrow, and let you know how thinks will 
go. Thank you once again for your support.

On Monday, January 13, 2014 8:19:43 PM UTC+2, Alexei Peters wrote:
Sorry clicked send prematurely...

Hi Cristina,
Many times the error messages that can get during issues like this seem 
intimidating, but in reality usually give you clues as to what happened.
In this case, if you look at the error message you'll notice that it tells you 
exactly what to do.

pip can't proceed with requirement 'distribute==0.6.27 (from -r 
/home/cris/ArchesWeb/Install/requirements.txt (line 5))' due to a pre-existing 
build directory.
 location: /home/cris/ArchesWeb/virtualenv/ENV/tmp/distribute
This is likely due to a previous installation that failed.
pip is being responsible and not assuming it can delete this.
Please delete it and try again.

It's saying that you had an installation that previously failed.  During that 
failed installation, the build process created a tmp folder in 
/home/cris/ArchesWeb/virtualenv/ENV/
Now that you're trying to redo the build, it's saying that you need to delete 
the tmp folder.  It's not going to delete it for you.
I would delete that folder and then begin the install process all over again.
That should work.

Cheers,
Alexei


Director of Web Development - Farallon Geographics, Inc. - 971.227.3173

On Mon, Jan 13, 2014 at 10:14 AM, Alexei Peters 
ape...@fargeo.comjavascript: wrote:
Hi Cristina,
Many times the error messages that can get during issues like this seem 
intimidating, but in reality usually give you clues as to what happened.
In this case, if you look at the error message you'll notice that it tells you 
exactly what to do.

pip can't proceed with requirement 'distribute==0.6.27 (from -r 
/home/cris/ArchesWeb/Install/requirements.txt (line 5))' due to a pre-existing 
build directory.
 location: /home/cris/ArchesWeb/virtualenv/ENV/tmp/distribute
This is likely due to a previous installation that failed.
pip is being responsible and not assuming it can delete this.
Please delete it and try again.


Director of Web Development - Farallon Geographics, Inc. - 971.227.3173

On Mon, Jan 13, 2014 at 9:51 AM, Tudor Cristina 
tudor.crist...@gmail.comjavascript: wrote:
So, I finally edited install.py as you said, but I still get this error :

pip can't proceed with requirement 'distribute==0.6.27 (from -r 
/home/cris/ArchesWeb/Install/requirements.txt (line 5))' due to a pre-existing 
build directory.
 location: /home/cris/ArchesWeb/virtualenv