[Therion] Hopefully a bunch of new users; Optimum Data Managment

[Bruce Mutton]

Hi Jonny

You've lost me. I'm familiar with subversion, and I use it from time to time
for the greater good. Don't know what cvs is, and that leads me to suspect
we're talking about a different type of subversion.  Perhaps it is some form
of web based storage and data management?



My friend Mike and I are working on some drawings collaboratively, and we
synchronize our data manually.  Essentially we agree who, at any one time
has the 'master' version of a particular set of files.  It works reasonably
well, and would theoretically work for multiple users, admittedly with a
high margin of error.

Bruce



  _  

From: therion-bounces at speleo.sk [mailto:therion-boun...@speleo.sk] On Behalf
Of Jonny Prouty
Sent: Monday, 16 June 2008 3:41 p.m.
To: therion at speleo.sk
Subject: Re: [Therion] Hopefully a bunch of new users; Optimum Data
Managment



On Mon, Apr 28, 2008 at 6:47 AM, Bruce Mutton 
wrote:

I am also trying to optimise my system for collaborative data entry and
development of the Therion cave dataset, which Therion seems ideally suited
to.



Bruce,

I don't know how I missed this message when you posted it initially. I just
stumbled onto it searching through the archives for an example of something
else. I too have been hoping to use Therion collaboratively, but I haven't
been able to convince anyone else I survey with to learn Therion, so I've
made little headway. 

I was thinking of using cvs or subversion to maintain collaboratively
generated data. This would cut down a fair bit on the hassle of manually
keeping your workspace up to date. Of course, there is the additional hassle
of using cvs (or subversion), but it shouldn't be too bad.

Again, I haven't actually done this, but I haven't thought of any major
obstacles either.

-Jonny ^v^

-- next part --
An HTML attachment was scrubbed...
URL: 
<http://mailman.speleo.sk/pipermail/therion/attachments/20080616/bde8ae26/attachment.html>

[Therion] Hopefully a bunch of new users; Optimum Data Managment

[lue...@vaw.baug.ethz.ch]

Bruce Mutton writes:
 > You've lost me. I'm familiar with subversion, and I use it from time to time
 > for the greater good. Don't know what cvs is, and that leads me to suspect
 > we're talking about a different type of subversion.  Perhaps it is some form
 > of web based storage and data management?

CVS is the ancestor of subversion, and there is no reason to use it for a new
project. Frankly, I would go with a distributed version control system, such
as Mercurial (hg) or bazaar.

 > My friend Mike and I are working on some drawings collaboratively, and we
 > synchronize our data manually.  Essentially we agree who, at any one time
 > has the 'master' version of a particular set of files.  It works reasonably
 > well, and would theoretically work for multiple users, admittedly with a
 > high margin of error.

There is always diff and graphicals frontends to the help. However, using a
version control system always makes sense, even if you are the only user.

If you are using Emacs to edit Therion files, version control integration
comes for free (as is the case with many other editors).

Best, Martin

[Therion] Hopefully a bunch of new users; Optimum Data Managment

[Jonny Prouty]

On Mon, Apr 28, 2008 at 6:47 AM, Bruce Mutton 
wrote:

> I am also trying to optimise my system for collaborative data entry and
> development of the Therion cave dataset, which Therion seems ideally suited
> to.
>

Bruce,

I don't know how I missed this message when you posted it initially. I just
stumbled onto it searching through the archives for an example of something
else. I too have been hoping to use Therion collaboratively, but I haven't
been able to convince anyone else I survey with to learn Therion, so I've
made little headway.

I was thinking of using cvs or subversion to maintain collaboratively
generated data. This would cut down a fair bit on the hassle of manually
keeping your workspace up to date. Of course, there is the additional hassle
of using cvs (or subversion), but it shouldn't be too bad.

Again, I haven't actually done this, but I haven't thought of any major
obstacles either.

-Jonny ^v^
-- next part --
An HTML attachment was scrubbed...
URL: 

[Therion] Hopefully a bunch of new users; Optimum Data Managment

[Bruce Mutton]

-Original Message-
From: therion-bounces at speleo.sk [mailto:therion-boun...@speleo.sk] On Behalf
Of Wookey
...
The above is correct, but I think doesn't do justice to Tunnel. It has
significant advantages too, and thus is worth looking at. 
...
Wookey

Thanks for maintaining the balance Wookie
Appreciate your posts.
Bruce

[Therion] Hopefully a bunch of new users; Optimum Data Managment

[Wookey]

On 2008-04-28 09:08 +1200, Bruce Mutton wrote:

>I still haven't seen anything written about Walls or Tunnel that would
>make me try them...  Well, I would try Walls, but it wouldn't download,
>and now I like Therion for its conceptual simplicity, even if user error
>detection can be a bit complex.
> 
>From the Tunnel website...
> 
>The disadvantages of Tunnel's way are as follows:
> 
>1.You have to do a lot of extra editing just to join in a new piece of
>the survey.
> 
>2.You wind up with multiple copies of the same piece of cave in your
>workspace.
> 
>3.You lose the connection to the original drawings so you can't
>correct them in the original state and see the changes in the final
>drawing.
> 
>4.It's not known how many times a survey could be copied and morphed
>before unexpected errors of distortion would become a problem.
> 
>5.The files for a large survey can become rather large.

The above is correct, but I think doesn't do justice to Tunnel. It has
significant advantages too, and thus is worth looking at. It produces
nicer output than Therion, because it is possible to edit the final
version, e.g. arranging labels in the context of the whole map, not
just the current scraps (this is a _major_ problem with the current
Therion implementation that pretty-much killed my work with until I
get a faster computer). You can also do everything graphically, rather
than editing mysterious text files, which is a big deal for the
less-nerdy. Being able to use SVG symbols rather than write them in
metapost is nice too.

I avoided it for years because it was java, but that's less of
a problem than it used to be, and I have been very impressed by the
quality of the output:
http://cucc.survex.com/expo/smkridge/204/surveys/plan2006.png

Now that you've learned Therion there is not much point jumping ship,
but I just thought I should correct theimpression given above.

I will package Tunnel for Debian/Ubuntu when I get a chance so it will
be a that bit easier to try it out.

Wookey
-- 
Principal hats:  Balloonz - Toby Churchill - Aleph One - Debian
http://wookware.org/

[Therion] Hopefully a bunch of new users; Optimum Data Managment

[Bruce Mutton]

If I interpret Marco's post regarding data structure description correctly,
it seems you have folders for 

(hope my ascii pictures survive intact)



-Cave folder

 |-'surveys', 

 |-'groups of surveys' and 

 |-'whole cave - collection of the groups' [thconfig file((s) for each
output task]



all at the same hierarchical level, but able to be stacked up or nested if
other caves were to be incorporated into the data set.



Seems nice and tidy to me, but the folder hierarchy does not match the
conceptual hierarchy of the data, which would be more like Jonathan
described,



-Region folder

 |-cave folder 

|-Surveys 

   |-Notes [scanned notes taken in cave]

   |-Therion [survey *.th data files]

|-Pictures

   |-Date [pictures taken on this date]



I like the concept that the folder hierarchy matches the data hierarchy and
especially that it is very cave centric, ie not just Therion data, but all
data related to the cave



Both the above seem quite tidy, and as Martin has just posted, each user
will have their own "cave project circumstances and preferences"

Marcos seems to me like it would have many versions of a number of thconfig
files.  I have found maintaining consistency among many thconfigs to be
quite a problem, as I am still learning and refining my opinion of what is a
good way to set them out.



I am also trying to optimise my system for collaborative data entry and
development of the Therion cave dataset, which Therion seems ideally suited
to.  Therefore the tiny size of the Therion batch files is an advantage I
want to keep.  So I would adopt Jonathans 'single cave centric' model at
once, except I don't want to be emailing back and forth large chunks of
non-Therion data amongst the people developing the dataset, nor do I want to
have to pick out fiddly bits of non relevant data each time.  I have adopted
the perhaps the lesser  approach of having two parallel folders, one for
Therion data and one for non-Therion data such as photos, cave related texts
and articles, scanned cave survey notes etc.  



The Therion folders look something like this;

(My thconfig files have the extension .thc - much nicer if using with
Windows OS)

(and as with the others, highly modular, one survey and very few scraps per
file)

(and with a fairly highly structured and explicit naming convention so you
know what is in a file without peering into it - not really described
herein)



-Regional folder [RegionINDEX.th contains very simple equates and 'maps',
Single Region.thc contains layouts and all exports for region wide outputs]

 |-common folder [layoutSTANDARD.thc, club logos and graphics common to all
layouts]   

 |-outputs folder  [all output files go here]

 |-surface folder [overlandsurveys.th, overlandINDEX.th, surfaceINDEX.th,
surface.jpg]

 |-cave1 folder [surveys.th contains centrelines & maps relating to
associated scraps, caveAElev290.th2 contains scraps generally relating to
survey A, caveINDEX.th containing entrance co-ords; equates; joins and maps,
single cave1.thc contains layouts and all exports for cave1 wide outputs,
and if there is no actual survey data, caveplan.jpg - a picture of an old
fashioned cave plan for which there is no data]

|-sketches [sketchAElev290.jpg containing sketches for scraps relating
to this cave.]

 |-cave2 folder [caveINDEX.th contains very simple equates and 'maps',
Single cave2.thc contains layouts and all exports for cave2 wide outputs

|-group21 folder [same as for cave1 folder above]

   |-sketches [sketchAElev290.jpg containing sketches for scraps
relating to this cave group or branch.]

|-group22 folder [same as for cave1 folder above]

   |-sketches [as above.]





Cave1 is a simple cave a few km long perhaps; cave2 might have many km or
distinct 'branches'.

This is perhaps flatter than Jonathans, but still hierarchical.  Has few
thconfig files, one per 'family' of passages 'selected' for output and one
'singlesurveytest.thc per cave, to enable frequent and quick checking of
survey and sketch data as it is being entered (and still they drive me
crazy).  The common, outputs and surface folders live at the top level and
do not need to be shared (often) among users.  The cave data folders can
easily be zipped without the sketches and shared by email.  This is working
well informally with two users, but would require good discipline and record
keeping with more simultaneous users.



I plan to keep single thconfig files for exporting all outputs, but break
them up into referenced modules according to function; ie layoutMapPlan.thc,
layoutMapElevation290.thc, layoutAtlasPlan.thc, etc.  This might make them
easier to manage.



We don't do splay legs (well, perhaps one or two for every 100 survey legs,
or if the passage is 20m wide and the surveyors' fussy) so I don't make a
distinction for them.



I have yet to deal with significant levels of the cave which ov

[Therion] Hopefully a bunch of new users; Optimum Data Managment

[Martin Sluka]

On 27.4.2008, at 23:08, Bruce Mutton wrote:

> While the information that you just posted from the Therion Book

I apologize myself it was not meant as RTFM in any matter. :)

m.
-- next part --
An HTML attachment was scrubbed...
URL: 

[Therion] Hopefully a bunch of new users; Optimum Data Managment

[Bruce Mutton]

Martin

While the information that you just posted from the Therion Book is accurate
and helped me in some way get started, in that the data structure I use
follows a similar 'system', it is only once I have my 'system' working that
I can see that what I have done in fact follows the Therion Book.  Does that
make sense?



As a new user a diagrammatic illustration of the 'structure' and perhaps
itemized lists of what goes where, with some comprehensive example batch
file contents would have helped me, and still would.  (The simple examples
that are available are necessary and helpful however - for illustrating one
aspect at a time - without the clutter of other things going on)

>From some of the questions asked on this forum over the last six months, and
the forum link that Jonathan provided
http://forums.caves.org/viewtopic.php?f=26
<http://forums.caves.org/viewtopic.php?f=26&t=6757&p=55739> &t=6757&p=55739
it would seem there are other users for which the same would apply.  It
would seem to me that Bill Gee either had some extremely bad luck, or is
overstating his skill base.  I have a fraction of his computer experience
and have been lucky enough to have had fewer problems even though I started
on a relatively large and complex cave system.  He does seem to have tried
quite a lot though - very comprehensive questions.



Jonathan, His unanswered questions regarding cross section orientation and
joining scraps are well explained in the Wiki, Wookies article, and this
forum as I recall.  If scrap outlines overlap themselves, it's either
because you haven't finished drawing the scrap (open ends of passage very
far apart with the wall wiggling out through the 'opening') or because the
'drag handles' from adjacent points 'overlap' each other. (Not always a
problem, but often is, after 'join' morphing has added extra distortion to
the Bezier curves)



Regarding his units troubles.  The funny thing is that Compass and
On-Station give me the same trouble, and Therion never has.  At least
Therion stores the input data as the user enters it - makes it easier to pin
point bugs or user errors.



Any way, back to the topic: Therion Data Structures.  

I'm putting something together for my own use, ('cos it won't all fit in my
head) but the details are evolving quite rapidly, as I gain more experience,
try new things, and learn new tricks.  I'll post it here or on the Wiki if I
can figure out how to.  Won't be for some months perhaps.



I still haven't seen anything written about Walls or Tunnel that would make
me try them.  Well, I would try Walls, but it wouldn't download, and now I
like Therion for its conceptual simplicity, even if user error detection can
be a bit complex.

>From the Tunnel website.

The disadvantages of Tunnel's way are as follows: 

1.You have to do a lot of extra editing just to join in a new piece of
the survey. 

2.You wind up with multiple copies of the same piece of cave in your
workspace. 

3.You lose the connection to the original drawings so you can't correct
them in the original state and see the changes in the final drawing. 

4.It's not known how many times a survey could be copied and morphed
before unexpected errors of distortion would become a problem. 

5.The files for a large survey can become rather large. 



I seem to remember that the Tunnel web site 'said' it could not do elevation
projections, as of a couple of months ago, but I can not find that just now.
Anyway, alpine caves need elevation projections, so Tunnel would seem to be
no good for me.



Keep up the good work.



Bruce



  _  

From: therion-bounces at speleo.sk [mailto:therion-boun...@speleo.sk] On Behalf
Of Martin Sluka
Sent: Monday, 28 April 2008 5:48 a.m.
To: therion at speleo.sk
Subject: Re: [Therion] Hopefully a bunch of new users; Optimum Data
Managment



On 23.4.2008, at 10:32, marco corvi wrote:



maps hierarchies to organize the scraps, and select/unselect to 

specify what to export



Two section regarding organization of data from therion book:

m.



How to enter centreline?



Text deleted .



How to draw maps?



Text deleted .



-- next part --
An HTML attachment was scrubbed...
URL: 
<http://mailman.speleo.sk/pipermail/therion/attachments/20080428/714acdb6/attachment.html>

[Therion] Hopefully a bunch of new users; Optimum Data Managment

[Martin Sluka]

On 27.4.2008, at 23:08, Bruce Mutton wrote:

> As a new user a diagrammatic illustration of the ‘structure’ and  
> perhaps itemized lists of what goes where, with some comprehensive  
> example batch file contents would have helped me, and still would.


Bruce,

only short answer. We all have experience with our particular cave  
system or areas. They are very very different.

Therion was developed for documentation of quite complicated cave  
with many levels overlapping each other. It works in many other cases  
depending on imagination of its user.

The main principle is that the map itself is divided into small  
pieces - scraps. The scraps are absolutely independent from survey  
structure created more or less historically. You may use in one scrap  
data from several surveys made historically in very long interval by  
different surveyors. Only condition is all the data are accessible in  
survey which include the map structure with that particular scrap.

The main rules to create such structure is to divide the system into  
horizontal layers (in case of map) which the atlas will be able to  
recognize and shows in enough simple way. There one may see the  
importance of idea of scraps -you may reorganize your structure  
anytime in the future.

The main problem is organize the maps and submaps. The structure of  
survey data itself is existing more or less automatically.

I hope MartinB or Stacho may add more.

Martin

-- next part --
An HTML attachment was scrubbed...
URL: 

[Therion] Hopefully a bunch of new users; Optimum Data Managment

[Martin Sluka]

On 23.4.2008, at 10:32, marco corvi wrote:

> maps hierarchies to organize the scraps, and select/unselect to
> specify what to export

Two section regarding organization of data from therion book:

m.

How to enter centreline?

The basic building block is the centreline command. If the cave is  
larger than a few
meters it's a good idea to split data in more  les and separate  
centreline data from map
data.
We usually use one *.th  le containing centreline per survey trip.  
It's handy to start with
an empty template  le as shown bellow, where dots will be replaced  
with appropriate
texts.
37
encoding ISO8859-1
survey ... -title "..."
centreline
team "..."
team "..."
date ...
units clino compass grad
data normal from to compass clino length
... ... ... ... ...
endcentreline
endsurvey
To create a unique namespace the centreline command is enclosed in  
survey ... end-
survey command. It's useful when the survey has the same name as the   
le which
contains it.38 The points will than be referenced using @ character| 
see the survey
command description.
For really large caves it's possible to build a hierarchical  
structure of directories. In such
a case we create one special  le called INDEX.th which includes all  
other *.th  les from
given directory and contains equate commands to de ne connections  
between surveys.

How to draw maps?

The most important thing is to devise division of the cave into  
scraps. Scrap is the basic
building block of the map. It's almost always a bad idea to try to  t  
each scrap to corresponding
*.th  le with centreline from one survey trip. The reason is that  
connections
between scraps should be as simple as possible. Scraps in general are  
independent on
centreline hierarchy so try to forget the survey hierarchy when  
drawing maps and choose
best scrap joins.
We usually insert maps in the last-but-one level in survey hierarchy. 
39 Each scrap may
than contain arbitrary part of any survey in the last level of  
hierarchy. For example,
there is a survey main which contains surveys a, b, c and d. Surveys  
a { d contain
centreline data from four survey trips and each of them is in a  
separate  le. There is a
map main_map which contains scraps s1 and s2. If the main_map is  
located in the main
survey, scrap s1 may cover part of the centreline from survey a,  
complete survey b and
part of c; s2 will cover part of the a and c surveys and a complete d  
survey. The survey
stations names will be referenced using @ symbol (e.g. 1 at a) in the  
scraps.40
Scraps are usually stored in *.th2  les. Each  le may contain more  
scraps. To keep
data well organized, we have some naming conventions: in the  le  
foo.th2 all scraps are
named foo_si, where i is 1, 2 an so on. Cross-sections are named  
foo_ci, lines foo_li
etc. This helps a lot with large cave systems: if some scrap is  
referenced, you immediately
know in which  le it had been de ned.
Similar to *.th  les, there may be one  le INDEX.th2 per directory  
which includes all
*.th2  les, de nes scrap joins and maps.
When drawing scraps you should check if the outline is properly de  
ned: all lines creating
the outer border should have -outline out option; all lines  
surrounding inner pillars -
outline in option. Scrap outlines can't intersect themselves| 
otherwise the inner side
of the scrap can't be determined. There are two simple tests that  
scrap outline is correct:
   there is no METAPOST warning \scrap outline intersects itself"
   when you set passage  ll to any color (color map-fg   
option in layout),
you may see what Therion considers to be inside of the scrap.

38 E.g. survey entrance in the  le entrance.th.

39 Remember that surveys create namespaces, so you may reference only  
objects in the given survey
and all subsurveys.

40 If you include maps in the top-level survey, you may reference any  
survey station in any scrap,
which is very flexible. On the other side you have than use longer  
names in stations references, like
3 at dno.katakomby.jmn.dumbier

-- next part --
An HTML attachment was scrubbed...
URL: 

[Therion] Hopefully a bunch of new users; Optimum Data Managment

[marco corvi]

On Wed, 2008-04-23 at 08:11 +1200, Bruce Mutton wrote:
> Thanks Wookie.
> Can not believe paper is better, but then in the 6 months I've been learning
> Therion, it has improved a lot...  Any more information or examples
> appreciated.
> 
> At present it's only my thconfig files that I think are messy and confusing.
> I'm working on making them very much more modular, a separate sub-thconfig
> file for every task.  
> 
> A thought; being able to pass a variable between thconfig files or even from
> .th files to thconfig files could make things easier by avoiding, for
> example, changing the name of maps to match the cave name.  See the
> following example
> 
> ...

hi,

i'm no expert about "data management", i can just tell what i tried
for a cave that's more than a few surveys/scraps.
it is still in a fuzzy shape, and it's just what i experienced
(ie, tested on caves, a few km long).

a directory for surveys, with a subdirectory for each survey
(each with its own thconfig, more on this later).
a second directory with subdirectories to group the surveys
together (each group with its thconfig).
finally other directories to put together the whole cave 
(with their thconfigs)

different thconfigs for different tasks (pdf maps, 3d model, etc.)

some "include thconfig" with the common layouts for the surveys
and groups and cave thconfigs. 

maps hierarchies to organize the scraps, and select/unselect to 
specify what to export

makefile to make what's needed.

one could create thconfigs on the fly 
(never tried: never had the need).

marco

[Therion] Hopefully a bunch of new users; Optimum Data Managment

[Bruce Mutton]

Thanks Wookie.
Can not believe paper is better, but then in the 6 months I've been learning
Therion, it has improved a lot...  Any more information or examples
appreciated.

At present it's only my thconfig files that I think are messy and confusing.
I'm working on making them very much more modular, a separate sub-thconfig
file for every task.  

A thought; being able to pass a variable between thconfig files or even from
.th files to thconfig files could make things easier by avoiding, for
example, changing the name of maps to match the cave name.  See the
following example

In thconfig file...
/code
#SELECT DATA MAPS
#
# selects all surveys/maps in 'sources' if not explicitly specified
# if is specified, prints 'title' in map legend 

select MiddleEarthPlanMap  # from MiddleEarthINDEX.th
select MiddleEarthxElevMap # from MiddleEarthINDEX.th
select MiddleEarthElev290Map   # from MiddleEarthINDEX.th

#MAP/ATLAS IMAGE EXPORTS
#===
export map -projection plan \
  -layout ThisCaveMapLayout \
  -layout ThisMapPlanLayout \  
  -output ../Outputs/MiddleEarthPlan.pdf  

export map   -projection [elevation 290 deg]  \
  -layout ThisCaveMapLayout \
  -layout ThisMapElev290Layout \ 
  -output ../Outputs/MiddleEarthElev290.pdf 

export map   -projection extended  \
  -layout ThisCaveMapLayout \
  -layout ThisMapxElevLayout \   
  -output ../Outputs/MiddleEarthExtendedElev.pdf  
/endcode

In the above, if the text "MiddleEarth" was replaced with a variable that
represented the cave name defined in the file MiddleEarthINDEX.th, then that
would save quite a lot of fiddling about when dealing with multi-cave or
multi-part-of-cave datasets.

Regards
Bruce

-Original Message-
From: therion-bounces at speleo.sk [mailto:therion-boun...@speleo.sk] On Behalf
Of Wookey
Sent: Tuesday, 22 April 2008 10:53 p.m.
To: therion at speleo.sk
Subject: Re: [Therion] Hopefully a bunch of new users

On 2008-04-19 20:52 +1200, Bruce Mutton wrote:
> which brings me
> to the single biggest hurdle that continues to dog me with Therion - how
to
> arrange ones data?  I realise there's probably about 50 good ways to do
> this, depending on ones needs and expectations, and some are described
very
> vaguely in the documentation.  I think I'm heading towards a reasonable
> strategy that's scalable from single caves to 60km systems with numerous
> small surficial caves nearby.  When I've got the bugs ironed out I'll see
if
> I can get something into the Wiki.  Probably in the form of some actual
> data, suitably scrambled so that it does not reflect a real cave, and with
> commonly used statements pre-entered and commented and well documented.
> 
> If anyone has already documented a Therion data management system I would
be
> most interested.  

The mulu project considered this at length a couple of years ago and
came up with a couple of approaches. These were confined by the fact
that the base centreline data was entered as Survex data, not as
therion data, and this makes things much more finiky. We did get it
all working eventually, but the people who ran the project eventually
decided they didn't like therion ('too complicated') and have gone
back to paper drawing (!), so my efforts on it kind of stopped.

I'm not sure if our final conclusions are properly documented anywhere
other than as our existing dataset, although you can probably infer
some of it from threads on this list. What there is may be the 'vague'
stuff on the wiki. 

The optimium method if exclusivley using Therion would be different
(and simpler). 

Wookey
-- 
Principal hats:  Balloonz - Toby Churchill - Aleph One - Debian
http://wookware.org/
___
Therion mailing list
Therion at speleo.sk
http://www.speleo.sk/mailman/listinfo/therion