sortable.py, sortable_help.py -- a script for sorting tables in LyX, & its help file sortlist.py, sortlist_help.py -- a script for sorting lists in LyX, & its help file
pLyXSorting(compressed).lyx -- an explanatory document with examples
The LyX document is in compressed LyX format. For the examples to work it must be saved in uncompressed format. (The pLyX system depends on LyX files being *text* files.) It contains some example tables and a substantial multi-level list to play with.
The table sorting script reworks for the pLyX system a script aired on this list back in September. On the developer's list I've noticed recent work to swap columns or swap rows in tables. I imagine once that is done, the next step will be hard to resist: go the whole way and sort the table. This might be a short-lived script.
The list sorting script is new and was harder to write because of the recursive calls to the sorting routine required by the (possible) presence of sub-lists. My underlying interest in sorting lists arose from a wish to sort indexes. (With their headings, subheadings, sub-subheadings, ... the logic is the same.)
Andrew
def helpnote(hv):
if hv > 1:
return header + version
else:
return header + tail
header = r'''\begin_layout LyX-Code
\family roman
\series bold
.sort table
\end_layout
'''
version = r'''\begin_layout LyX-Code
\family roman
Version 1.0 (15 December 2012) Columns can be sorted more than once for
inter-filed mixed-case alphabetical sorting.
\end_layout
\begin_layout LyX-Code
\family roman
Version 0.4 (1 November 2012) First version for pLyX system.
\end_layout
\begin_layout LyX-Code
\family roman
Version 0.3 (17 September 2012) Use of custom insets; yellow notes option,
hrules and vrules preserved.
\end_layout
\begin_layout LyX-Code
\family roman
Version 0.2 (13 September 2012) Script now ignores ERT insets.
\end_layout
\begin_layout LyX-Code
\family roman
Version 0.1 (12 September 2012 Table sorting script posted to user's list.
\end_layout
'''
tail = r'''\begin_layout LyX-Code
\family roman
Sort the
\emph on
rows
\emph default
by the values in specified columns.
\end_layout
\begin_layout LyX-Code
\end_layout
\begin_layout LyX-Code
\family roman
\series bold
Global options
\end_layout
\begin_layout LyX-Code
\family roman
-
\series bold
h --help
\series default
show this help note.
\end_layout
\begin_layout LyX-Code
\family roman
\series bold
-v --version
\series default
show version information.
\end_layout
\begin_layout LyX-Code
\family roman
\series bold
-n --notes
\series default
make LyX's (yellow) notes sortable; notes are
sort-neutral by default.
\end_layout
\begin_layout LyX-Code
\end_layout
\begin_layout LyX-Code
\family roman
\series bold
Local options
\end_layout
\begin_layout LyX-Code
\family roman
A sort specification is a sequence like
\series bold
2a3A1+
\series default
(or
\series bold
2a 3A 1+
\series default
or
\series bold
2a
\series default
,
\series bold
3A
\series default
,
\series bold
1+
\series default
, etc.) where the number indicates the column and the qualifying letter or
sign indicates the kind of sort. A specification may involve from one to
all columns in the table and a column may appear in the spec. more than once.
The primary sort is by the first column specified, the secondary sort by
the second column, etc.
\end_layout
\begin_layout Itemize
\family roman
\series bold
a, A, +
\series default
indicate ascending sorts;
\end_layout
\begin_layout Itemize
\family roman
\series bold
z, Z, -
\series default
indicate descending sorts;
\end_layout
\begin_layout Itemize
\family roman
letters indicate alphabetical sorts, uppercase indicating case sensitivity;
\end_layout
\begin_layout Itemize
\family roman
\series bold
+, -
\series default
indicate numerical sorts.
\end_layout
\begin_layout LyX-Code
\family roman
The
\emph on
next
\emph default
and subsequent rows of a table following a sort specification are sorted.
For neat alphabetical sorts involving inter-filed mixed case, specify columns
twice, e.g.
\series bold
1a1A
\series default
for an AaBbCc ... sort, or
\series bold
1a1Z
\series default
for an aAbBcC ... sort.
\end_layout
'''
def helpnote(hv):
if hv > 1:
return header + version
else:
return header + tail
header = r'''\begin_layout LyX-Code
\family roman
\series bold
.sort list
\end_layout
'''
version = r'''\begin_layout LyX-Code
\family roman
Version 1.0 (16 December 2012) Allow secondary, tertiary, etc. sorts.
\end_layout
\begin_layout LyX-Code
\family roman
Version 0.2 (13 December 2012) Include -a and -n options.
\end_layout
\begin_layout LyX-Code
\family roman
Version 0.1 (8 November 2012)
\end_layout
'''
tail = r'''\begin_layout LyX-Code
\family roman
Sort one or more lists and sub-lists.
\begin_layout LyX-Code
\end_layout
\begin_layout LyX-Code
\family roman
\series bold
Global options
\end_layout
\begin_layout LyX-Code
\family roman
\series bold
-h --help
\series default
show this help message.
\end_layout
\begin_layout LyX-Code
\family roman
\series bold
-v --version
\series default
show version information.
\end_layout
\begin_layout LyX-Code
\family roman
\series bold
-a
\series default
sort across changes of list-type.
\end_layout
\begin_layout LyX-Code
\family roman
\series bold
-c
\emph on
n
\emph default
\series default
\emph on
n
\emph default
= number of characters in sort-key
strings; default is 20 characters.
\end_layout
\begin_layout LyX-Code
\family roman
\series bold
-n --notes
\series default
make LyX's (yellow) notes sortable; notes are sort-neutral by default.
\end_layout
\begin_layout LyX-Code
\end_layout
\begin_layout LyX-Code
\family roman
A change of list type at the top level (e.g.
from
\family sans
Itemize
\family roman
to
\family sans
Description
\family roman
) usually halts a sort; the
\series bold
-a
\series default
option ensures the sort continues across the list-type boundary.
(The result is generally a mess, but a sorted one.) Sorting always
continues across list-type boundaries in sub-lists (but, again,
generates a sorted mess).
\end_layout
\begin_layout LyX-Code
\end_layout
\begin_layout LyX-Code
\family roman
\series bold
Local options
\end_layout
\begin_layout LyX-Code
\family roman
A sort specification is a sequence like
\series bold
1a2a3+
\series default
where the number indicates the list level (1 = first or top level, 2 =
sub-list, etc.) and the qualifying letter or sign indicates the kind
of sort. A specification may involve from one to (potentially) all six list
levels; if desired, only sub-levels may be sorted. For secondary, tertiary ...
sorts, separate the sort specs. by a solidus, e.g.
\series bold
1a2a3+/1A2A
\series default
.
\end_layout
\begin_layout Itemize
\family roman
\series bold
a, A, +
\series default
indicate ascending sorts;
\end_layout
\begin_layout Itemize
\family roman
\series bold
z, Z, -
\series default
indicate descending sorts;
\end_layout
\begin_layout Itemize
\family roman
letters indicate alphabetical sorts, uppercase indicating case sensitivity;
\end_layout
\begin_layout Itemize
\family roman
\series bold
+, -
\series default
indicate numerical sorts.
\end_layout
\begin_layout LyX-Code
\family roman
The specification is placed either
\emph on
before
\emph default
or within a
\emph on
top-level
\emph default
item in a list. The
\emph on
next
\emph default
and subsequent items in a list following a sort specification are sorted. If
the
\emph on
kind
\emph default
of list changes (say from
\family sans
Labeling
\family roman
to
\family sans
Description
\family roman
), the sort stops at the change, unless the
\series bold
-a
\series default
option has been invoked.
\end_layout
'''
# Sort the rows of a table according to values in specified
# columns. Part of the pLyX system; not an independent script.
#
# sortable.py
#
# Andrew Parsloe (apars...@clear.net.nz)
#
import re, argparse
from operator import itemgetter
def main(infl, outfl, opts, guff):
'''Sort table rows according to the values in specified columns'''
# no change to LyX header material required
outfl.write(guff)
guff = ''
# parse options; this is unnecessarily complicated but it allows
# for additions to the option list
parser = argparse.ArgumentParser(description = 'Sort notes?', version = '1.0')
parser.add_argument('-n', '--notes', dest='n', action ='store_true', \
default = False, help="Makes LyX's (e.g. yellow) notes sortable")
sort_notes = parser.parse_args(opts).n
flex_sortable = r'\begin_inset Flex .sort table'
row_start = r'<row'
cell_start = r'<cell'
cell_end = r'</cell>'
row_end = r'</row>'
table_end = r'</lyxtabular>'
re_lyxcmds = re.compile(r'(\\\w+)|(status (open|collapsed))')
begin_inset = r'\begin_inset'
end_inset = r'\end_inset'
begin_text = r'\begin_inset Text'
bottom = r'bottom'
botline = r'bottomline="true" '
topline = r'topline="true" '
# table_status
# -1 = read & write lines unaltered; 0 = no marker yet; 1 = get sort spec;
# 2 = find next row; 3 = this row; 4 = this cell; 5 = ERT|Note inset
table_status = 0
current_row = ''
rows = []
for line in infl:
if table_status == 0: #no marker yet
outfl.write(line)
if flex_sortable in line:
table_status += 1
sort_spec = ''
insets = 1
elif table_status == 1: #get sort spec
outfl.write(line)
if begin_inset in line:
insets += 1
elif end_inset in line:
columns = re.findall(r'\d+',sort_spec)
colsorts = re.findall(r'[-+AZaz]',sort_spec)
numsorts = len(columns)
qeys = ['' for i in range(numsorts)]
updn = [False for i in range(numsorts)]
# reverse order since primary sort is last one done
cols = [int(columns[numsorts-i-1]) for i in range(numsorts)]
sort_type = [colsorts[numsorts-i-1] for i in range(numsorts)]
insets -= 1
table_status += 1
elif re_lyxcmds.search(line):
continue
else:
sort_spec += line.strip()
elif table_status == 2: # find next row
if row_start == line[:4]:
current_row = line
colnum = 0
table_status += 1
# end of table
elif table_end in line:
for i in range(numsorts):
rows.sort(key=itemgetter(i), reverse = updn[i])
numrows = len(rows)
r = 0
for item in rows:
r += 1
if r == numrows:
# restore the bottom hrule
item[numsorts] = item[numsorts].replace(topline, topline + botline)
outfl.write(item[numsorts])
outfl.write(line)
table_status = 0
current_row = ''
rows = []
else:
outfl.write(line)
elif table_status == 3: # this row
if cell_start == line[:5]:
colnum += 1
if bottom in line:
line = line.replace(botline, '') # del bottom hrule
current_row += line
if colnum in cols:
table_status += 1
qey = ''
elif row_end == line[:6]:
current_row += line
table_status -= 1
rows.append(qeys + [current_row])
else:
current_row += line
elif table_status == 4: # this cell
current_row += line
if cell_end == line[:7]:
c = -1
for i in range(cols.count(colnum)):
# in case same col. sorted more than once
c = cols.index(colnum, c + 1)
if sort_type[c] in 'az':
qeys[c] = qey.lower()
elif sort_type[c] in 'AZ':
qeys[c]= qey
elif sort_type[c] in '+-':
qeys[c]= float(qey)
if sort_type[c] in 'zZ-':
updn[c] = True # reverse order
table_status -= 1
elif begin_inset in line:
if 'Text' in line:
continue
elif sort_notes and 'Note Note' in line:
continue
else:
table_status += 1
insets = 1
elif re_lyxcmds.match(line):
continue
else:
qey += line.strip()
elif table_status == 5: # ert inset
current_row += line
if begin_inset in line:
insets += 1
elif end_inset in line:
insets -= 1
if insets == 0:
table_status -= 1
else:
outfl.write(line)
return 1
# Sort a list & sub-lists according to a sort specifation.
# Part of of the pLyX system; not an independent script.
#
# Andrew Parsloe (apars...@clear.net.nz)
#
import re, argparse, tempfile
from operator import itemgetter
re_lyxcmds = re.compile(r'(\\\w+)|(status (open|collapsed))')
re_itemkind = re.compile(r'\\begin_layout (Labeling|Enumerate|Description|Itemize)')
begin_inset = r'\begin_inset'
end_inset = r'\end_inset'
begin_layout = r'\begin_layout'
end_layout = r'\end_layout'
flex_list = r'\begin_inset Flex .sort list'
item_desc = r'\begin_layout Description'
item_enum = r'\begin_layout Enumerate'
item_label = r'\begin_layout Labeling'
item_ize = r'\begin_layout Itemize'
begin_deeper = r'\begin_deeper'
end_deeper = r'\end_deeper'
end_body = r'\end_body'
def main(infl, outfl, opts, guff):
'''Sort (nested) lists.'''
parser = argparse.ArgumentParser(description = 'Sort notes?')
parser.add_argument('-n', '--notes', dest='n', action ='store_true', \
default = False, help="Makes LyX's (e.g. yellow) notes sortable")
parser.add_argument('-a', action ='store_true', \
default = False, help="Sort across list-type boundaries")
parser.add_argument('-c', action ='store', type = int, default = 20, \
help="No. of chars in the sort key")
sort_notes = parser.parse_args(opts).n
sort_all = parser.parse_args(opts).a
nchars = parser.parse_args(opts).c
def get_list_kind(infile, outfile):
'''Itemize? Labeling? Enumerate? Description?'''
for line in infile:
# find item kind
if re_itemkind.match(line):
temp = r'\begin_layout ' + \
re_itemkind.match(line).expand(r'\1')
return temp
else:
outfile.write(line)
def sort_list(infile, outfile, dpth, sorter, updn, list_kind):
'''Do the sort.'''
def include_inset(notes):
'''Include insets without sorting unless notes is True.'''
count = 1
temp = nqey = ''
for line in infile:
temp += line
if begin_inset in line:
count += 1
elif end_inset in line:
count -= 1
if count == 0:
return temp, nqey
elif notes:
if re_lyxcmds.match(line):
continue
else:
nqey += line.strip()
#######################################
list_type_change = False
layouts = 1
items = []
qey = ''
if list_kind == None:
list_kind = ''
current_item = list_kind + '\n'
for line in infile:
# get item & sort key
if begin_deeper in line:
current_item += line
lkind = get_list_kind(infile, outfile)
current_item += sort_list(infile, outfile, dpth + 1, sorter, updn, lkind)
items[-1] = (qey, current_item)
elif end_deeper in line:
items.sort(key=itemgetter(0), reverse=updn[dpth])
cur_items = ''
for i in range(len(items)):
cur_items += items[i][1]
cur_items += line
return cur_items
elif begin_inset in line:
current_item += line
if sort_notes and 'Note Note' in line:
tmp1, tmp2 = include_inset(sort_notes)
else:
tmp1, tmp2 = include_inset(False)
current_item += tmp1
qey += tmp2
elif begin_layout in line:
# list-type change; continue sorting?
if list_kind in line or \
(re_itemkind.match(line) and (dpth > 0 or sort_all)):
current_item = line
layouts = 1
qey = ''
else:
items.sort(key=itemgetter(0), reverse=updn[dpth])
cur_items = ''
for i in range(len(items)):
cur_items += items[i][1]
cur_items += line
return cur_items
elif end_body in line:
items.sort(key=itemgetter(0), reverse=updn[0])
cur_items = ''
for i in range(len(items)):
cur_items += items[i][1]
cur_items += line
return cur_items
elif end_layout in line:
current_item += line
layouts -= 1
if layouts == 0:
if sorter[dpth] in 'az':
qey = qey.lower()[:nchars]
elif sorter[dpth] in 'AZ':
qey = qey[:nchars]
elif sorter[dpth] in 'nN':
qey = '0'
elif sorter[dpth] in 'PM':
qey = float(qey)
items.append((qey, current_item))
elif re_lyxcmds.search(line):
current_item += line
else:
current_item += line
qey += line.strip()
###############################################
# write the prelims to file
outfl.write(guff)
list_status = 0
for line in infl:
# find sort spec. inset
if list_status == 0:
outfl.write(line)
if flex_list in line:
list_status += 1
sort_spec = ''
insets = 1
#get sort spec.
elif list_status == 1:
outfl.write(line)
if begin_inset in line:
insets += 1
elif end_inset in line:
insets -= 1
if insets == 0:
# groom sort_spec to std form
sp = sort_spec.replace('+', 'P')
sp = sp.replace('-', 'M')
specs = sp.split('/')
repeats = len(specs)
for s in range(repeats):
specs[s] = re.sub(r'\W', '', specs[s])
levels = [[] for i in range(repeats)]
sort_type = [[] for i in range(repeats)]
for i in range(repeats):
levels[i] = [int(n) for n in re.findall(r'(\d)', specs[i])]
sort_type[i] = re.findall(r'\d(\w)', specs[i])
sorting = ['N' for j in range(6)]
ftemp = ['' for i in range(repeats+1)]
ftemp[0] = infl
ftemp[repeats] = outfl
for i in range(repeats):
# do lesser sorts before primary
for j in levels[repeats - i - 1]:
k = levels[repeats - i - 1].index(j)
sorting[j - 1] = sort_type[repeats - i - 1][k]
if i < repeats - 1:
ftemp[i+1] = tempfile.TemporaryFile(mode = 'w+t')
updn = [s in '-Zz' for s in sorting]
depth = 0
list_kind = get_list_kind(ftemp[i], ftemp[i+1])
ftemp[i+1].write(sort_list(ftemp[i], ftemp[i+1], depth, sorting, updn, list_kind))
if i > 0:
ftemp[i].close()
if i < repeats - 1:
ftemp[i+1].seek(0)
list_status = 0
elif re_lyxcmds.search(line):
continue
else:
sort_spec += line.strip()
else:
outfl.write(line)
return 1
pLyXSorting(compressed).lyx
Description: Binary data
