I wrote this script to traverse a directory and process all the image files
of a certain type. I am using this to process a large set of differential
interference contrast images taken on an optical microscope, but I imagine
this script could easily be adapted for other uses (like AFM images).
Mainly I'm interested in picking out and marking small spots in the image.
I then save the processed images and count the spots using a separate
program (DotCount), which can batch process a whole folder very easily.
I did this in the PyGwy console because I'm running on Windows which
doesn't have a standalone python/gwyddion module. I tried writing
something on Linux but had trouble with the dependencies for the standalone
gwyddion python module. And I wasn't really in the mood to fuss with
dependencies (shame on me!). I imagine this code could be adapted to run
as a standalone python script on Linux without too much trouble.
The main features:
- The script prompts the use on the first image load and first image save.
After that it runs with those settings automatically.
- only filters images of a certain type. In my case this is BMP, but
should easily handle other pixmap formats, and with a little adaptation
most other formats Gwyddion supports.
- does a polynomial level on data. Originally it was a plane level, but I
found a polynomial level was much better for my data.
- I'm not concerned with precise height information. More with making the
images have similar levels so the thresholding algorithm of dotcount works
predictably. I normalize the data a couple times to achieve this, which
might not be ideal for other people.
- Does an FFT 1D filter in the horizontal and then vertical directions. I
do this to remove a periodic noise introduced by the CCD camera. The
filter weights are set to remove this noise. It is likely other will want
to change the weights. I've left in some code that generates a 2D fourier
transform and extracts a DataLine from it. I then copied that DataLine
into Excel and picked the indices that needed reweighting. Your mileage
may vary, but this worked well for me since the frequency of the CCD noise
was constant across all images.
The MAJOR issue I'm having, and would appreciate some help with:
- This script can't process more than about 50-100 images without crashing
Gwyddion. I haven't nailed down the exact number. Based on the GTK errors
I'm getting this is almost definitely a memory management issue (unable to
allocate memory). When I added the FFT filters which declare a couple
extra DataFields and DataLines, the problem got worse. Yeti has warned
people about trying to do what I'm doing in the pygwy console, and I'm
guessing it's because of this. I spent a long time trying to figure out
how to release memory (ie: garbage collection), but Python is supposed to
do this automatically, and doing explicitly doesn't seem to work. It
occurs to me that perhaps Gwyddion needs to release the memory and there's
a function I need to call, but I can't figure out what that is. Removing
from the data browser is not sufficient.
- The other issue I have, but know there's no easy solution for is that I
can't do the dot counting in Gwyddion using this script. Yeti has already
covered this though, and it basically requires someone to write some new
Python wrappers for the native C functions. Sadly, I don't really have the
expertise, nor the time to make up for my lack of expertise, to write the
wrappers myself. Perhaps if I get a chance to revisit this problem in the
future.
- Finally, this is a tiny issue that is more annoying than anything else.
For some reason when I save, close, and reopen scripts in the Pygwy
console, carriage returns keep being inserted between my lines of code.
After a few cycles a hundred lines of code become a thousand lines of
code. Doesn't break my script just makes it a pain to work on. I
periodically have to delete all the extra blanks lines to make it
readable. Notepad++/TextFX is a real savior here.
Anyway, I hope this is all helpful to somebody, and maybe someone can help
me sort out the memory issue.
Here is the script:
import os, gwyutils, gwy, time
# Released under GPLv2.
# This gwypy script will traverse a directory and process all files of a
certain file type using Gwyddion.
# You can insert any pygwy processing code you've developed in the section
below.
# When I run this Gwyddion hangs/freezes after ~50-100 images being opened,
processed, and saved.
# This is probably why Yeti recommends deveoping a script in standalone
Python.
# However, if you are on Windows, this is not an option since gwy.so
(python module)
# is not compiled/working (as of Gwyddion 2.33). I think this script would
port to a standalone
# python script on Linux/Mac where the gwy.so module is available.
#set slope threshold for grain marking.
slope_threshold = 20
#set height deviation percentage from mean for grain marking. 50 should
turn this off.
above_dev_perc=50
below_dev_perc=50
#a counter
i=0
#set image format to import. will only load files of this type.
img_format='bmp'
#set input file directory
#The filename parsing routine sometimes fails unless there are double
slashes before the last directory. My python knowledge is limited, so I'm
not sure why that happens. Escape characters most likely.
in_directory = "C:\INPUT\\DIRECTORY"
print in_directory
#Output directory and filename format
out_directory = "c:/processed/"
out_file_name = "%s-marked.png"
basename = out_directory + out_file_name
print basename
for dirname, dirnames, filenames in os.walk(in_directory):
#print path to all subdirectories first.
for subdirname in dirnames:
print os.path.join(dirname, subdirname)
#print path to all filenames.
#print filenames
#This is a debug line, really useful for debugging processing routine
using a single image instead of a whole folder.
#filenames = {"C:\INSERT\DEBUG\FILE\PATH\HERE"}
#Then uncomment this line...
for filename in filenames:
#generate full path
path = os.path.join(dirname, filename)
#if file extension matches desired image format open and do analysis
if path.split(".")[-1].lower() in [img_format]:
print path
#load file
if i == 0:
c = gwy_file_load(path, RUN_INTERACTIVE)
else :
c = gwy_file_load(path, RUN_IMMEDIATE)
#Add container to data browser.
gwy.gwy_app_data_browser_add(c)
#add original path to container
if not c.contains_by_name('/filename') :
c.set_string_by_name('/filename', path)
#make container visible (window should appear). Pretty sure this
code is redundant.
#if not c.contains_by_name('/0/data/visible') :
# c.set_boolean_by_name('/0/data/visible', 1)
#get directory of datafields where key is key in container
dfields = gwyutils.get_data_fields_dir(c)
for key in dfields.keys():
# get processed datafield object
datafield = dfields[key]
# create new grainfield by using active datafield
grainfield = datafield.new_alike(True)
# retrieve datafield number in container from key (for example
'/0/data')
datafield_id = key.split('/')[1]
# set palette of datafield
c.set_string_by_name("/%s/base/palette" % datafield_id, "Gray")
#get filename without directory or file extension. I think there
might be a pre-built python routine for doing this, but I found out too
late.
filename = c.get_string_by_name('/filename')
filename = filename.rsplit('\\',1)[1]
filename = filename.rsplit('.',1)[0]
# BEGIN PROCESSING CODE HERE
# call 'level' process module as non-interactive DOESN'T WORK in
this code for some reason. Works from the console command line and in
other scripts, though. Problem is probably in this code somewhere.
#if i == 0:
# gwy.gwy_process_func_run("level", c, gwy.RUN_IMMEDIATE)
#else :
# gwy.gwy_process_func_run("level", c, gwy.RUN_IMMEDIATE)
#Plane Leveling Routine to replace above non-working level
routine.
#old_plane = datafield.fit_plane()
#print old_plane #debug
#datafield.plane_level(old_plane[0],old_plane[1],old_plane[2])
#new_plane = datafield.fit_plane()
#print new_plane #debug
#datafield.normalize()
#Polynomial Level Routine (2nd order in x and y)
datafield.subtract_polynom(2,2,datafield.fit_polynom(2,2))
datafield.data_changed()
#Begin Fourier Filter Code.
xres = datafield.get_xres()
yres = datafield.get_yres()
#Declare empty weight lists
hweights = gwy.DataLine(xres//2,1.0,True)
vweights = gwy.DataLine(yres//2,1.0,True)
#Generate 2D Transform and get frequency domain representation.
This can be useful for deciding what indices to set the weights for the
filters.
#fft_rfield = gwy.DataField.new_alike(datafield, False)
#fft_ifield = gwy.DataField.new_alike(datafield, False)
#datafield.a_2dfft_raw(datafield, fft_rfield, fft_ifield,
ORIENTATION_VERTICAL)
#fft_rfield.get_column_part(hweights, 0, 0, xres//2)
#fft_rfield.get_column_part(vweights, 0, 0, yres//2)
#print vweights.get_data()
#Explicitly set 1D Horizontal FFT filter weights
hweights.fill(0)
hweights.part_fill(103,118,.3)
hweights.part_fill(106,110,.7)
hweights.part_fill(107,109,.9)
hweights.part_fill(108,108,1)
#print hweights.get_data()
#Explicitly set 1D Vertical FFT filter weights
vweights.fill(0)
vweights.part_fill(119,121,.5)
vweights.set_val(120,1)
#A datafield for storing the filter datafield
filtered = gwy.DataField.new_alike(datafield,False)
#Generate horizontal 1D Fourier Filter
datafield.fft_filter_1d(filtered, hweights,
gwy.ORIENTATION_HORIZONTAL, gwy.INTERPOLATION_LINEAR)
#Subtract the filter field from the data field. ie: Apply
Filter.
result_field_h = gwy.DataField.new_alike(datafield, 0)
result_field_h.subtract_fields(datafield, filtered)
filtered.clear()
#Generate vertical 1D Fourier Filter
result_field_h.fft_filter_1d(filtered, vweights,
gwy.ORIENTATION_VERTICAL, gwy.INTERPOLATION_LINEAR)
result_field_v = gwy.DataField.new_alike(datafield, 0)
result_field_v.subtract_fields(result_field_h, filtered)
#Insert result field in new container and make it visible
#result_container = gwy.Container()
#result_container.set_object_by_name('/0/data', result_field_v)
#gwy.gwy_app_data_browser_add(result_container)
#result_container.set_boolean_by_name('/0/data/visible', 1)
#Overwrite original datafield
result_field_v.copy(datafield, 0)
datafield.data_changed()
#Begin Grain Marking Code
# mark grains by threshold/slope
grainfield = datafield.new_alike(1)
inv_grainfield = datafield.new_alike(1)
slope_grainfield = datafield.new_alike(1)
#Mark grains dev_perc above or below mean.
datafield.grains_mark_height(grainfield,50+above_dev_perc,0)
datafield.grains_mark_height(inv_grainfield,50-below_dev_perc,1)
#Mark Grains with slope larger than slope_threshold
datafield.grains_mark_slope(slope_grainfield,slope_threshold,0)
slope_grainfield.normalize()
inv_grainfield.normalize()
grainfield.normalize()
grainfield.max_of_fields(grainfield, inv_grainfield)
grainfield.max_of_fields(grainfield, slope_grainfield)
grainfield.plane_level(.5,0,0)
datafield.max_of_fields(datafield, grainfield)
datafield.normalize()
# request redraw before export
datafield.data_changed()
# END PROCESSING CODE HERE
# export datafield to png, for the first time show export dialog
# A little more specific naming routine. Preserves original
filename in exported name.
if i == 0:
gwyutils.save_dfield_to_png(c, key, basename % filename,
gwy.RUN_INTERACTIVE)
else :
gwyutils.save_dfield_to_png(c, key, basename % filename,
gwy.RUN_NONINTERACTIVE)
i += 1
# request redraw datawindows with datafields
datafield.data_changed()
time.sleep(1)
#Close container/window after export. Gives some visual feedback
when dealing with large number of windows.
gwy_app_data_browser_remove(c)
#clean-up. get list of containers and close them all. This is probably
pointless given the remove routine above.
cons = gwy.gwy_app_data_browser_get_containers()
for c in cons:
gwy_app_data_browser_remove(c)
------------------------------------------------------------------------------
DreamFactory - Open Source REST & JSON Services for HTML5 & Native Apps
OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
Free app hosting. Or install the open source package on any LAMP server.
Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
_______________________________________________
Gwyddion-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/gwyddion-users
