Hello colleague, i somehow agree, that the pieces are there
- toolkits like Gtk/tk - datastructures like quadtrees or other spacial trees - drawing with cairo or GL like drawing - coordinate transformation - 'raw' plotting like Winston - your example of selection by box but they are not assembled in a way that i would consider simple to use (like an API to attach a call back function, if a point in a plot is hit with a mouse pointer).
