Here's an older paper about an innovative hex editor: Abstract The analysis of binary data remains a challenge, especially for large or potentially inconsistent files. Traditionally, hex editors only make limited use of semantic information available to the user. We present an editor that supports user-supplied semantic data definitions. This semantic information is used throughout the program to realize semantic data visualization and data exploration capabilities not present in similar systems. Visualization and human-computer interaction techniques are applied. We show that this makes recognizing the structure of unknown or inconsistent data much more effective. Our approach demonstrates concepts that can be applied to the visual analysis of raw data in general.
https://www.researchgate.net/publication/220836091_Vide_An_editor_for_the_visual_exploration_of_raw_data -----Original Message----- From: Mike Beckerle <mbecke...@apache.org> Sent: Monday, November 1, 2021 10:31 PM To: dev@daffodil.apache.org Subject: Re: Hex editor operations This is a good list. One thing I think is important is that often one is dealing with hex data, but one needs to consider bit fields that do not respect byte boundaries. I suggest we need a way to expand a single byte into a small presentation of 8 bits, allowing editing of just those 8 bits individually, is of value as part of a hex-editor. Perhaps that is what you meant by Mask/Set operations? I think switching to a full-fledged all 1's and 0's display mode is only for entirely non-byte-oriented data. Anything byte-oriented people users will want to use hex, and occasionally if they need to flip bits, a way for them to expand a byte or small run of bytes to 1's and 0's, but then collapse back to hex is likely very helpful. But I also suggest creating a minimal hex editor version first, then we get experience with it. Any little paper scribble exercises we find ourselves having to do on the side of using the editor, those are good candidates for things the UI should directly support. E.g. I have written things down on paper or in a text editor like: BF.32.A5.AC.(01|10 1101.101|1 0001) In a real UI these distinctions could be done quite differently. But the distinctions I'm making are dots separating bytes, pipes separating bit fields of length 2, 9, and 5 respectively, and parentheses indicating bytes expanded out to a bits region for display, e.g., so that 01 for 2 bits isn't confused with hex 01 8 bits, and here I have each nibble of 4 bits space-separated. Ultimately, we need a graphical means providing: (a) a way of escaping from hex to bits for just a small region of the data where you care about the partial byte fields. (b) a way of setting off bit fields from each other, that doesn't entirely lose the separation of hex digits. Then if say, that right-most bit field value is incorrect, you should be able to set editing focus on a bit and flip it. I would add to the "parse until this byte", or "start from this byte" to enable narrowing at both the start and end - parse just these bytes (in an identified region by start and end). The use case I have in mind for this is sort of like unit testing. You narrow the data to just one part, then you specify to parse it not with the root element of the DFDL schema, but with a sub-element that you want to test. On Mon, Nov 1, 2021 at 6:19 AM John Wass <jwa...@gmail.com> wrote: > Some thoughts on ways the hex editor would interact with an input > stream and the debugger. I visualize this using a context sensitive > menu in the hex editor as an entrypoint, where these are some of the > operations that could be offered. > > 1. Add/Delete/Mask/Set > * Individual bytes > * Blocks of bytes > 2. Copy/Paste bytes > * Use system clipboard for interoperability 3. Find/Replace bytes > * In a selection > * Across entire file > 4. Set breakpoints on bytes > * Stops execution at related point in schema > * Would require custom additions to DAP backend 5. Set run options > on bytes > * Debug-to this byte (and break) > * Start from this byte (skipping previous) > > The idea with breakpoints and run operations is that they would behave > as a normal breakpoint in code would. I don't know if this is > functionality for the first pass of the editor, but it is definitely > something to keep in mind while designing that first pass. > > What other operations could be supported? >
