Hello back, and sorry for not answering earlier.
> To return to your point about extending Org's link format to arbitrary > metadata: I think we need to be clear about what "arbitrary" actually > means here. The current system is already extensible through user > customizations—that's one of Org's great strengths. Thx that's a good question. I'm talking about arbitrary metadata more from the point of view of emacs package authors rather than normal users. As you say, the current format enables to embed arbitrary metadata inside links anyway. The idea is rather to create a unified way to add metadata that would be hopefully used by future package authors, for instance to embed relevant data to make their own link types work --- or even better, to remove the need to create a new link type. I list here a couple benefits of such unification. That's just at the top of my head but I guess it's quite clear why that would be smth desirable. - remove the need for new link types (in some cases) and help with compatibility issues/interoperability. As of now, package authors usually create their own link type and come up with their own formatting to embed metadata. Usually, they stack the metadata they need one after the other, separated by a custom separator. Here is a link from the org-pdftools (https://github.com/fuxialexander/org-pdftools) package for instance: [[pdf:filename.pdf::6++8.94]] But I would prefer something that looks like this: [[file:~/file.pdf::(:page 5 :annotation 8.94)]] So that even if you dont have the package installed or it became ummaintained, you can still open the link (discard the unrecognized :annotation property and simply open at page 5), also other authors can create multiple pdf link packages that work nicely with one another, etc. - fallback options. For instance, a pdf link could contain two fields :highlight (containing the edges of the region to highlight) and :text (containing the string text to highlight). A natural logic to apply at link opening is then: if highlighting a given region defined by its edges is possible then do it, otherwise if it doesnt work (depending on the used browser/pdf viewer) then search for the string text and highlight it instead. If everyone uses the same keywords, one could generalize such logic directly for many link types at once. - possibility to extend the power of links. For instance, embed multiple links into one (as Max suggested, I also thought about this at some point), create links that will just run arbitrary elisp when open, and so on. > You know, this might just come down to different goals and use > cases. And maybe the thread title doesn't quite capture what we're > each talking about—though my input has been about arbitrary data in > Org links and the architecture behind it. I think the same. I would say a database approach for links is something close to perfection as we know the benefits of one centralized database that everyone access (I really like nixos for that for instance). However I think this does not adapt well to the workflow of >99% of users: if you organize your whole workflow with those very efficient links then that's good, but otherwise if you want to work with hybrid solutions (say you sometimes move your files from outside your computer, or you want to have some directory hierarchy for X and Y reason) then it will feel quite uncomfortable on the long run. The good point about linkin-org workflow (and therefore metadata directly inside the link) is that it does not get in your way, and you can use any other workflow in combination. BTW Jean Louis I'm wondering what you think about such links for your database workflow? [[file:<path to database>::(:database-id <id to find in database> :file-id <id of the linked file>)]] Not that I suggest you to change as you seem to have a very efficient workflow there, just asking if there would be value. With this link (and if I understood what you do), even if the database or the file changes place, you can still retrieve it with the ids: if the database is not to be found at <path to database> then apply the linkin-org retrieval mechnism (eg, look for the id inside <paht to database> to retrieve the lost database), or in the worst case (that is, the database is lost for good or the file is not in the database/its path is broken) then you can still retrieve the file with the :file-id property (and then fix the corresponding database entry). > What happens when the file itself changes? Not just moves or renames, > but the actual content. Your system resolves the file location, but > the link has no way to know if the file has been modified, what > version it is, or whether it still contains what the link originally > referenced. The ID identifies the file, not the specific information The linkin-org system has not the ambition to track file versions or so. It's thought to be a quite primitive, simple way to access your data in a reliable way from your written notes. More involved stuff could be added, but in further versions/other packages. As of now, if the user wants its links to point to another file, then he/she cuts the id from the old file and paste the id inside the name of the new file. Julien "Jean Louis" <[email protected]> writes: > * Julien Dallot <[email protected]> [2026-06-23 09:26]: >> >> Hello back, >> >> we are moving away from the original goal of this thread. > > You know, this might just come down to different goals and use > cases. And maybe the thread title doesn't quite capture what we're > each talking about—though my input has been about arbitrary data in > Org links and the architecture behind it. > > Perhaps it would help if you defined what you mean by "arbitrary" more > precisely. That might clear up a lot of the confusion. > > I think my use case is pretty unique here. I work daily with arbitrary > data that's connected to links—and those links appear in Org files but > also get used in other environments. Each of my links has 94 defined > static metadata fields, plus unlimited additional metadata through > properties, tags, and other classifications. > > With almost 100,000 of these links, all related to each other, to > files, and to various kinds of resources, I have some real-world > experience with handling arbitrary data in practice. > > So if you're talking about limited data—data that fits within a > predefined set of fields that you decide upfront—then maybe we're > talking about different things. And that's totally fine! I'd > understand your goals better if that's the case. > > But if you genuinely want to discuss arbitrary data—truly flexible, > extensible, queryable metadata—then I'm here. It's what I work with > every day, and I'd be happy to share how I approach it. > >> If possible, I would like to focus back on the current matter, which is to >> extend org's existing link format to arbitrary metadata. > > To return to your point about extending Org's link format to arbitrary > metadata: I think we need to be clear about what "arbitrary" actually > means here. The current system is already extensible through user > customizations—that's one of Org's great strengths. But for larger > groups or long-term knowledge management, I don't see embedding > metadata directly into links as a scalable solution. It creates > maintenance headaches, makes querying difficult, and complicates > sharing. That said, if you find ways to genuinely contribute > improvements to Org's link handling, I'm welcoming of that. My only > small nitpick is that I don't agree what you're proposing qualifies as > "arbitrary" metadata—a link will always be constrained by its format. > >> The "linkin-org" workflow is fundamentally different than standard >> link systems, as in obsidian or in the database workflow that Jean >> Louis mentions, in the sense that the link's metadata are not >> updated in the background, but *only when the user opens a link*. It >> acts in reaction to the user's actions instead of passively in the >> background --- which aligns with org's and emacs's ways of doing, >> and make the whole system scale well. >> >> The key idea is that linked files have ids in their names. >> For instance: >> filename--20250425T155433.txt >> and a linked line contains a commented id, for instance (in a python syntax): >> # [id:20250425T160020] link gets you there >> Technically, this turns your file system into a distributed hash >> table. > > I see the value in your resolution system—it's clever and addresses a > real problem. It doesn't solve it. > > What happens when the file itself changes? Not just moves or renames, > but the actual content. Your system resolves the file location, but > the link has no way to know if the file has been modified, what > version it is, or whether it still contains what the link originally > referenced. The ID identifies the file, not the specific information > being linked to. So while you've solved file location reliability, you > haven't solved the problem of link integrity when content evolves. A > truly robust link system would also track content changes, versions, > or at least provide some way to verify that the link still points to > the same information it originally referenced. > >> When you try to open a link, 3 simple routines are run in a row >> until one succeeds (from fastest to slowest): 1. check that the path >> inside the link is correct, if yes then we're done 2. if the path is >> incorrect, try to greedily reconstruct the path by local id searches >> (which is fast even when run with elisp, and always works if the >> file was just renamed but not moved) 3. if all above have failed, >> perform an extensive search of the file's id inside the computer (or >> the directories marked for search) (slow with elisp, but fast by >> delegating to fd for instance). > > Sounds like external management. I can't see metadata in the link... > >> linkin-org proposes a fast, safe way to create links: store your >> file (call linkin-org-store in dired) and paste the link >> wherever. Doing so added an id to your file and moved it to a >> location that the extensive will not miss. > > As idea I agree on that, but it seems I may understand something > different out of it then what you think. > > I actually do the same on my side. > > - there is file, video, or any type of link, picture, or WWW > hyperlink, or page in the PDF file, many types, and when I store it, > I forget where it is because I do not use file names or directories > to remember where the file is. It is simply stored, and remains > static somewhere on the file system. My files look something like > this: > > /home/data1/protected/hyperscope/1/2/7/4/8/8/2026/06/2026-06-25/John-Gelasi-Card-Sampler-2EFA97CF-026F-43A6-AC66-63137757CA60.pdf > > and that directory is automatically generated. I can still adjust > the file name and later freeze it. > > - Once file has been indexed, I can move the ID of the file anywhere > else in the tree of elementary objects. > > - I do not move files ever again, unless I wish to delete them, but > that happens usually if I delete the ID in the index. > >> With this method, you have truly reliable links at the cost of >> following some simple rules (see the 4 golden rules in linkin-org's >> readme) and seeing ids in your file names. > > I agree in principle on that, just that I don't find it managaeable > without database. > >> In my view, no centralized solution is suited (or at least should >> not be the default) as either it breaks easily (try moving a file in >> your dropbox app. Once your laptop synced, centralized solutions >> break) or it needs dedicated suite of softwares that will take care >> of the centralized update, which is not in emacs's philosophy (and >> not in mine as well). > > With this principle of storing the file statically, and forgetting > about the storage but using index to find it, you do not need to move > it in any remote system. It would simply be somewhere remotely and you > never need to think of its file path. You think of sets or categories > which are analogous to directories in your index. I do not look into > file system, I am looking into my index only. I call it index, yet it > is more than index, it is the Dynamic Knowledge Repository. > > See here: > > TECHNOLOGY TEMPLATE PROJECT OHS Framework : > https://www.dougengelbart.org/content/view/110/460/ > > There are several ways how we agree on some principles. I just have > intention that you do not run into something not manageable on long > term.
