Hi Denis, Are you still around?
I have been thinking: reverting our heavily-patched libtorrent to work with both attached files and twister posts is probably going to take quite a bit of work... What if, instead of doing that, we link again to another copy of libtorrent (a system library provided by distribution, unpatched upstream) just for handling the files users may want to attach? twister executable code size in memory would increase, but not a big deal. we would also need to open another port. However enforcing per-user, per-file limits would be easier. Also, completely disabling file attachment support doesn't disrupt what already works (for example, torrent is not permitted in my university's network) Food for thought. regards, Miguel On Sun, Sep 21, 2014 at 9:14 AM, Miguel Freitas <[email protected]> wrote: > Hi Denis, > > Thanks a lot for the thoughts on this issue and for concisely expressing the > new proposal. This is a great idea! > > One thing I'd like to comment is the new "contract" that is embodied into > this proposal. I mean the rule to decide where data is stored/duplicated in > twister network. > > Currently, twister "contract" is that nodes should join the effort in > storing posts from other users and do so by getting randomly assigned to > keep some posts on DHT entries or to join some user's torrents. Given the > size limits on posts I believe this is a fair agreement: one knows that his > HD space won't be abused by others but he must still contribute a little > bit. In exchange he gets the ability to go offline and have his posts still > available somewhere else, hosted by other users. > > With image or arbitrary file storage the game changes: space occupied on > one's HD may rise fast due to some abuser, or one's may try to store > contents on your computer you disapprove (i don't want to contribute my own > bandwidth and HD sharing any beheading video, for example). > > So what I've suggested Denis is that this new "media" torrent doesn't get to > be governed by the same rules as current "posts" torrent does. That is: > users would never get randomly assigned to join anyone's new "media" torrent > to help sharing the contents, unless he agrees with that. > > My proposal is a simple concept: if you follow some user, then you join his > "media" torrent and helps sharing the files. If you don't follow, you don't > host/share any of his data. > > Additional limitations should be implemented, as Denis noted. That may > include time-to-expire, maximum file or size etc. > > regards, > > Miguel > > PS, implementation detail: since tracker doesn't automatically joins the > media torrent, we should rely on the same standard mechanism of bittorrent > trackers. that is: swarm members should periodically notify the tracker > through DHT of their own IP/port. > > On Sat, Sep 13, 2014 at 7:51 PM, Denis Ryabov <[email protected]> wrote: >> >> In this letter I would like to start discussion of an idea to add "data" >> torrent in additional to existing "messages" torrent (with all user's >> posts). There may be a lot of ways to use this torrent, and first of all we >> can implement files attaching (mainly, images, but it might be videos, >> files, long URLs for a built-in URL shortener, long posts beyond 140 chars >> limit, etc.). We started to discuss it in Russian Twister community few >> months ago, and Miguel Freitas supports this idea as well. >> >> Current things about implementation of this idea: >> >> 1. Twister stores user's messages as separate pieces of the torrent file, >> and we could use the same idea for additional "data" torrent (that is, 1 >> files = 1 piece), but it's better to split large file into several pieces to >> speed up loading from several peers. Usually, piece size of about 256Kb is >> used in large torrent, so we can use this value as well (note that piece >> size limits maximal amount of data that may be stored in the torrent). So, >> each reference to a file in "data" torrent should contain file size and >> starting piece index in the torrent (amount of pieces to load is easily >> derived from known file size). Each piece of data torrent should contain >> data block followed by PGP signature to be sure it's valid content (the same >> way is used currently in Twister's messages torrent). >> >> 2. Main torrent should support new type of messages (in addition to posts, >> RTs, and DMs): message (description) + file information. Additional type >> allows to easily filter messages and display all "file" messages separately >> (like Twitter does in photos/videos tab). This type of messages should >> include following info: >> >> - description message (standard 140 chars limit) >> - start piece number in data torrent >> - file size >> - original file name (optionally) >> - mime type (optionally) >> >> 3. Full data torrent may be quite big in size (e.g. if it contains >> videos), so this torrents shouldn't be joined into swarm to minimize total >> size of data stored on user's nodes, and followers are the only users who >> distribute the torrent. Most likely there should be a way to set maximal >> file size to load/share automatically, and bigger files can be loaded by >> direct request only (e.g. by click on "load" button in client's interface). >> In additional, we can use some time-to-live technique and remove loaded >> files (pieces of the torrent file) if they are not requested for certain >> time (but continue to store on author's Twister instance(s)), or to don't >> load all of very old files (older than time-to-live) by a new follower. >> There is a chance that old files will be "forgotten" if data torrent of >> author's Twister instance is damaged, but maybe it's a reasonable compromise >> between functionality and required storage. Note that Miguel's opinion is >> "The problem is availability: the data might still be there, but if the >> author is offline then for all practical effects the data is lost. That's >> the reason i don't believe in torrents requesting data only by-demand, >> requesting earlier we increase data availability for the whole network. Too >> low availability will kill this cool concept imho.", so any feedback and >> ideas are welcome. >> >> 4. Note that there is a way to send files in direct messages. The idea is >> to encrypt file using a symmetric algorithm with random key and include this >> key in the direct message. >> >> Any response will be greatly appreciated. >> >> ------- >> Best regards, >> Denis Ryabov (twister: @denis) >> >> -- >> You received this message because you are subscribed to the Google Groups >> "twister-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "twister-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. For more options, visit https://groups.google.com/d/optout.
