Speed up trace mode (GSoC Project)

2019-03-23 Thread Mihir Luthra
Hi, I wanted to contribute to the MacPorts documentation. The guide says that it currently is in docboook format with work in progress to change it to adoc. Actually I got a bit confused with this. Where exactly should I add a file in order to create a new section? Like currently there are 7 secti

Re: Speed up trace mode (GSoC Project)

2019-03-24 Thread Mihir Luthra
Hi, I had a few questions regarding Darwin trace library. Darwintrace library being injected, most I/O operations get reimplemented. If a “single” process is working on files, it more or less should call most of these functions again and again. Like that particular process may call open, rename,

Re: Speed up trace mode (GSoC Project)

2019-03-24 Thread Christopher Chavez
Hi Mihir, I am a MacPorts user and port maintainer. I have a suggestion, but am not a member, so do not take my advice as canon. > On Mar 23, 2019, at 1:52 PM, Mihir Luthra <1999mihir.lut...@gmail.com> wrote: > > I was thinking to add an 8th section which gives a small high level tour of > the

Re: Speed up trace mode (GSoC Project)

2019-03-25 Thread Mihir Luthra
Thanks for the helpful information ^_^. Wiki seems to be a the right place to put all this information and later (if possible), maybe some quick links on the main website to these wiki pages would be helpful. For now I guess you are right, will put anything I document on wiki. Regards, Mihir

Re: Speed up trace mode (GSoC Project)

2019-03-25 Thread Clemens Lang
Hi, On Sun, Mar 24, 2019 at 08:40:30PM +0530, Mihir Luthra wrote: > I had a few questions regarding Darwin trace library. > > Darwintrace library being injected, most I/O operations get > reimplemented. If a “single” process is working on files, it more or > less should call most of these functio

Re: Speed up trace mode (GSoC Project)

2019-03-25 Thread Mojca Miklavec
On Mon, 25 Mar 2019 at 01:28, Christopher Chavez wrote: > > Hi Mihir, I am a MacPorts user and port maintainer. I have a suggestion, but > am not a member, so do not take my advice as canon. > > > On Mar 23, 2019, at 1:52 PM, Mihir Luthra wrote: > > > > I was thinking to add an 8th section which g

Re: Speed up trace mode (GSoC Project)

2019-03-27 Thread Mihir Luthra
Hi, I have shared my draft application from the GSoC dashboard. Please provide me with feedbacks. :) Also, should I share the link to document here as well? Regards, Mihir

Re: Speed up trace mode (GSoC Project)

2019-03-27 Thread Mojca Miklavec
Dear Mihir, On Wed, 27 Mar 2019 at 17:25, Mihir Luthra wrote: > > Hi, > > I have shared my draft application from the GSoC dashboard. > Please provide me with feedbacks. :) Please note again that I'm not familiar with the contents at all, so I'm providing just some general feedback. What I miss

Re: Speed up trace mode (GSoC Project)

2019-03-27 Thread Mihir Luthra
On Wed, Mar 27, 2019 at 10:22 PM Mojca Miklavec wrote: > Dear Mihir, > > On Wed, 27 Mar 2019 at 17:25, Mihir Luthra wrote: > > > > Hi, > > > > I have shared my draft application from the GSoC dashboard. > > Please provide me with feedbacks. :) > > Please note again that I'm not familiar with the

Re: Speed up trace mode (GSoC Project)

2019-03-28 Thread Mihir Luthra
Hi there, I had a question. Before the build, dependencies are checked. Enabling trace mode hides incompatible versions of the current software being installed or the versions installed by other package managers and more such. The injected darwintrace.dylib will replace file operations if needed t

Re: Speed up trace mode (GSoC Project)

2019-03-28 Thread Clemens Lang
Hi Mihir, On Thu, Mar 28, 2019 at 12:07:20AM +0530, Mihir Luthra wrote: > On Wed, Mar 27, 2019 at 10:22 PM Mojca Miklavec wrote: > > What I miss a bit is some clear definition of deliverables, what > > pieces of code would be suitable enough for merging them into base > > and when. > > > > Backgr

Re: Speed up trace mode (GSoC Project)

2019-03-31 Thread Mihir Luthra
Hi, I need help understanding *static* *int* TracelibRunCmd(Tcl_Interp *in), it has blown my mind, I am totally new with these kevents. I have been trying to understand that from a while now. Till what I understand, the thread responsible for creating server eventually calls tracelib run and then

Re: Speed up trace mode (GSoC Project)

2019-03-31 Thread Mihir Luthra
Hi there, @Mojca Miklavec As I told you before, I will be documenting trace mode alongside with understanding it, I have been working on that https://docs.google.com/document/d/15cVbH6f6hBr9HryJEHUZEbRsToN1BAjY8My-oRstO9A/edit#heading=h.oby3p7ljhsu Sorry I told you I will get it done by the en

Re: Speed up trace mode (GSoC Project)

2019-03-31 Thread Mihir Luthra
Hi Clemens, I was wondering if the trace mode optimisation project would be sufficient for entire summer? Because maximum effort is about understanding the code which needs to be done before gsoc starts. After that maybe the shared memory concept can be achieved within a month. Maybe testing that

Re: Speed up trace mode (GSoC Project)

2019-03-31 Thread Clemens Lang
On Sun, Mar 31, 2019 at 10:06:12PM +0530, Mihir Luthra wrote: > I need help understanding *static* *int* TracelibRunCmd(Tcl_Interp > *in), it has blown my mind, I am totally new with these kevents. > > I have been trying to understand that from a while now. Till what I > understand, the thread res

Re: Speed up trace mode (GSoC Project)

2019-04-04 Thread Mihir Luthra
Hi Clemens, I had some points in mind about mapping more memory when we run out of memory, which I wanted to discuss with you. Also you are right, this project is not at all short. I realise now, many conditions need to be taken care of while doing tasks. Each process shared 2 types of memory. O

Re: Speed up trace mode (GSoC Project)

2019-04-04 Thread Mihir Luthra
Hi, I have made the memory expansion idea precise and filtered out bugs from the one I sent previously. Please check the pdf in the attachment. Regards, Mihir MemoryExpansion.pdf Description: Adobe PDF document

Re: Speed up trace mode (GSoC Project)

2019-04-04 Thread Clemens Lang
Hi Mihir, On Thu, Apr 04, 2019 at 06:25:00PM +0530, Mihir Luthra wrote: > I have made the memory expansion idea precise and filtered out bugs > from the one I sent previously. > Please check the pdf in the attachment. I see you're getting the hang of the difficulties of the project now :) You ha

Re: Speed up trace mode (GSoC Project)

2019-04-05 Thread Mihir Luthra
Hi Clemens, I see you're getting the hang of the difficulties of the project now :) > That’s really encouraging for me to know that you think so ^_^ > You have the right ideas to solve the problem. Do keep in mind though > that CAS will only work up to a word size or a double word size at most,

Re: Speed up trace mode (GSoC Project)

2019-04-07 Thread Clemens Lang
Hi, On Sat, Apr 06, 2019 at 01:05:08AM +0530, Mihir Luthra wrote: > > You have the right ideas to solve the problem. Do keep in mind > > though that CAS will only work up to a word size or a double word > > size at most, i.e. swapping more than 64 bit with CAS atomically is > > probably not going

Re: Speed up trace mode (GSoC Project)

2019-04-07 Thread Mihir Luthra
Hi, I was trying to test the code by making changes but I am stuck with one issue. If I install from git and set it up I always receive this error message. [1] 1) I tried `make` on base code taken as it is from git without checking out to latest version, it showed this error. 2) I tried `make` on

Re: Speed up trace mode (GSoC Project)

2019-04-07 Thread Mihir Luthra
> That's unfortunately not the same, since while you're swapping the > second value, a different thread could swap the first for a different > value again. > I get your point. Generally what comes to the mind in such circumstances is blocking the other thread some way like mostly by spin locks. Bu

Re: Speed up trace mode (GSoC Project)

2019-04-08 Thread Clemens Lang
Hi, On Sun, Apr 07, 2019 at 01:03:12AM +0530, Mihir Luthra wrote: > > Can you pastebin a main.log of a build that fails like this? > > https://pastebin.com/FVdp4WTw The problematic lines are :debug:archivefetch failed verification with key /opt/local/share/macports/macports-pubkey.pem :debug:a

Re: Speed up trace mode (GSoC Project)

2019-04-08 Thread Mihir Luthra
On Tue, Apr 9, 2019 at 1:59 AM Clemens Lang wrote: > Hi, > > On Sun, Apr 07, 2019 at 01:03:12AM +0530, Mihir Luthra wrote: > > > Can you pastebin a main.log of a build that fails like this? > > > > https://pastebin.com/FVdp4WTw > > The problematic lines are > > :debug:archivefetch failed verifica

Re: Speed up trace mode (GSoC Project)

2019-04-12 Thread Mihir Luthra
Hi, I needed some advise regarding ctrie implementation. I was constructing the trie data structure which has to be operated from the address space mapped by mmap(2). In place of “array of next nodes”, I am using “array of offsets”. the void * returned from calling mmap(2), I would type cast it a

Re: Speed up trace mode (GSoC Project)

2019-04-12 Thread Clemens Lang
Hi, On Fri, Apr 12, 2019 at 11:16:46PM +0530, Mihir Luthra wrote: > I was constructing the trie data structure which has to be operated > from the address space mapped by mmap(2). In place of “array of next > nodes”, I am using “array of offsets”. Sounds correct. > the void * returned from calli

Re: Speed up trace mode (GSoC Project)

2019-04-16 Thread Mihir Luthra
Hi Clemens, Kindly provide your suggestions for this. In the path search Ctrie data structure, I categorised the paths with the hash function working like: If I input a path /test/files/abc.h for check Here we have open to abc.h, the hash function simply makes it “12:abc.h" where 12 is the length

Re: Speed up trace mode (GSoC Project)

2019-04-20 Thread Clemens Lang
Hi Mihir, On Tue, Apr 16, 2019 at 11:59:53PM +0530, Mihir Luthra wrote: > Kindly provide your suggestions for this. > In the path search Ctrie data structure, I categorised the paths with > the hash function working like: > > If I input a path /test/files/abc.h for check > Here we have open to ab

Re: Speed up trace mode (GSoC Project)

2019-04-21 Thread Mihir Luthra
Hi, Thanks for the tips. ^_^ This is almost the same way as in the arXiv paper you shared with me. As we are dealing with paths, I saw implementing the above hash function would make the search faster as this happens quite a few times that files with same name have same path length from the root.

Re: Speed up trace mode (GSoC Project)

2019-04-22 Thread Clemens Lang
Hi, On Sun, Apr 21, 2019 at 11:01:09PM +0530, Mihir Luthra wrote: > This is almost the same way as in the arXiv paper you shared with me. > As we are dealing with paths, I saw implementing the above hash > function would make the search faster as this happens quite a few > times that files with sa

Re: Speed up trace mode (GSoC Project)

2019-04-24 Thread Mihir Luthra
Hi, Sorry for the late response. What's your current progress? Do you have some code already? > Currently I am done with a basic Ctrie implementation as suggested in the arXiv paper that is capable of inserting and searching paths. The implementation is still with pointers which I will convert s

Re: Speed up trace mode (GSoC Project)

2019-04-28 Thread Mihir Luthra
Hi Clemens, > What's your current progress? Do you have some code already? > I made a complete offset based ctrie implementation in which any process can insert and search on basis of a shared memory. Kindly provide me with your views on it :) All main code is in [1]. The header file contains

Re: Speed up trace mode (GSoC Project)

2019-04-30 Thread Mihir Luthra
Hi Clemens, For making my implementation of shared memory data structure more space efficient, I was trying to implement a stack which stores offsets to unused locations in the shared memory file. But as stack is being shared it also needs to be edited in a lock free way. While editing stack I nee

Re: Speed up trace mode (GSoC Project)

2019-05-04 Thread Clemens Lang
Hi, On Wed, May 01, 2019 at 11:57:14AM +0530, Mihir Luthra wrote: > For making my implementation of shared memory data structure more > space efficient, I was trying to implement a stack which stores > offsets to unused locations in the shared memory file. But as stack is > being shared it also ne

Re: Speed up trace mode (GSoC Project)

2019-05-05 Thread Mihir Luthra
> > > > > From what I understand from the stackoverflow post you're right that > cmxpchg16b will not give a consistent view of the 16 bytes of memory > across multiple NUMA nodes. However, maybe two 4 byte values right next > to each other would be sufficient for your use case and could then be > c