Re: D compilation is too slow and I am forking the compiler
On Friday, 23 November 2018 at 16:21:53 UTC, welkam wrote: If you want to read data from that bool CPU needs to fetch 8 bytes of data(cache line of 64 bits). What this means is that for one bit of information CPU fetches 64 bits of data resulting in 1/64 = 0.015625 or ~1.6 % signal to noise ratio. This is terrible! Cache line of 64 bits. 64 BITS. This forum is full of knowledgeable people and they should have spoted this mistake or they didnt read it. Cache lines on most processors are 64 bytes. Now I know why it felt weird when I wrote that post. So the real math for when you read one bit in cache line is 1/(64*8) = 0.001953125 or ~ 0.2% signal to noise ratio
Re: D compilation is too slow and I am forking the compiler
On Thursday, 29 November 2018 at 09:18:35 UTC, Laeeth isharc wrote: The innovator's dilemma, which is really an insight that dates back to Toynbee, and before that Ibn Khaldun, is not so obvious. I am not sure that you have understood it. I suggest reading the book if you are interested, but otherwise I unfortunately don't have so much time at the moment to try to persuade you of what this phenomenon is like and there's limited value to talking about talking rather than having a discussion based on a shared understanding of what this is about. No, indeed I've not read about it (yet, it looks like a tale of our lives). My market has indeed a distribution where the best customer brings 3:1 vs the average, it's not inf:1 like it would be in programming language "customers". "Impulse buy" is predominent too, which does not exist for technical decisions. I understand that a few big players will bring a lot more than hundreds of smaller ones. Especially if the smaller ones keep writing on the D newsgroup :) In my market too, I prefer if D is kind of unpopular! I don't tell details to competitors other than "works for me". And they aren't much interested, which is satisfying. This secrecy has to be balanced by the fact we have to hire, and D being a critical piece of infrastructure I'd like it to simply receive more money, being a small player I contribute what I can to the Foundation - and direct others that came to Dplug to do so.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 28 November 2018 at 13:30:37 UTC, Guillaume Piolat wrote: On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc wrote: Nassim Taleb raises the question of how do you choose between two surgeons, both recommended. One looks the part and hangs his many certificates on his office wall. The other looks scruffy with the appearance of a tradesman. Who do you pick? Taleb says pick the guy who doesn't look the part because if he got there without signalling he must have something going for him. It's definately the kind of surgeon one should choose - programmers that are not necessarily well groomed etc.. - but is it the kind of surgeon people will actually recommend? I'm doubtful. If X has the social signalling then people will recommend X even without trying, because it's socially safe. If one doesn't have the signalling, I've found the hard way even supporters will hesitate a bit before making recommendations, because of the social standing _cost_ it may have. But then, perhaps recommendations don't matter, because opinions don't matter much? I think they matter to be even heard on public places. And I think early adopters need a nudge, the influent need to be bothered by less influents (influencers are not especially on the lookout for new options, as they are already influent). Above all I think the niche of early-adopters is smaller than the larger market for languages, and the early-adopters are going elsewhere. The innovator's dilemma, which is really an insight that dates back to Toynbee, and before that Ibn Khaldun, is not so obvious. I am not sure that you have understood it. I suggest reading the book if you are interested, but otherwise I unfortunately don't have so much time at the moment to try to persuade you of what this phenomenon is like and there's limited value to talking about talking rather than having a discussion based on a shared understanding of what this is about.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 28 November 2018 at 13:05:34 UTC, Guillaume Piolat wrote: On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc wrote: D isn't really marketed and it's definitely not sold. That's an implicit strategy in itself. What I see in my (absurdly competitive) market is that the people that truly do no-marketing tend to close shop, sometimes despite very competitive offerings. It colors my perception of course, since it can be very tempting to appeal to a limited pool of discerning customers; but that would mean death. What is the ratio of expenditure of your best customer to an average customer? Not much. That's one main reason why your intuition developed by organising your emotions according to your business domain fits this domain less. What is the ratio of expenditure of the biggest 'customer' of Python to the average 'customer'? Measured by resources lent to the community directly or indirectly, or by the wage bill of programmers at that firm working in Python this ratio is enormous.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc wrote: Nassim Taleb raises the question of how do you choose between two surgeons, both recommended. One looks the part and hangs his many certificates on his office wall. The other looks scruffy with the appearance of a tradesman. Who do you pick? Taleb says pick the guy who doesn't look the part because if he got there without signalling he must have something going for him. It's definately the kind of surgeon one should choose - programmers that are not necessarily well groomed etc.. - but is it the kind of surgeon people will actually recommend? I'm doubtful. If X has the social signalling then people will recommend X even without trying, because it's socially safe. If one doesn't have the signalling, I've found the hard way even supporters will hesitate a bit before making recommendations, because of the social standing _cost_ it may have. But then, perhaps recommendations don't matter, because opinions don't matter much? I think they matter to be even heard on public places. And I think early adopters need a nudge, the influent need to be bothered by less influents (influencers are not especially on the lookout for new options, as they are already influent). Above all I think the niche of early-adopters is smaller than the larger market for languages, and the early-adopters are going elsewhere.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc wrote: I think that there are different strategies - decent appeal to a broad market and having a very high appeal to a small market (but there has better be something good about your potential customer base ie 'D, if you find VBA too difficult' is probably not a good strategy!). And you probably don't get to pick which situation you are in, and then one had better realise it and play the game you're in. The particular kind of market will shape what works - in my business you approach a retail client base differently from regular institutional investors and then the worlds' largest pools of money involved something else again. D isn't really marketed and it's definitely not sold. That's an implicit strategy in itself. But one doesn't decide to have no strategy (at least if they any common sense!), one simply has no strategy. Unfortunately I think D falls into the latter, certainly not more than "Build it and they will come", irrespective of it effectiveness. Actually no less than 3 programmer friends came to (I'm the weirdo-using-D and people are _always_ in disbelief and invent all sorts of reasons not to try) saying they saw an article on D on HN, with "D compilation is slow", and on further examination they didn't read or at best the first paragraph. But they did remember the title. They may rationally think their opinion of D hasn't changed: aren't we highly capable people? I hope so! It doesn't matter what most people think. It matters what people who are on the fence or using D already a bit think. Or people who have a lot of problems to which D is in part a solution only they didn't know about or think of D yet. Then we should try to subtly (for some value of subtlety) make ourselves noticed.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc wrote: D isn't really marketed and it's definitely not sold. That's an implicit strategy in itself. What I see in my (absurdly competitive) market is that the people that truly do no-marketing tend to close shop, sometimes despite very competitive offerings. It colors my perception of course, since it can be very tempting to appeal to a limited pool of discerning customers; but that would mean death. Imho the ones that succeed doing "no marketing" are actively telling others they don't do marketing. It can be sometimes funny, when the "no marketing" products comes with walls of text and videos that very much sound like... storytelling. But Like in the "no-makeup makeup", there is still makeup but not obvious. 2018's marketing trend is to subtly fake complete honesty.
Re: D compilation is too slow and I am forking the compiler
On Monday, 26 November 2018 at 16:00:36 UTC, Guillaume Piolat wrote: On Thursday, 22 November 2018 at 04:48:09 UTC, Vladimir Panteleev wrote: On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright wrote: Unfortunately, you're right. The title will leave the impression "D is slow at compiling". You have to carefully read the article to see otherwise, and few will do that. Sorry about that. I'll have to think of two titles next time, one for the D community and one for everyone else. If it's of any consolation, the top comments in both discussion threads point out that the title is inaccurate on purpose. Please don't get me wrong, it's an excellent article, a provocative title, and fantastic work going on. I didn't meant to hurt! In my opinion language adoption is a seduction/sales process very much like business-to-consumer is, the way I see it it's strikingly similar to marketing B2C apps, unless there will be no "impulse buy". I think that there are different strategies - decent appeal to a broad market and having a very high appeal to a small market (but there has better be something good about your potential customer base ie 'D, if you find VBA too difficult' is probably not a good strategy!). And you probably don't get to pick which situation you are in, and then one had better realise it and play the game you're in. The particular kind of market will shape what works - in my business you approach a retail client base differently from regular institutional investors and then the worlds' largest pools of money involved something else again. D isn't really marketed and it's definitely not sold. That's an implicit strategy in itself. Nassim Taleb raises the question of how do you choose between two surgeons, both recommended. One looks the part and hangs his many certificates on his office wall. The other looks scruffy with the appearance of a tradesman. Who do you pick? Taleb says pick the guy who doesn't look the part because if he got there without signalling he must have something going for him. But in general you can appeal on merits mostly to an audience that is highly discerning and very capable. If you haven't got any money to appeal to an audience that judges based on heuristics and social factors well then you can try to avoid accidentally putting people off, you can be creative with guerilla marketing but the key thing is to make the most of what you got. If everyone else does things a certain way then if for some reason that's closed off to you for now then if you look closely, with active perception,you may well see opportunities that are neglected to approach the problem another way. Actually no less than 3 programmer friends came to (I'm the weirdo-using-D and people are _always_ in disbelief and invent all sorts of reasons not to try) saying they saw an article on D on HN, with "D compilation is slow", and on further examination they didn't read or at best the first paragraph. But they did remember the title. They may rationally think their opinion of D hasn't changed: aren't we highly capable people? It doesn't matter what most people think. It matters what people who are on the fence or using D already a bit think. Or people who have a lot of problems to which D is in part a solution only they didn't know about or think of D yet. The messenger matters too. If someone you trust and rate highly tells you something based on their experience that counts for a lot more than all the blog posts in the world. And working code and lived experience dominates the social talk about it. I've talked about D with the CTO of Bloomberg, the outgoing COO of Barclays investment bank, the number two guy at a 30bn hedge fund, the COO of the largest hedge fund in the world (depending on how you count) and more. That's not going to change anything tomorrow but in time those kinds of conversations matter much more than what people might say on Reddit. It's not either /or of course, but it's just not worth sweating your reviews. Finally the reasons people buy things are not what you might reasonably think! Ask Walter how he was able to compete successfully for so long as a one man band with Microsoft. I don't think his edge was in the beginning something calculated. Reasonable people may think marketing and biases don't apply to them but they do, it works without your consent. The thing is that we had a bubble in synthetic manufactured marketing. And now increasingly people are tired of that and seek what's authentic, real and that doesn't pretend to be perfect. That doesn't mean a bit of thought is a bad idea,just that it might matter less than you think that the D community isn't particularly interested in marketing. Sometimes one can see that hidden in what superficially seems to be a weakness is a strength.
Re: D compilation is too slow and I am forking the compiler
On 11/26/18 11:53 AM, Guillaume Piolat wrote: How many times have you been in this conversation: -- - What language are you using? - D. - I know next to nothing about D. - Oh, it's very good, I even built a business on it! arguments and features>. - Oh no thanks. I should try Rust, it's secure, fast, modern blah blah; facts don't matter to me. But in reality I won't even learn a new language, I'm happy with a language without multi-threading. -- It happens to me ALL THE TIME. This pattern is so predictable it's becoming boring so now I just keep silent. That's no surprise. The modern tech world is absolutely flooded with imbeciles and fashion-driven people.
Re: D compilation is too slow and I am forking the compiler
On Tuesday, 27 November 2018 at 14:19:12 UTC, Joakim wrote: As Laeeth always says, you're best off looking for people who're actually capable and empowered to make such risky decisions, rather than aiming for the majority too early, because they only jump on board once the bandwagon is stuffed and rolling downhill. I get the (I read "D compilation is slow" + "I'm not interested") vibe from people that actually _are_ early adopters, very interested in programming, who adopted Scala and Nim despite obvious risks. I disagree with your opinion but just lacks the time to defend my own in a lengthy forum post. Let me make a giant argument by authority: I sell B2C software for a living. Convincing people to try software despite their will is what I do for a living. "Build it and they will come" would be a fantasy to believe if we hadn't competitors who had people come way before they built it. Complicated triple negation arguments (D compile fast! no wait, it compiles slowly! no wait, it compiles fast!) don't work. In a few months whenever someone bring out D, forum replies will say "but D compilation is not that fast".
Re: D compilation is too slow and I am forking the compiler
On Monday, 26 November 2018 at 16:42:40 UTC, bachmeier wrote: On Monday, 26 November 2018 at 16:21:39 UTC, Joakim wrote: I agree that it was a risky title, as many who don't know D will simply see it and go, "Yet another slow compiler, eh, I'll pass" and not click on the link. Whereas others who have heard something of D will be intrigued, as they know it's already supposed to compile fast. And yet more others will click on it purely for the controversy, just to gawk at some technical bickering. I don't actually think it was risky. What are the odds that someone was going to start using D for a major project but then changed her mind upon seeing a title on HN or Reddit? Probably very small that even one person did that. Yes, but you're ignoring the much larger group I mentioned- those who only vaguely heard of D, if at all- and the negative title gives them a reason not to look into it further. And then there is always the fact that there was a story on HN/Reddit about D. It's hard for publicity for a language like D to be bad when so few people use it. The quote that "there's no such thing as bad publicity" comes from art and show business though, don't think it's true for tech and other markets. When your audience is looking for a tool and not entertainment, there's lots of ways for bad publicity to sink it. Anyway, I noted in this case that the provocative title may actually have gotten more people to read a positive post, so the pros likely outweighed the cons. We can just never know how large the unclicked-on downside was: you can never measure how many people heard of but _didn't_ buy your book, because they didn't like the title or something else about its exterior. On Monday, 26 November 2018 at 16:53:59 UTC, Guillaume Piolat wrote: On Monday, 26 November 2018 at 16:21:39 UTC, Joakim wrote: In my opinion language adoption is a seduction/sales process very much like business-to-consumer is, the way I see it it's strikingly similar to marketing B2C apps, unless there will be no "impulse buy". I find that hard to believe: we are talking about a technical tool here. How many times have you been in this conversation: -- - What language are you using? - D. - I know next to nothing about D. - Oh, it's very good, I even built a business on it! list of arguments and features>. - Oh no thanks. I should try Rust, it's secure, fast, modern blah blah; facts don't matter to me. But in reality I won't even learn a new language, I'm happy with a language without multi-threading. -- It happens to me ALL THE TIME. This pattern is so predictable it's becoming boring so now I just keep silent. Never, I don't go around trying to convince people one-on-one to use D. I have given talks to groups introducing the language, that's how I go about it. What happens? Rust / Go have outmarketed us with words. The battle (of marketing) is on words not technical features, Rust happen to own "programming language" + "safety", what do we own? D is good in all kinds of directions and the marketing message is less simple. The leaders choose to own the word "fast" (see our new motto "fast code, fast" which is very accurate) and it's important to get aligned. I'll note that in your example they haven't actually learnt Rust either. I don't think marketing is that relevant for D at this stage, nor for Rust/Go either. The way anything- tech, fashion, TV shows- becomes popular is that some early tastemaker decides that it's worth using or backing. Eventually, enough early adopters find value that it spreads out to the majority, who simply follow their lead. Most people aren't early adopters of most things. They like to think they are, so they'll give you all kinds of rational-sounding reasons for why they don't like some new tech, but the real underlying thought process goes something like this, "I have no idea if this new tech will do well or not. I could take a risk on it but it's safer not to, so I will just wait and see if it gets popular, then follow the herd." Very few will admit this though, hence the list of plausible-sounding reasons that don't actually make sense! ;) As Laeeth always says, you're best off looking for people who're actually capable and empowered to make such risky decisions, rather than aiming for the majority too early, because they only jump on board once the bandwagon is stuffed and rolling downhill. Also, regardless of how languages are chosen as they get into the majority, D is very much still in the innovators/early-adopters stage: But the current state of D would very much accomodate the middle-of-the-curve adopters. The language rarely breaks stuff. People making money with it, making long-term bets. Hell, I could make a laundry list of things that are better in D versus any alternatives! That doesn't bring users. I'm not talking about the quality of
Re: D compilation is too slow and I am forking the compiler
On Monday, 26 November 2018 at 16:21:39 UTC, Joakim wrote: In my opinion language adoption is a seduction/sales process very much like business-to-consumer is, the way I see it it's strikingly similar to marketing B2C apps, unless there will be no "impulse buy". I find that hard to believe: we are talking about a technical tool here. How many times have you been in this conversation: -- - What language are you using? - D. - I know next to nothing about D. - Oh, it's very good, I even built a business on it! list of arguments and features>. - Oh no thanks. I should try Rust, it's secure, fast, modern blah blah; facts don't matter to me. But in reality I won't even learn a new language, I'm happy with a language without multi-threading. -- It happens to me ALL THE TIME. This pattern is so predictable it's becoming boring so now I just keep silent. What happens? Rust / Go have outmarketed us with words. The battle (of marketing) is on words not technical features, Rust happen to own "programming language" + "safety", what do we own? D is good in all kinds of directions and the marketing message is less simple. The leaders choose to own the word "fast" (see our new motto "fast code, fast" which is very accurate) and it's important to get aligned. Also, regardless of how languages are chosen as they get into the majority, D is very much still in the innovators/early-adopters stage: But the current state of D would very much accomodate the middle-of-the-curve adopters. The language rarely breaks stuff. People making money with it, making long-term bets. Hell, I could make a laundry list of things that are better in D versus any alternatives! That doesn't bring users. With people like that, it's almost impossible to get them in the early adopter stage. They will only jump on the bandwagon once it's full, ie as part of the late majority. There is a gap where we are, but "People like that" are almost everyone. Those people actually are middle-of-the-curve adopter, if you see a true late adopter in the wild it takes 3 relatives programming in D so that they start to be interested. Who doesn't want to be out of the early adopter stage, and get into the "officially endorsed safe choice" cohort? D is remarkably ready as a safe choice for lots of software. Given how well it did on HN/reddit/lobste.rs, I think Vlad's gamble probably paid off. We can't run the counterfactual of choosing a safer title to see if it would have done even better, let's just say it did well enough. ;) Alternative darker view: ever remarked how D articles often goes downvoted on HN? The title who says something bad about D is upvoted ; it's easy to see events as going our way. I, for one, didn't really read the article. Who has time for that?
Re: D compilation is too slow and I am forking the compiler
On Monday, 26 November 2018 at 16:21:39 UTC, Joakim wrote: I agree that it was a risky title, as many who don't know D will simply see it and go, "Yet another slow compiler, eh, I'll pass" and not click on the link. Whereas others who have heard something of D will be intrigued, as they know it's already supposed to compile fast. And yet more others will click on it purely for the controversy, just to gawk at some technical bickering. I don't actually think it was risky. What are the odds that someone was going to start using D for a major project but then changed her mind upon seeing a title on HN or Reddit? Probably very small that even one person did that. On the other hand, it says a lot of other things: - There's an active community that cares about the language. - It's not a dying language. - Fast compilation is a realistic possibility. - There are users with the technical ability to make the compiler faster. And then there is always the fact that there was a story on HN/Reddit about D. It's hard for publicity for a language like D to be bad when so few people use it.
Re: D compilation is too slow and I am forking the compiler
On Monday, 26 November 2018 at 16:00:36 UTC, Guillaume Piolat wrote: On Thursday, 22 November 2018 at 04:48:09 UTC, Vladimir Panteleev wrote: On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright wrote: Unfortunately, you're right. The title will leave the impression "D is slow at compiling". You have to carefully read the article to see otherwise, and few will do that. Sorry about that. I'll have to think of two titles next time, one for the D community and one for everyone else. If it's of any consolation, the top comments in both discussion threads point out that the title is inaccurate on purpose. Please don't get me wrong, it's an excellent article, a provocative title, and fantastic work going on. I didn't meant to hurt! In my opinion language adoption is a seduction/sales process very much like business-to-consumer is, the way I see it it's strikingly similar to marketing B2C apps, unless there will be no "impulse buy". I find that hard to believe: we are talking about a technical tool here. Also, regardless of how languages are chosen as they get into the majority, D is very much still in the innovators/early-adopters stage: https://en.m.wikipedia.org/wiki/Technology_adoption_life_cycle That is a very different type of sales process, much more geared towards what the new tech can actually do. Actually no less than 3 programmer friends came to (I'm the weirdo-using-D and people are _always_ in disbelief and invent all sorts of reasons not to try) saying they saw an article on D on HN, with "D compilation is slow", and on further examination they didn't read or at best the first paragraph. But they did remember the title. They may rationally think their opinion of D hasn't changed: aren't we highly capable people? With people like that, it's almost impossible to get them in the early adopter stage. They will only jump on the bandwagon once it's full, ie as part of the late majority. I'm not making that up! So why is it a problem ? HN may be the only time they hear about D. The words of the title may be their only contact with it. The first 3 words of the title may be the only thing associated with the "D language" chunk in their brain. The associative mind doesn't know _negation_ so even a title like "D compilation wasn't fast so I forked the compiler" is better from a marketing point of view since it contains the word "fast" in it! That's why marketing people have the annoying habit of using positive words, you may think this stuff is unimportant but this is actually the important meat. Reasonable people may think marketing and biases don't apply to them but they do, it works without your consent. I agree that it was a risky title, as many who don't know D will simply see it and go, "Yet another slow compiler, eh, I'll pass" and not click on the link. Whereas others who have heard something of D will be intrigued, as they know it's already supposed to compile fast. And yet more others will click on it purely for the controversy, just to gawk at some technical bickering. Given how well it did on HN/reddit/lobste.rs, I think Vlad's gamble probably paid off. We can't run the counterfactual of choosing a safer title to see if it would have done even better, let's just say it did well enough. ;)
Re: D compilation is too slow and I am forking the compiler
On Thursday, 22 November 2018 at 04:48:09 UTC, Vladimir Panteleev wrote: On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright wrote: Unfortunately, you're right. The title will leave the impression "D is slow at compiling". You have to carefully read the article to see otherwise, and few will do that. Sorry about that. I'll have to think of two titles next time, one for the D community and one for everyone else. If it's of any consolation, the top comments in both discussion threads point out that the title is inaccurate on purpose. Please don't get me wrong, it's an excellent article, a provocative title, and fantastic work going on. I didn't meant to hurt! In my opinion language adoption is a seduction/sales process very much like business-to-consumer is, the way I see it it's strikingly similar to marketing B2C apps, unless there will be no "impulse buy". Actually no less than 3 programmer friends came to (I'm the weirdo-using-D and people are _always_ in disbelief and invent all sorts of reasons not to try) saying they saw an article on D on HN, with "D compilation is slow", and on further examination they didn't read or at best the first paragraph. But they did remember the title. They may rationally think their opinion of D hasn't changed: aren't we highly capable people? I'm not making that up! So why is it a problem ? HN may be the only time they hear about D. The words of the title may be their only contact with it. The first 3 words of the title may be the only thing associated with the "D language" chunk in their brain. The associative mind doesn't know _negation_ so even a title like "D compilation wasn't fast so I forked the compiler" is better from a marketing point of view since it contains the word "fast" in it! That's why marketing people have the annoying habit of using positive words, you may think this stuff is unimportant but this is actually the important meat. Reasonable people may think marketing and biases don't apply to them but they do, it works without your consent.
Re: D compilation is too slow and I am forking the compiler
On Friday, 23 November 2018 at 19:21:03 UTC, Walter Bright wrote: On 11/23/2018 5:23 AM, welkam wrote: Currently D reads the all files that are passed in command line before starting lexing/parsing, but in principle we could start lexing/parsing after first file is read. In fact we could start after first file`s first line is read. DMD used to do that. But it was removed because: 1. nobody understood the logic 2. it didn't seem to make a difference You can still see the vestiges by the: static if (ASYNCREAD) blocks in the code. I didnt expect huge wins. This would be useful when you start your computer and files have to be read from old spinning rust and the project has many files. Otherwise files will be cached and memcopy is fast. I was surprised on how fast modern computers copy data from one place to another. Speaking of memcpy here is a video you might like. It has memcpy, assembler and a bit of compiler. Its very easy watch for when you want to relax. Level1 Diagnostic: Fixing our Memcpy Troubles (for Looking Glass) https://www.youtube.com/watch?v=idauoNVwWYE
Re: D compilation is too slow and I am forking the compiler
On 11/23/2018 5:23 AM, welkam wrote: Currently D reads the all files that are passed in command line before starting lexing/parsing, but in principle we could start lexing/parsing after first file is read. In fact we could start after first file`s first line is read. DMD used to do that. But it was removed because: 1. nobody understood the logic 2. it didn't seem to make a difference You can still see the vestiges by the: static if (ASYNCREAD) blocks in the code.
Re: D compilation is too slow and I am forking the compiler
On 11/23/2018 6:37 AM, welkam wrote: Your post on reddit received more comments than D front ends inclusion to GCC. If you titled your post differently you probably wouldn't had such success so from my perspective its a net positive. Sure there are few people that took the wrong message but there are more people who saw your post It definitely shows the value of a provocative title!
Re: D compilation is too slow and I am forking the compiler
On 11/23/2018 2:12 AM, Jacob Carlborg wrote: Would it be possible to have one string table per thread and merge them to one single shared string table before continuing with the next phase? It'd probably be even slower because one would have to rewrite all the pointers into the string table.
Re: D compilation is too slow and I am forking the compiler
On Friday, 23 November 2018 at 14:32:39 UTC, Vladimir Panteleev
wrote:
On Friday, 23 November 2018 at 13:23:22 UTC, welkam wrote:
If we run these steps in different thread on the same core
with SMT we could better use core`s resources. Reading file
with kernel, decoding UTF-8 with vector instructions and
lexing/parsing with scalar operations while all communication
is done trough L1 and L2 cache.
You might save some pages from the data cache, but by doing
more work at once, the code might stop fitting in the
execution-related caches (code pages, microcode, branch
prediction) instead.
Its not about saving tlb pages or fitting better in cache.
Compilers are considered streaming applications - they dont
utilize cpu caches effectively. You cant read one character and
emit machine code then read next character you have to go over
all data multiple times while you modify it. I can find white
papers if you interested where people test GCC with different
cache architectures and it doesnt make much of a difference. GCC
is popular application when testing caches.
Here are profiling data from DMD
Performance counter stats for 'dmd -c main.d':
600.77 msec task-clock:u #0.803 CPUs
utilized
0 context-switches:u#0.000 K/sec
0 cpu-migrations:u #0.000 K/sec
33,209 page-faults:u # 55348.333
M/sec
1,072,289,307 cycles:u # 1787148.845
GHz
870,175,210 stalled-cycles-frontend:u # 81.15%
frontend cycles idle
721,897,927 stalled-cycles-backend:u # 67.32%
backend cycles idle
881,895,208 instructions:u#0.82 insn
per cycle
#0.99
stalled cycles per insn
171,211,752 branches:u# 285352920.000
M/sec
11,287,327 branch-misses:u #6.59% of
all branches
0.747720395 seconds time elapsed
0.497698000 seconds user
0.104165000 seconds sys
Most important data in this conversation is 0.82 insn per cycle.
My CPU could do ~2 IPC so there are plenty of CPU resources
available. New Intel desktop processors are designed to do 4
insn/cycle. What is limiting DMD performance is slow RAM, data
fetching and not what you listed.
code pages - you mean TLB here?
microcode cache. Not all processors have it and those who have
only benefit trivial loops. DMD have complex loops.
branch prediction. More entries in branch predictor wont help
here because branches are missed because data is unpredictable
not because there are too many branches. Also branch
missprediction penalty is around 30 cycles while reading from RAM
could be over 200 cycles.
L1 code cache. You didnt mention this but running those tasks in
SMT mode might trash L1$ so execution might not be optimal.
Instead of parallel reading of imports DMD needs more data
oriented data structures instead of old OOP inspired data
structures. Ill give you example why its the case.
Consider
struct {
bool isAlive;
}
If you want to read data from that bool CPU needs to fetch 8
bytes of data(cache line of 64 bits). What this means is that for
one bit of information CPU fetches 64 bits of data resulting in
1/64 = 0.015625 or ~1.6 % signal to noise ratio. This is terrible!
AFAIK DMD doesnt make this kind of mistake but its full of large
structs and classes that are not efficient to read. To fix this
we need to split those large data structures into smaller ones
that only contain what is needed for particular algorithm. I
predict 2x speed improvement if we transform all data structures
in DMD. Thats improvement without improving algorithms only
changing data structures. This getting too longs so i will stop
right now
Re: D compilation is too slow and I am forking the compiler
On Thursday, 22 November 2018 at 04:48:09 UTC, Vladimir Panteleev wrote: Sorry about that. I'll have to think of two titles next time, one for the D community and one for everyone else. If it's of any consolation, the top comments in both discussion threads point out that the title is inaccurate on purpose. Your post on reddit received more comments than D front ends inclusion to GCC. If you titled your post differently you probably wouldn't had such success so from my perspective its a net positive. Sure there are few people that took the wrong message but there are more people who saw your post
Re: D compilation is too slow and I am forking the compiler
On Friday, 23 November 2018 at 13:23:22 UTC, welkam wrote: If we run these steps in different thread on the same core with SMT we could better use core`s resources. Reading file with kernel, decoding UTF-8 with vector instructions and lexing/parsing with scalar operations while all communication is done trough L1 and L2 cache. You might save some pages from the data cache, but by doing more work at once, the code might stop fitting in the execution-related caches (code pages, microcode, branch prediction) instead.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 10:56:02 UTC, Walter Bright wrote: Wouldn't it be awesome to have the lexing/parsing of the imports all done in parallel? From my testing lexing/parsing takes small amount of build time so running it in parallel might be small gain. We should consider running in parallel more heavy hitting features like CTFE and templates. Since we are in wish land here is my wishes. Currently D reads the all files that are passed in command line before starting lexing/parsing, but in principle we could start lexing/parsing after first file is read. In fact we could start after first file`s first line is read. Out of all operation before semantic pass, reading from hard disk should be the slowest so it might be possible to decode utf-8, lex and parse at the speed of reading from hard disk. If we run these steps in different thread on the same core with SMT we could better use core`s resources. Reading file with kernel, decoding UTF-8 with vector instructions and lexing/parsing with scalar operations while all communication is done trough L1 and L2 cache. I thought about using memory mapped files to unblock file reading as a first step but lack of good documentation about mmf and lack of thorough understanding of front end made me postpone this modification. Its a change with little benefit. The main difficulty in getting that to work is dealing with the shared string table. At begging of parsing a thread could get a read-only shared slice of string table. All strings not in table are put in local string table. After parsing tables are merged and shared slice is updated so new thread could start with bigger table. this assumes that table is not sorted
Re: D compilation is too slow and I am forking the compiler
On 2018-11-21 11:56, Walter Bright wrote: Wouldn't it be awesome to have the lexing/parsing of the imports all done in parallel? The main difficulty in getting that to work is dealing with the shared string table. Would it be possible to have one string table per thread and merge them to one single shared string table before continuing with the next phase? -- /Jacob Carlborg
Re: D compilation is too slow and I am forking the compiler
On Thursday, 22 November 2018 at 13:19:58 UTC, Andrej Mitrovic wrote: On Thursday, 22 November 2018 at 11:16:26 UTC, Paolo Invernizzi wrote: BTW, it's nice to see again the Secret Squirrel on the forum, in these days: welcome back Andrej! /Paolo Oh hey there too! I'm sorry if I can't recall you, though. ¯\_(ツ)_/¯ Oh, no problem... eheh I mostly lurk around here these days. Yep, the same... But I still use D heavily, at work. Well, the same here; not so heavily right now, my CTO is not sure anymore about the "case for D", but well, we have just delivered a D (medical) codebase to one of our customer... Let's see... /Paolo
Re: D compilation is too slow and I am forking the compiler
On Thursday, 22 November 2018 at 11:16:26 UTC, Paolo Invernizzi wrote: BTW, it's nice to see again the Secret Squirrel on the forum, in these days: welcome back Andrej! /Paolo Oh hey there too! I'm sorry if I can't recall you, though. ¯\_(ツ)_/¯ I mostly lurk around here these days. But I still use D heavily, at work.
Re: D compilation is too slow and I am forking the compiler
On Thursday, 22 November 2018 at 10:51:45 UTC, Andrej Mitrovic wrote: BTW, it's nice to see again the Secret Squirrel on the forum, in these days: welcome back Andrej! /Paolo
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright wrote: Unfortunately, you're right. The title will leave the impression "D is slow at compiling". You have to carefully read the article to see otherwise, and few will do that. Well comparative to itself sometimes it is. When you initially write D code you get used to the blazing fast speeds, but when eventually the compilation speed slows down as a project grows then this has a real effect on productivity. Maybe a better title would have been "D compilation sometimes slows down too much", but it wouldn't get as many hits. On the upside, people who read the article - or even just read the comments section, will quickly realize that D's compilation speed is still miles faster than the competition. They might actually try the language. :)
Re: D compilation is too slow and I am forking the compiler
On 11/21/2018 8:48 PM, Vladimir Panteleev wrote: On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright wrote: Unfortunately, you're right. The title will leave the impression "D is slow at compiling". You have to carefully read the article to see otherwise, and few will do that. Sorry about that. I'll have to think of two titles next time, one for the D community and one for everyone else. If it's of any consolation, the top comments in both discussion threads point out that the title is inaccurate on purpose. It has indeed done well. In fact, the article is so good it is probably worth it to have that attention-getting title. It is a risky strategy, though :-)
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright wrote: Unfortunately, you're right. The title will leave the impression "D is slow at compiling". You have to carefully read the article to see otherwise, and few will do that. Sorry about that. I'll have to think of two titles next time, one for the D community and one for everyone else. If it's of any consolation, the top comments in both discussion threads point out that the title is inaccurate on purpose.
Re: D compilation is too slow and I am forking the compiler
I would say opposite :) On Wed, Nov 21, 2018 at 9:55 PM Walter Bright via Digitalmars-d-announce < digitalmars-d-announce@puremagic.com> wrote: > On 11/21/2018 5:55 AM, Guillaume Piolat wrote: > > On Wednesday, 21 November 2018 at 11:18:22 UTC, Nicholas Wilson wrote: > >> On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir Panteleev > wrote: > >>> > https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/ > >>> > >> > >> This is #2 on HN at the moment. > > > > I would be wary of such titles. > > > > Whatever will remain in minds will be something like "D compilation is > slow" > > which isn't accurate when compared to the competition. > > > > The article is clever, the title is clever, but most people will only > read the > > title. > > Unfortunately, you're right. The title will leave the impression "D is > slow at > compiling". You have to carefully read the article to see otherwise, and > few > will do that. >
Re: D compilation is too slow and I am forking the compiler
I neglected point out, however, that the article itself is a home run! Thank you, Vladimir!
Re: D compilation is too slow and I am forking the compiler
On 11/21/2018 5:55 AM, Guillaume Piolat wrote: On Wednesday, 21 November 2018 at 11:18:22 UTC, Nicholas Wilson wrote: On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir Panteleev wrote: https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/ This is #2 on HN at the moment. I would be wary of such titles. Whatever will remain in minds will be something like "D compilation is slow" which isn't accurate when compared to the competition. The article is clever, the title is clever, but most people will only read the title. Unfortunately, you're right. The title will leave the impression "D is slow at compiling". You have to carefully read the article to see otherwise, and few will do that.
Re: D compilation is too slow and I am forking the compiler
On 11/21/2018 3:19 AM, Iain Buclaw wrote: On Wednesday, 21 November 2018 at 10:56:02 UTC, Walter Bright wrote: Wouldn't it be awesome to have the lexing/parsing of the imports all done in parallel? The main difficulty in getting that to work is dealing with the shared string table. What about creating a new Fiber for each module needing lexing/parsing/semantic to be ran? Compilation of one module would then get as far as it can until it needs to defer, then calls yield() to continue compilation of the next module. This in hope that when the round trip returns, the AST will be sufficiently complete enough for compilation to continue. Since the lexing/parsing does not do any blocking calls, fibers won't help. It has to be another hardware thread. Trying to parallelize semantic over multiple modules will be very hard to do.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 11:18:22 UTC, Nicholas Wilson wrote: This is #2 on HN at the moment. Also on reddit: https://www.reddit.com/r/programming/comments/9z36xg/d_compilation_is_too_slow_and_i_am_forking_the/
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 11:58:25 UTC, Vladimir Panteleev wrote: On Wednesday, 21 November 2018 at 11:35:02 UTC, Atila Neves wrote: [...] Looking forward to it! [...] That particular problem is in large part due to that the -unittest switch is not namespaced. I ran into the same issue with -allinst - with std.path, it breaks only if -unittest is also specified. I don't want to compile std.path unit tests, just my own! Have we tried disabling -unittest for modules that aren't on the compiler's command line yet (or, in case of -i, not excluded)? I'm not sure what the real issue is here or what the solution would be. There was a PR to fix it that was closed without merging.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 11:18:22 UTC, Nicholas Wilson wrote: On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir Panteleev wrote: https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/ This is #2 on HN at the moment. I would be wary of such titles. Whatever will remain in minds will be something like "D compilation is slow" which isn't accurate when compared to the competition. The article is clever, the title is clever, but most people will only read the title.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 13:05:27 UTC, Nicholas Wilson wrote: On Wednesday, 21 November 2018 at 11:58:25 UTC, Vladimir Panteleev wrote: Have we tried disabling -unittest for modules that aren't on the compiler's command line yet (or, in case of -i, not excluded)? Not that I know of, thats a great idea! Yes it's sadly a well-known problem e.g. https://github.com/dlang/dmd/pull/8124
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 13:05:27 UTC, Nicholas Wilson wrote: On Wednesday, 21 November 2018 at 11:58:25 UTC, Vladimir Panteleev wrote: Have we tried disabling -unittest for modules that aren't on the compiler's command line yet (or, in case of -i, not excluded)? Not that I know of, thats a great idea! Maybe this hack could be developed further into a more generic "compiler server" idea. Wasn't that Robert's Masters thesis (Dconf 2013(?) presentation)? ;) The main problem with this, is the amount of context a compilers needs. And the current design of DMD does not lend itself splitting out the context. If you wanted you could consider the semantic pass of the compiler as a database, which answers queries such as: - which size does this type have. - which arguments does this function have - can the type A be casted to type B - which conversion function should be invoked for (B)A ? - is this function known to be pure? The data-base containing this information needs to be maintained on the compile-nodes, and that possibly leads to many data-dependencies. Which may degrade the performance of the "compiler server" to the point where it is quicker to do it locally. I am currently working (albeit very slowly due to lack of time and focus) to enable programmers to circumvent slow parts in compiler. When completed this should make a compiler-server unnecessary for some time to come.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 11:58:25 UTC, Vladimir Panteleev wrote: Have we tried disabling -unittest for modules that aren't on the compiler's command line yet (or, in case of -i, not excluded)? Not that I know of, thats a great idea! Maybe this hack could be developed further into a more generic "compiler server" idea. Wasn't that Robert's Masters thesis (Dconf 2013(?) presentation)? ;)
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 11:35:02 UTC, Atila Neves wrote: I'm also currently working on a project to save my bloodstream from the cortisol drip that happens when anything a computer does takes over a second, which these days means waiting for dmd to compile my code so I can see the result of the tests. I'll share more details when I have time. Looking forward to it! But: one of the things I want to do is a version of this / precompiled headers. I've complained before that compiling std.path with -unittest takes forever (0.5s or so, and most of it due to std.uni). That's a price I pay every time I make a one line change to any file, and the linker hasn't even been invoked yet. Here's the thing: Phobos only changes from one release to the next. Why am I waiting to recompile a read-only file that won't change unless I update the compiler, over and over again? That particular problem is in large part due to that the -unittest switch is not namespaced. I ran into the same issue with -allinst - with std.path, it breaks only if -unittest is also specified. I don't want to compile std.path unit tests, just my own! Have we tried disabling -unittest for modules that aren't on the compiler's command line yet (or, in case of -i, not excluded)? I'd love it if I could precompile Phobos and just use the digested information every time I'm iterating. I agree, it would be nice if we could ship some "precompiled module" files along with Phobos .lib / .so files, but it looks like implementing this feature correctly might be non-trivial. Maybe this hack could be developed further into a more generic "compiler server" idea.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir Panteleev wrote: https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/ Very interesting. I'm also currently working on a project to save my bloodstream from the cortisol drip that happens when anything a computer does takes over a second, which these days means waiting for dmd to compile my code so I can see the result of the tests. I'll share more details when I have time. But: one of the things I want to do is a version of this / precompiled headers. I've complained before that compiling std.path with -unittest takes forever (0.5s or so, and most of it due to std.uni). That's a price I pay every time I make a one line change to any file, and the linker hasn't even been invoked yet. Here's the thing: Phobos only changes from one release to the next. Why am I waiting to recompile a read-only file that won't change unless I update the compiler, over and over again? I'd love it if I could precompile Phobos and just use the digested information every time I'm iterating.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir Panteleev wrote: https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/ This is #2 on HN at the moment.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 10:56:02 UTC, Walter Bright wrote: Wouldn't it be awesome to have the lexing/parsing of the imports all done in parallel? The main difficulty in getting that to work is dealing with the shared string table. What about creating a new Fiber for each module needing lexing/parsing/semantic to be ran? Compilation of one module would then get as far as it can until it needs to defer, then calls yield() to continue compilation of the next module. This in hope that when the round trip returns, the AST will be sufficiently complete enough for compilation to continue.
Re: D compilation is too slow and I am forking the compiler
On 11/21/2018 2:16 AM, Vladimir Panteleev wrote: On Wednesday, 21 November 2018 at 09:46:44 UTC, Walter Bright wrote: It works by allocating memory from a memory-mapped file, which serves as the precompiled header. Hey, that's a great idea! Can we do this for DMD? :D On a more serious note: do you think that with D's features (type system / metaprogramming), you could have avoided some of those bugs? For example, one thing we can do in D which is still impossible in C++ is to automatically serialize/deserialize all fields of a struct/class (using tupleof / allMembers). Memory mapped files really were the key to success, because if you could reload the mmf at the same address, the pointers did not have to be patched. In the DMC++ source code, "dehydrating" a pointer meant subtracting a value from it so it was correct for the base address of the mmf, and "hydrating" a pointer was the inverse. The two bug prone problems were: 1. separating out the tangled data structures into what goes into the pch, and what does not. Obviously, nothing in the pch could point outside of it. 2. .h files are simply not compatible with this, so you've got to detect when it won't work. For example, anything like a command line switch or a macro that might cause different code to be generated in the pch had to invalidate it. Maybe I should have done your fork idea? :-) My experience with this drove many design decisions for D modules, for example, D modules are unaffected by where they are imported, the order they are imported, or the number of times they are imported. (Yes, I know about https://digitalmars.com/d/archives/digitalmars/D/D_needs_to_be_honest_320976.html) Anyhow, what I've thought about doing since the beginning was make DMD multithreaded. The language is designed to support multithreaded compilation. For example, lexing, parsing, semantic analysis, optimization, and code generation can all be done concurrently. DMD 1.0 would read imports in a separate thread. This would speed things up if you were using a slow filesystem, like NAS or a USB stick, but it was eventually disabled because there wasn't a perceptible speedup with current filesystems. Wouldn't it be awesome to have the lexing/parsing of the imports all done in parallel? The main difficulty in getting that to work is dealing with the shared string table.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 09:46:44 UTC, Walter Bright wrote: It works by allocating memory from a memory-mapped file, which serves as the precompiled header. Hey, that's a great idea! Can we do this for DMD? :D On a more serious note: do you think that with D's features (type system / metaprogramming), you could have avoided some of those bugs? For example, one thing we can do in D which is still impossible in C++ is to automatically serialize/deserialize all fields of a struct/class (using tupleof / allMembers).
Re: D compilation is too slow and I am forking the compiler
On 11/21/2018 12:07 AM, Vladimir Panteleev wrote: https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/ I implemented precompiled headers for Digital Mars C++. It took a lng time to work all the bugs out of it. It's also a brittle system. It works by allocating memory from a memory-mapped file, which serves as the precompiled header.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir Panteleev wrote: https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/ You might want to have a brush up on which direction C++ modules are heading in. Notable talks would be those given at the GNU Cauldron for both 2017 and 2018. The general run-down as I understand it. === Problem to solve: Compiler asks an Oracle about module A. Phrased this way, Compiler is a client, Oracle is a server. Oracle could be a file, socket, remote server, anything that can be read from or written to. Communication can be done via a standard format (such as json). This means that the Oracle (the implementation of) that keeps track of compilation and dependencies of the build is now someone else's problem as far as the Compiler is concerned. === I think what you've already started would fit well into this. Iain.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir Panteleev wrote: https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/ Not only an interesting read, but also interesting research!
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 08:32:39 UTC, Nicholas Wilson wrote: You gave me a fright there with the title there for a moment. :) Awesome stuff though. Not sure how easy it will be to upstream considering this needs to not wreck Windows and needs to work with LDC/GDC (at least we have inlining in the backend). All the DMD-side logic is all encapsulated in one function: https://github.com/CyberShadow/dmd/blob/dmdforker/src/dmd/mars.d#L501-L673 Its body can be versioned out in incompatible platforms/implementations.
Re: D compilation is too slow and I am forking the compiler
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir Panteleev wrote: https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/ You gave me a fright there with the title there for a moment. Awesome stuff though. Not sure how easy it will be to upstream considering this needs to not wreck Windows and needs to work with LDC/GDC (at least we have inlining in the backend).
D compilation is too slow and I am forking the compiler
https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/
