Re: LDC 1.30.0
On 7/21/22 6:18 AM, Christian Köstlin wrote: Thanks for the new release. On a sidenote I am having problem downloading any 2.100.1 dmd version from dlang.org. Is it just me? Old releases are available on http://downloads.dlang.org/releases but not the 2.100.1. Kind regards, Christian There is a known problem with this for at least a week or two. I believe ryukk_ filed a PR to fix, not sure if has been merged yet.
Re: On the Blog: DLang News for September/October 2021
On 10/31/21 3:26 PM, Mike Parker wrote: On Friday, 29 October 2021 at 15:03:46 UTC, Mike Parker wrote: I'll be putting together a video version this weekend for the foundation's YouTube channel that I should be able to publish by Sunday. The video version is done and is available here: https://youtu.be/jX9grHMTGAU I really want this to be a regular thing every two months, for both a news blog post and video. My request from you: share, share, share! "Stick around for the D-tails" *rofl* Nice, I love it
Re: D Language Foundation Monthly Meeting Summary (September 24, 2021)
On 10/1/21 8:32 AM, Mike Parker wrote: Walter would like to establish a new slogan, or tagline, for D. He asked us all to think about this for a future meeting. (Ali has since solicited some good advice from a relative who is a professional marketer that has gotten us on the right track.) 1. "Move fast" Pros: More succinct than "Write fast code, fast" or whatever the old slogan was. Still captures the essence of both speed of execution and ease of writing. Potential tie-in to D-Rocket logo. Cons: Potential association with silicon valley phrase "move fast and break things". 2. "Future code" Pros: Association with the idea that we are often at the leading edge of new features which are later incorporated into other languages. May also carry the implication that D is great language for the reader to learn in the future. Potential tie-in to space and mars theme. Cons: Could be open to criticism that garbage collected language is not "the future," but this would likely be a tiny number of detractors. ... As always, if anyone has anything they'd like me to put on the meeting agenda, please let me know beforehand. I don't know for sure if we'll be able to fit anything onto this month's meeting, though. These meetings can run long if the industry reps have anything big to discuss, and we're going to have a major item on the foundation's agenda that I expect will consume most of post-industry portion of the meeting (`-preview=in`). But if you have something, I'll see what I can do. Thanks for doing these writeups as community liaison.
Re: [Semi-OT] Sdlang Initiative
On 9/6/21 2:01 PM, Steven Schveighoffer wrote: But I actually am using sdlang-d for configuration files on my web server. However, I do not like the interface for it very much. SDLang itself is OK, but I find actually that I don't love the format. Like yaml, I have to research how the file format works every time I want to use it (maybe just because I very rarely have need to edit sdl files). For that reason I prefer json, but I hate that json doesn't allow comments and is full of quote spam (for config files). I am planning at some point to replace the config system with a [json5](https://json5.org/) implementation (based on jsoniopipe), otherwise this would definitely interest me! I have finally come around to TOML as the best alternative for human-centered configuration. It seems to be really popular in Rust ecosystem. I see 3 D libraries; haven't tested them but will with my next D project: https://code.dlang.org/search?q=toml
Re: dhtslib v0.12.0 (high-throughput sequencing library)
On Thursday, 2 September 2021 at 10:32:19 UTC, Johan wrote: On Wednesday, 1 September 2021 at 05:27:38 UTC, James Blachly [2] Here is a contemporary example of D used in high-throughput sequencing: DENTIST by Arne Ludwig at Max Planck institute https://github.com/a-ludi/dentist I am surprised to see the use of DMD (see the Dockerfile). If you want runtime performance, the first thing I would do is switch to LDC or GDC. Perhaps DENTIST's particular use of D and dhtslib is mainly forwarding calls to htslib (C) and thus D performance is not relevant? DENTIST is someone else's unrelated project that does not to my knowledge use `dhtslib`.
Re: Announcement and Request: Typesafe Coordinate Systems for High-Throughput Sequencing Applications
On 9/1/21 5:01 AM, Arne Ludwig wrote: I am happy to hear of your latest idea of creating type-safe coordinate systems. It's a great idea! After reading the code on GitHub, I have only one major remark: IMHO, it would be great to separate the novel coordinates systems from any `htslib` dependencies ([see lines 47-50](https://github.com/blachlylab/dhtslib/blob/e3b5af14e9eefa54bcc27bc0fcc9066dc3a4ea54/source/dhtslib/coordinates.d#L47-L50)) as there are only auxiliary functions that use both the novel coordinates systems and `htslib`. The greater goal I have in mind is to provide the coordinate systems in a separate DUB sub-package (e.g. `dhtslib:coordinates`) that requires only a D compiler. That makes integration into existing projects that do not need `htslib` much easier. This is an absolutely **outstanding** idea. Those imports were only to reuse an htslib `chr:X-Y` string parsing function, but we can trivially rewrite this in native D to enable sub-package independence! Also, I have a short list of minor, technical remarks: 1. The returned type in [line 114](https://github.com/blachlylab/dhtslib/blob/e3b5af14e9eefa54bcc27bc0fcc9066dc3a4ea54/source/dhtslib/coordinates.d#L114) has a typo, there is an additional 's'. Ahh, the curse of templates. Without 100% test coverage these things which would cause failure to compile in non-template code seem to always sneak in. Thank you so much. 2. The array of identifiers `CoordSystemLabels` in [line 203](https://github.com/blachlylab/dhtslib/blob/e3b5af14e9eefa54bcc27bc0fcc9066dc3a4ea54/source/dhtslib/coordinates.d#L203) is a bit unsafe and not strictly required for two reasons: A very excellent suggestion. I am still a metaprogramming novice. 3. The function `unionImpl` in [line 326](https://github.com/blachlylab/dhtslib/blob/e3b5af14e9eefa54bcc27bc0fcc9066dc3a4ea54/source/dhtslib/coordinates.d#L326) actually computes the convex hull of the two intervals which should be noted in the doc comment for completeness' sake. Yes, we had some internal debate about the appropriate result of both union and intersect operations when intervals are non-overlapping and return type is a non-array. Will leave as is and document as convex hull in this case. 4. I have noted that you use operator overloading for union and intersection of `Interval`s. You may also add overloads for the `offset` function in both `Interval` and `Coordinate` with `auto opBinary(string op, T)(T off) if ((op == '+' || op == '-') && isIntegral!T)` and `auto opBinaryRight(string op, T)(T off) if ((op == '+' || op == '-') && isIntegral!T)`. Very nice. I do miss operator overloading in some of the other languages I explored recently. I enjoyed reading the manuscript. It highlights the issue clearly and presents the solution without getting lost in details. Ignoring typos at this stage, I have no remarks on it – keep going! Thanks again for this critical review. As you know we are really pleased with how D has accelerated our science and wish to share it with the world. James
Announcement and Request: Typesafe Coordinate Systems for High-Throughput Sequencing Applications
In another post, I've just announced our D-based high throughput sequencing library, dhtslib. One feature that is, AFAIK, novel in the field is leveraging the compiler's type system to enforce correctness regarding different genome/reference sequence coordinate systems. Clearly, the encoding of domain specific knowledge in a language's type system is nothing new, but it is surprising that this has not been done before in bioinformatics, and it is an idea that IMO is long overdue given the trainwreck of different coordinate systems in our field. You can find dhtslib's develop branch, with Typesafe Coordinates merged and ready to use, here: https://github.com/blachlylab/dhtslib/ **Now the request:** We've drafted a manuscript describing Typesafe Coordinates as a sort of low-key endorsement of the D language and our library package `dhtslib`. You can find the manuscript here: https://github.com/blachlylab/typesafe-coordinates/ We would be very grateful to those of you who would take the time to read the manuscript and post comments (publicly or privately), _especially if we have made any incorrect statements_ or our language regarding type systems is awkward or nonstandard. We did praise D, and gently criticized Rust and OCaml* somewhat as it appeared to me that they lacked the features required to implement Typesafe Coordinate Systems in as ergonomic a way as we could in D. However, being a true novice at both of these other languages there is the possibility that I've missed something significant, and that the Rust and OCaml implementations could be retooled to match the D implementation. I'd still be glad to hear it if that's the case. I plan to make a few minor cleanups and submit this to a preprint server as well as a scientific journal in the next week or so. Kind regards James S Blachly, MD The Ohio State University * as a side note, I actually find the OCaml code quite attractive in its terseness: `let j = cl_interval_of_ho (ob_interval_of_zb i)`
dhtslib v0.12.0 (high-throughput sequencing library)
I'm delighted to finally post an official announcement of our package for high-throughput sequencing (HTS), also called Next-generation sequencing (NGS): `dhtslib`. It's not a very clever name, and we are working on a new one. ;) https://github.com/blachlylab/dhtslib/ https://code.dlang.org/packages/dhtslib Once upon a time, BioD[1] was fairly active, but I am afraid D is not heavily used in bioinformatics and computational biology, especially in high-throughput (genome) sequencing applications when compared to its peers.[2] However, our group (cancer genomics) has found D an ideal language which is easy to pick up for Python programmers and yet retains powerful features for C/C++ programmers. `dhtslib` began as a thin wrapper over the ubiquitous, but very low-level and hard to use `htslib` C library (https://github.com/samtools/htslib/). We use `dhtslib` extensively in both public and private projects for computational biology, and over the years it has grown from simply a (huge) set of `extern (C)` definitions to a fully featured, RAII-enabled genome sequencing focused bioinformatics package. If you are working in this field, or know someone open to D who works in this field, I strongly encourage you to point them at `dhtslib`! * `htslib` namespace with complete bindings to htslib * `dhtslib` namespace with high level object-oriented interfaces, many using underlying htslib calls for high performance, but via convenient and idiomatic D including RAII, Forward ranges, etc. * htslib-backed read/write of SAM/BAM/CRAM, VCF/BCF * Readers for BED and GFF3/GTF (not part of htslib) * FASTQ streamer * CIGAR manipulations The next version, v0.13.0, adds a novel feature "Typesafe Coordinates", which I'll post about separately in a moment! Kind regards James S Blachly, MD The Ohio State University [0] https://github.com/blachlylab/dhtslib/ https://code.dlang.org/packages/dhtslib [1] https://github.com/biod/BioD [2] Here is a contemporary example of D used in high-throughput sequencing: DENTIST by Arne Ludwig at Max Planck institute https://github.com/a-ludi/dentist -- if you know of more, please let me know!
Re: Contacting DlangScience maintainers
On 3/28/21 8:58 PM, rikki cattermole wrote: On 29/03/2021 12:16 PM, Chris Piker wrote: On Sunday, 28 March 2021 at 04:06:57 UTC, mw wrote: On Friday, 26 March 2021 at 21:55:18 UTC, Chris Piker wrote: Let's discuss it here: https://github.com/orgs/dlang-community/teams/science/discussions @wilzbach is the maintainer of the group. Sounds good to me, but the link above returns a 404, could be a temporary error. Maybe I'm supposed to join a team first? "A visible team can be seen and @mentioned by every member of this organization." Does this [hiding to non org members] really help D's visibility and adoption? What sorts of things are discussed that do not benefit from openness? For example, I am a bona fide scientist using Dlang, but had no idea dlang-science was even an active group (I was aware of the org, and repos, but assumed it was not very active)
Re: Contacting DlangScience maintainers
On 3/26/21 5:55 PM, Chris Piker wrote: Hi DlangScience I've setup D prototypes for the CDF (Common Data Format) file reading/writing library. Since it's mostly just basic D prototypes for a C library the module's name is deimos.cdf and can be found here: https://github.com/das-developers/deimos.cdf I'm attempting to get the repository hosted within a github organization run by the Goddard Space Physics Data facility. If the SPDF declines to host the repo, could it find a home in the DlangScience organization? I'd be happy to sign up to maintain it. I didn't know that your group existed when I wrote the module or would have contacted you before work began. Instead of responding to this message, please use the Discussion section for deimos.cdf, at least temporarily anyway. Thanks, (cross posted from https://gitter.im/DlangScience/public since that site looks unused) Good luck! Interestingly Github flags this repo as a "C" repo , with the balance of code tilting slightly (>51%) in favor of C (perhaps headers), compared to D. I wonder to what degree this affects overall stats of # Dlang repos on Github?
Re: On the D Blog--Symphony of Destruction: Structs, Classes, and the GC
On 3/18/21 5:21 AM, Per Nordlöw wrote: On Thursday, 4 March 2021 at 13:54:48 UTC, Mike Parker wrote: The blog: https://dlang.org/blog/2021/03/04/symphony-of-destruction-structs-classes-and-the-gc-part-one/ Btw, what is the motive behind D's GC not being able to correctly handle GC allocations in class destructors. Is it by design or because of limitations in D's current GC implementation? And how does this relate to exception-throwing destructors in other managed languages such as C# and Go; are they forbidden or allowed and safe thanks to a more resilient GC? It is a frustrating rough edge, esp for non-experts; I cut myself when I tried to use a custom logging function (which of course GC allocates) in class destructors.
Re: Bubel ECS - Entity-Component-System architecture implementation
On 3/9/21 4:02 PM, Meta wrote: On Tuesday, 9 March 2021 at 15:21:26 UTC, Mergul wrote: On Tuesday, 9 March 2021 at 14:29:37 UTC, Meta wrote: Awesome! In the demo, I noticed that if there are >3000 entities spawned, there will be periodic dips in the FPS. Could this be Javascript's GC kicking in? What browser are you using? I've tested couple of browsers. In general JavaScript shouldn't be a problem, from what I know WASM only growth its memory. Library itself almost never call malloc (so memory is always allocated). On firefox I had problem with FPS most of a time and I'm not sure what cause this problem (I found several bug reports which could be related to that). Besides Firefox every web browser I tested was based on chromium engine. I didn't see any periodic dips, even with <200k entities. Maybe there is some problem related to WebGL or JS Workers. When JS side detects that WASM threads are enabled in browser it loads multithreaded version of WASM code and spawn workers. But overall multithreaded execution is always more stable for me. This is in Chrome 88.0.4324.192 on my 2015 Macbook Pro. Interesting; Here on a 2012 Macbook pro with Firefox I have consistent 60 FPS even at 3500 elements with no periodic drops
Re: I'm creating a game purely written in D with the arsd library
On 1/3/21 1:31 PM, Murilo wrote: On Sunday, 3 January 2021 at 02:42:46 UTC, James Blachly wrote: My final suggestion is that -- given this is a D programming enthusiasts' newsgroup, where we are interested in seeing D code, you might get great feedback on the code itself. Kind regards How did you have access to the source code? Did you use a decompiler? `strings Game |less`
Re: Package for differential equation solvers
On 1/2/21 4:47 AM, Flávio wrote: Hi folks, I have started a repository to port Differential equation solver code to D. The goal is to have efficient implementations in D that have a much simpler interface for users than current C, C++, and FORTRAN libraries. Currently is just a couple of solvers quickly whipped together (probably with bugs) as a proof of concept. Please join me to help create this! https://github.com/fccoelho/D-DifferentialEquations Contributors are very, very, VERY MUCH welcome! Have you talked to Ilya Yaroshenko (9il) ? This might slot in well with libmir.
Re: I'm creating a game purely written in D with the arsd library
On 1/2/21 4:18 PM, H. S. Teoh wrote: On Sat, Jan 02, 2021 at 09:01:17PM +, Murilo via Digitalmars-d-announce wrote: [...] It's because I don't people to know the spoilers, so no one will see the source code. ... Providing source code is mainly for convenience to people who might want to compile it for platforms you do not have (thus spreading the word about your program). Yes, like me for example. My only graphical systems are Mac, whereas my linux machines are all headless. Sadly I have not been able to try the game.
Re: I'm creating a game purely written in D with the arsd library
On 1/2/21 2:10 PM, Murilo wrote:
On Saturday, 2 January 2021 at 05:43:48 UTC, James Blachly wrote:
On 1/1/21 11:12 PM, Murilo wrote:
Here is the link
https://drive.google.com/file/d/1Il1xLN8b5rzghYLXTQqq2apv5YMKv7rx/view?usp=sharing
Bro I would be shocked if people are excited to run a mystery binary
downloaded from Google Drive.
The ELF binary does contain arsd symbols, and at least VMs and Unix
systems give us privilege and resource isolation safety these days.
Still, why not source code on Github?
The reason why I won't show the source code is because I don't want
anyone hacking the game or find out the surprises hidden in the game, I
want the game to be a mystery, you have to play it to know what is gonna
happen. I also don't want anyone stealing my idea.
I promise I don't mean to be rude, just "confrontationally instructive." =)
But as far as the game, the strings are all in the binary:
```
Alabaster Amphora
Beautiful souvenir from Silmaria, you smell hippocrene water.
Those are delicious, and healthy.
Those are shiny gold Dracmas, use it well.
This is a cheap weapon which you can throw at the enemy.
Someone has been looking for this since 2003.
This should be 100 Brazilian Reals.
Eating this might teach me to speak Chinese.
It says Davy Jones on the label of this product.
```
(incidentally it look like a funny game to be sure)
And as far as the code, you have somehow imported the source text of one
of your files into the binary:
```
// Starting items
ItemForSale alabasterAmphora = ItemForSale("Alabaster Amphora",
"Beautiful souvenir from Silmaria, you smell hippocrene water.",
Image.fromMemoryImage(loadImageFromMemory(cast(immutable ubyte[])
import("items/alabaster amphora.jpeg"))),
Image.fromMemoryImage(loadImageFromMemory(cast(immutable ubyte[])
import("icons/icon alabaster amphora.jpeg"))),
200, false),
fruits = ItemForSale("Fruits", "Those are delicious,
and healthy.",
```
Anyway, my point is -- try to not to worry to much that others will
steal your code or your idea. A determined adversary will do what he wants.
My final suggestion is that -- given this is a D programming
enthusiasts' newsgroup, where we are interested in seeing D code, you
might get great feedback on the code itself.
Kind regards
Re: I'm creating a game purely written in D with the arsd library
On 1/1/21 11:12 PM, Murilo wrote: Here is the link https://drive.google.com/file/d/1Il1xLN8b5rzghYLXTQqq2apv5YMKv7rx/view?usp=sharing Bro I would be shocked if people are excited to run a mystery binary downloaded from Google Drive. The ELF binary does contain arsd symbols, and at least VMs and Unix systems give us privilege and resource isolation safety these days. Still, why not source code on Github?
Re: Victor Porton's book "D, the Best Programming Language, for Former Python Developers"
On 1/1/21 11:39 PM, Ali Çehreli wrote: I've come across the book on D Wiki Books. Amazon link: https://www.amazon.com/Programming-Language-Former-Python-Developers/dp/B08M83X6N8 Ali Wow -- A great find Ali! I and my staff find ourselves in the same position -- all former (actually , still current due to some ecosystem needs) Python programmers who find D to be "the best" in terms of productivity and enjoyability. I wish the author, and Dlang, great success.
Re: Computing the next SemVer for dlang packages with dsemver
On 10/21/20 10:47 AM, Robert burner Schadek wrote: https://code.dlang.org/packages/dsemver is a program that computes the next SemVer for your dlang package. As indicated downthread, this could be a great ecosystem partner with dmd-as-a-library. Thank you Robert!
Re: mir-stat
On 10/8/20 12:40 PM, 9il wrote: It is a pleasure to announce the Dlang Statistical Package by John Michael Hall. API http://mir-stat.libmir.org/ GitHub http://github.com/libmir/mir-stat DUB https://code.dlang.org/packages/mir-stat The initial release provides descriptive statistics and algorithms for transforming data that are useful in statistical applications. The very basic stuff like `gmean` [1] is located in the mir-algorithm package, it will be downloaded automatically. The generation of random numbers of various distributions is provided by mir-random package [2]. Outstanding work by all involved. Thank you for driving this (and all of mir) forward. We have already found use in our computational biology lab.
Re: HTTP frameworks benchmark focused on D libraries
On 9/27/20 6:08 AM, tchaloupka wrote: Hi all, I've just pushed the updated results. Thanks for continuing to work on this! vibe-core performs quite well -- scaling up with additional workers from 8 through 256, whereas vibe-d platform tops out around ~35,000-45,000 RPS irrespective of simultaneous workers (plateauing between 8-64 workers). Given the outstanding performance of vibe-core it looks like there is room to continue to improve the vibe-d platform. Cheers again for your work.
Re: HTTP frameworks benchmark focused on D libraries
On 9/20/20 4:03 PM, tchaloupka wrote: Hi, as it pops up now and then (last one in https://forum.dlang.org/thread/qttjlgxjmrzzuflrj...@forum.dlang.org) I wanted to see the various D libraries performance against each other too and ended up with https://github.com/tchaloupka/httpbench It's just a simple plaintext response testing (nothing fancy as in Techempower) but this interests me the most as it gives the idea about the potential of the library. More details in the README. Hope it helps to test some ideas or improve the current solutions. Tom Thank you for doing this! One of the most fascinating things I think is how photon really shines when concurrency gets dialed up. With 8 workers, it performs about as well, but below, the rest of the micro, including below Rust and Go /platforms/. However, at 64 concurrent workers, photon rises to the top of the stack, performing about as well as eventcore and hunt. When going all the way up to 256, it was the only one that demonstrated **consistent performance** -- about the same as w/64, whereas ALL others dropped off, performing WORSE with 256 workers.
Re: Beta 2.094.0
On 9/14/20 5:17 PM, mw wrote:
On Sunday, 13 September 2020 at 16:33:42 UTC, James Blachly wrote:
{
"name": "git-dependency",
"dependencies": {
"gitcompatibledubpackage": {
"repository":
"git+https://github.com/dlang-community/gitcompatibledubpackage.git;,
"version": "ccb31bf6a655437176ec02e04c2305a8c7c90d67"
}
}
}
On a git repo, where to get this version string?
BTW, pass in "master" does not work.
Looks like the commit hash ; use `git log`
Agree a branch name would be nice , then it could automatically take HEAD
Re: Beta 2.094.0
On 9/11/20 3:48 AM, Martin Nowak wrote:
Glad to announce the first beta for the 2.094.0 release, ♥ to the 49
contributors.
This is the first release to be built with LDC on all platforms, so we'd
welcome some more thorough beta testing.
http://dlang.org/download.html#dmd_beta
http://dlang.org/changelog/2.094.0.html
As usual please report any bugs at
https://issues.dlang.org
-Martin
The dub change allowing git repository is HUGE !
This allows us to use a private git repository instead of having to have
the repo already cloned to a specified path on the filesystem.
{
"name": "git-dependency",
"dependencies": {
"gitcompatibledubpackage": {
"repository":
"git+https://github.com/dlang-community/gitcompatibledubpackage.git;,
"version": "ccb31bf6a655437176ec02e04c2305a8c7c90d67"
}
}
}
Re: khash associative array / hash map / hash set
On 8/24/20 5:11 PM, ikod wrote: Thanks, but no ) This hashmap can't replace standard AA for next reason: with standard AA you can safely do: string[int] aa; aa[0] = "null"; auto v = 0 in aa; aa.remove(0); assert(*v == "null"); aa[0] = "one"; assert(*v == "null"); Hm, I see what you mean but I am not sure I love the behavior (or would call it safe habit to be in) to keep a pointer to something that was subsequently deleted (obviously I understand that in case of Dlang AA it is not really a pointer to what was deleted, it is a pointer to existing object and only the rference was deleted from the AA). If one is using pointers one should be mindful of deletes, etc. anyway =) This is because AA allocate memory in GC area for every value it store(and return pointer to it when "in" used), so even if you delete key from AA it is still safe to use pointer to value. But this require GC allocations. Correct me if I'm wrong - your and mine HashMaps avoid allocations and store values inline, so you can't use pointer to values in safe code (value can be deleted, or replaced on next table insertion/deletion). In order to be safe my hashmap do not support "in" operator and always return value. Correct, mine operates similarly (malloc plus GC.addRange), no opBinaryRight!in Also you may find useful safe map modification during iteration over map items (at the cost of creating temporary table copy). In search of absolute speed I am willing to forego this, but certainly it could be useful in concurrency type situation
Re: khash associative array / hash map / hash set
On Monday, 24 August 2020 at 05:51:59 UTC, ikod wrote: On Monday, 24 August 2020 at 01:39:26 UTC, James Blachly wrote: OH, I almost forgot the best part. It is crazy fast. https://attractivechaos.wordpress.com/2018/01/13/revisiting-hash-table-performance/ https://attractivechaos.wordpress.com/2019/12/28/deletion-from-hash-tables-without-tombstones/ My naive benchmark shows -- compared to emsi_containers.HashMap -- 30% faster inserts, 80% faster serial retrieval and 70% faster random retrieval. Perhaps I am doing something wrong? James Thanks, nice job! You also may want to compare performance with https://github.com/ikod/ikod-containers, just add dependency from ikod-containers, then: import ikod.containers; and use ikod.containers.hashmap.HashMap as alias for container. I squeezed everything I was able from the open-addressing hash map. Nice, thank you and great job! Performance looks very comparable and I would be happy to use your package as well. Perhaps it is time that Dlang have a faster canonical hashmap (phobos.next ?) ``` hashmap benchmarks Inserts for HashMap finished in 518 milliseconds. Inserts for khash finished in 549 milliseconds. Serial Lookups for HashMap finished in 21 milliseconds. Random lookups for HashMap finished in 41 milliseconds. Confirming stored value of last lookup: 7353ece9-506c-467f-9cb4-7686426fa828 Serial Lookups for khash finished in 12 milliseconds. Random lookups for khash finished in 36 milliseconds. Confirming stored value of last lookup: 1164a2f1-e6cb-4072-89d9-23cec5cadd95 ``` Repeated tests show that ikod.containers' HashMap is consistently faster on insertions, while khash is consistently faster on retrieval.
khash associative array / hash map / hash set
In addition to our interval tree implementation ( https://forum.dlang.org/thread/rhrgbl$2n1h$1...@digitalmars.com ), I wanted to share another tool we have been using extensively here, a D templates port of attractivechaos' klib (https://github.com/attractivechaos/klib) component khash. https://code.dlang.org/packages/dklib https://github.com/blachlylab/dklib I namespaced this as `dklib.khash` as ultimately we will place some other klib components in this package as we use them (kavl for instance is already used in our intervaltree implementation) Although it is a relatively faithful straight port of the C code, I tried to turn attractivechaos' classic C-style generic programming (#define hell) into templated code which looks a bit awkward in places, but I think turns out very nicely in practice. opAssign, opIndex, .require, .byKey() are all implemented for ergonomics. As always, as computational biologists rather than professional software engineers, we would be glad to hear feedback and hope others will find this useful. OH, I almost forgot the best part. It is crazy fast. https://attractivechaos.wordpress.com/2018/01/13/revisiting-hash-table-performance/ https://attractivechaos.wordpress.com/2019/12/28/deletion-from-hash-tables-without-tombstones/ My naive benchmark shows -- compared to emsi_containers.HashMap -- 30% faster inserts, 80% faster serial retrieval and 70% faster random retrieval. Perhaps I am doing something wrong? James
Re: SAOC 2020 Projects and Participants
On 8/23/20 4:44 PM, Mike Parker wrote: If you're interested in following along with the progress of Symmetry Autumn of Code 2020, you can start with this blog post, where I briefly introduce the participants and their SAOC projects: https://dlang.org/blog/2020/08/23/symmetry-autumn-of-code-2020-projects-and-participants/ Outstanding; congratulations to students and mentors. All of the projects are very exciting: Google Cloud APIs, generic data structures, DMD as library, minimal ARM runtime, Bison GLR parser. I am especially excited for DMD as a library -- this could enable some really exciting IDE functionality with respect to templates and code analysis.
Interval Trees library intervaltree
I am happy to share our interval tree library we built for some internal computational biology projects. I would also be very happy to have your feedback on implementation and execution as I am certainly no professional software engineer. James # Sources https://github.com/blachlylab/intervaltree https://code.dlang.org/packages/intervaltree # Discussion Implemented are AVL trees, Splay trees, and a newer type of packed array structure being explored in genomics, the Implicit Interval Tree (IIT; more info in documentation) The trees are implemented as generic templated containers for whatever payload you wish to include. Our applications require a payload, but certainly there are important applications of interval trees that require none. I haven't thought about whether one could include a NoneType, empty struct, etc. but would be glad to hear ideas. # Notes @nogc status: Currently everything is @nogc except the `findOverlapsWith` function, which returns a dynamic array of matching intervals. Since API remains unstable until 1.0 I would also be open to changing this (e.g. to a preallocated reusable buffer although this would make the structure non-threadsafe) if @nogc is important to library users. GC warning: since we manage our own memory (with stdx.allocator and arena allocations via Mallocator), placing objects in the tree having pointers to GC-managed memory (e.g. strings) will bite you hard, unless you use the IITree, which currently does include code that defaults to include `GC.addRange()` at `insert`. In the future I will likely make all treetypes `GC.addRange` over their entire arenas to improve safety AND speed `insert()` # References Genesis: https://forum.dlang.org/post/qxaqonnyezgtwxjqw...@forum.dlang.org
Re: code.dlang.org downtime
On 1/9/20 10:03 PM, suncarpet wrote: I can't say I like the idea of requiring users to use the aforementioned or similar services and necessitating the use of git over other VCS solutions. How is it that dub works today?
Re: code.dlang.org downtime
On 12/17/19 12:34 PM, H. S. Teoh wrote: This looks like something the D Foundation ought to fund. Buy one of the many VPS's out there to be used as the "official", Foundation-backed dub server, preferably with a hosting provider that has multiple redundant network connections and automatic backup service. Depending on availability of funds, add a hot failover server as backup in case things go wrong with the primary server. This ought to alleviate any concerns about downtime. ... I am sorry to post on an older (but < 30 days!) thread that I've just seen. While it would be very reasonable for the Foundation to be running this infra, Sebastiaan's proposal is quite right -- eliminate the need for a server entirely. I would add one thing to his proposal: The metadata are already stored in github/bitbucket/gitlab/etc. -- in the form of the dubfile. All we need is a K:V lookup mapping package name to canonical git repository. This K:V store could be served perhaps more cheaply , and more reliably, than the current infra which must serve all metadata (presumably). If dub could be retooled to only query for the git repo, it could then clone and work entirely with metadata stored in the repo, minimizing dependence on a hosted server, whether 6 nines reliable or not. If, though, the foundation takes over the current infra and hosting, please put out a call for donations -- I will donate.
Re: The DLang Swag Emporium is Open For Business!
On 8/1/19 11:30 AM, Mike Parker wrote: I know some of you have been waiting for this and I apologize for the delay. You can now get your hands on some DLang swag and support the D Language Foundation at the same time. See the blog post for details: https://dlang.org/blog/2019/08/01/the-dlang-swag-emporium-and-more/ Looking great. Can you set up die-cut stickers of the D Rocket logo?
Re: Variant Graph Support to BioD
On 5/28/19 5:41 AM, Njagi Mwaniki wrote: Hello I’m Njagi Mwaniki, I am part of the 2019 Google Summer of Code under the Open Bioinformatics Foundation with a project aimed to add variation graph support to BioD under mentors George Githinji and Pjotr Prins. ... VG is a set of tools that already implements variation graphs but which is a bit broad in its focus. In this project we are building upon the existing tools and knowledge from VG and looking for ways to improve its performance in terms of lookups and also its application with small genomes, specifically viruses and smaller mammals such as mice. This sounds like a great project. Be aware that the size of the organism (e.g. mouse) has naught to do with the size of its genome.
Re: eBay's TSV Utilities status update
On 4/29/19 11:23 AM, Jon Degenhardt wrote: An update on changes to this tool-set over the last year. ... The other main update is improved I/O read performance in many of the tools. This is from developing a buffered version of byLine. It is especially effective for skinny files (short lines). Most of the tools saw performance gains of 10-40%. One of the earlier performance improvements came from buffering output lines. Combined, the line-by-line read-write performance is quite a bit faster than what is available in Phobos. The iopipe / std.io packages (Steve Schveighoff, Martin Nowak) are faster still, these are the place to go for really high performance. (See the links below for a benchmark report.) Links: * tsv-utils repo: https://github.com/eBay/tsv-utils * tsv-sample user docs: https://github.com/eBay/tsv-utils/blob/master/docs/ToolReference.md#tsv-sample-reference * tsv-sample code docs: https://tsv-utils.dpldocs.info/tsv_utils.tsv_sample.html * Performance benchmarks on line-oriented I/O facilities: https://github.com/jondegenhardt/dcat-perf/issues/1 Jon: Thank you for this, and thanks for your blog post of a couple of years ago, which I referred to many times while learning D and writing fast(er) CLI tools. Looking forward to trying Steve's iopipe as well as your bufferedByLineReader. James
Re: DStep 1.0.0 on the Blog
On 4/23/19 3:23 PM, Jacob Carlborg wrote: On 2019-04-23 14:10, Mike Franklin wrote: Nice work, Jacob! Very cool! Thanks :) Seconded -- absolutely fantastic
Re: LDC 1.15.0-beta1
On 3/10/19 7:41 AM, kinke wrote:
On Sunday, 10 March 2019 at 02:05:37 UTC, Manu wrote:
Can you explain what this means:
* Fix: functions annotated with `pragma(inline, true)` are
implicitly cross-module-inlined again.
`pragma(inline, true)` functions have only been inlined in the same
compilation unit since LDC v1.1 (without explicit
`-enable-cross-module-inlining`). Now they are inlined across
compilation units again, as before v1.1 (and independent from the -O
level). E.g., this means that you don't need LTO to get rid of calls to
std.math trampolines for LLVM intrinsics such as:
pragma(inline, true)
real fabs(real x) @safe pure nothrow @nogc { return llvm_fabs(x); }
THANK YOU!
I did not know about this regression.
I get about a 4.5% speed increase on my limited dataset with
ldc-1.15-beta1 compared to ldc-1.14. Hopefully will be more on larger
dataset (smaller set overwhelmed by startup costs).
