Re: Statistics for standard library usage
> Is this an official source of things? If so, I'll create PRs for both > [https://github.com/nim-lang/website](https://github.com/nim-lang/website) > and > [https://github.com/nim-lang/packages](https://github.com/nim-lang/packages) > to add references. If not official, I'll just update the wiki perhaps. We want to make it official.
Re: Statistics for standard library usage
> It would contain less code, therefore less bugs It would also be less useful. > Nim language reference would be smaller and therefore compiler implementation > easier The language is not its stdlib. > It would be easier to develop a test suite that signals the > production-readiness of Nim We have a good test suite. > It would strengthen Nim's status as a systems programming language This sentence lacks substance. > There would be more time to develop new features instead of maintaining > legacy code Depends on how we do it. "The stdlib modules that nobody uses and thus get no bug reports and receive no commits" are not exactly slowing down Nim's development... I can see the value of cleaning things up, don't get me wrong, and it's important for version 1 but let's be realistic, it won't make developing a WebAssembly backend any easier.
Re: Statistics for standard library usage
quick correction regarding my earlier post: apparently there is already one website for nimble packages: [https://nimble.directory](https://nimble.directory)/ I just happened to come across it via a random google query. I don't see it referenced on nim-lang.org. Or even on the packages repo. It is on the wiki; though not under the packages part. It appears to be coming from [https://github.com/FedericoCeratto/nim-package-directory](https://github.com/FedericoCeratto/nim-package-directory) Is this an official source of things? If so, I'll create PRs for both [https://github.com/nim-lang/website](https://github.com/nim-lang/website) and [https://github.com/nim-lang/packages](https://github.com/nim-lang/packages) to add references. If not official, I'll just update the wiki perhaps.
Re: Statistics for standard library usage
>From the previous threads, I think there is two important thing to do: * Maintain the `nim compiler and core library` and keep them progress, which make the nim appeal to people who like a wonderful program language. * The `big standard package` which make most of user can develop with nim immediately, rather than spend lots of time to find/test/compare appropriate library in the nimble package. These two things are equals important, and I think they need different groups of people to manage them. Sorry if I doesn't make myself clear, English is not my native language.
Re: Statistics for standard library usage
I think a core std lib for 1.0 would solve the following problems: * It would contain less code, therefore less bugs * Nim language reference would be smaller and therefore compiler implementation easier * It would be easier to develop a test suite that signals the production-readiness of Nim * It would strengthen Nim's status as a systems programming language * There would be more time to develop new features instead of maintaining legacy code Ways to install Nim have nothing to do with the std lib quality/size trade-off. The end user should only be offered the Nim standard distribution. Nim core should be interesting only for people writing alternative compilers or having a non-standard OS like FreeRTOS. "Full experience distros" should be unofficial and their maintenance out-of-scope for Nim core devs. Every layer should build upon the lower layer, therefore testing and maintenance should become easier, not harder. If you integrate networking stack etc. to the compiler and continuously bootstrap the compiler with stdlib novelties and vice versa, Nim would not be just a programming language, it would become a programming environment (more like .NET than C). Nothing wrong with that - I would use Nim anyway - but making an alternative implementation would then become very hard.
Re: Statistics for standard library usage
Agreed with what @Araq said. Mantaining a sort of enlarged standard library seems a lot of opinionated work. And this can be done externally with a small tool that just takes a list of Nimble packages and `nimble install` them all
Re: Statistics for standard library usage
Well sadly these plans look more and more problematic to me. Nim - Core Everything `koch boot -d:release` and `koch tools` needs. Nothing more, nothing less. Problems: * The Nim compiler can often benefit from new stdlib developments too. * nimsuggest needs the networking stack. * The libraries that Nim/Nimble need are not fixed. Nim - Standard * What to include? All the official packages I guess. Nim - the full experience (TM) * Nim-Standard plus even more packages. Which ones? Why? And then on top of having 3 different ways to install Nim (zip, choosenim, build from source; which is at least one too many) we offer them 3 separate "distributions" that all have to be tested and maintained (!). And what problem does it even solve? To have some philosophically pleasing "small core"? To see if a module is essential for porting, remove it and see if "koch boot" still works...
Re: Statistics for standard library usage
This is valuable data. When creating my [proposal](https://forum.nim-lang.org/t/3772/5), I manually inspected all Rosetta Code Nim solutions and got very similar results. The top five modules where: * strutils * math * os * sequtils * times Of course data from real applications is still scarce. For example, _logging_ and _unittest_ should rank very high in reality. * * * Anyone demanding an all-batteries-included-stdlib for Nim 1.0 release should consider the enormous amount effort needed for reaching and maintaining production-ready quality. Below are some examples. I don't see why Nim could be the exception. [C (sample)](https://sourceware.org/git/?p=glibc.git;a=blob_plain;f=string/tester.c) Boost: * [Boost.Test](https://www.boost.org/doc/libs/1_67_0/libs/test/doc/html/index.html) * [Conformance status](https://www.boost.org/doc/libs/1_67_0/doc/html/thread/compliance.html) Python: * [doc](https://docs.python.org/3/library/test.html) * [code](https://hg.python.org/cpython/file/tip/Lib/test) [Rust](https://brson.github.io/2017/07/10/how-rust-is-tested) Java: * [JCK](http://openjdk.java.net/groups/conformance/JckAccess/) * [JDTS](https://en.wikipedia.org/wiki/Java_Device_Test_Suite) * [SigTest](https://docs.oracle.com/javame/test-tools/sigtest/2_2/sigtest2_2_usersguide.pdf) [Javascript](https://github.com/tc39/test262)
Re: Statistics for standard library usage
A "Nimble Canonical" category makes sense, but many other repositories (npm, IDE marketplaces etc.) crowdsource the decision of what's "canonical" by showing usage numbers and offering a rating system. If nimble had this, maybe even with a more differentiated rating system (compliance with best practices and conventions, performance, API cleanliness) maintained by curators... But I'm dreaming.
Re: Statistics for standard library usage
Adding a short opinion regarding what should be in the standard library for v1.0. Keep in mind my background is mostly python, however, I'm still a newb with Nim. One of Python's selling points is the large standard library that is distributed. But one of it's flaws, however, is that it only has two levels of libraries: * Standard Libraries * All external libraries (PyPi) That means if needed library is not in standard, it is _very_ non-obvious which competing library is a good one. One has to hunt-around the similar libraries and try to get a "general impression" and hope you are right. Looking at repo update traffic helps, etc. My recommendation for nim: Standard Library: These are libraries installed with the compiler itself. Always available. * Keep it large * but if a lib is not ready to be "frozen" to v1.0, keep it out; you can always easily add it in later. But change becomes problematic once it is distributed with the compiler. * Do not allow libraries dependent on external software projects. So, remove "db_postgres" and "db_mysql", for example. These can, potentially, change too fast for a standard lib. Dependencies on external protocols (such as http) are okay, however, as they change more slowly. Nimble Libraries: * Eventually add a website and server system to gather stats; it should use the JSON distribution (not replace it.) * Helps create consensus. * Possibly add forums for packages * Add **voluntary** collection system to nimble that sends a "I just installed this" to that server to help with stats. ...and, something not seen elsewhere, a possible third category... Nimble Canonical: This is a list of libraries "highly recommended" by the nim-lang developers. * Could be a single "nim-canon" distribution package on some OSes. Otherwise, requires nimble. * This is where libraries pending "standard lib" are placed. * perhaps all repo'd at something like github.com/nim-canonical * Part of the point is to focus developer attention Just ideas... BTW, I've toyed with the idea of arbitrarily creating the nimble website; as I would have found it useful myself. But I don't want to step on anyone's toes.
Re: Statistics for standard library usage
I agree that a big standard library is useful, and that there are areas that could be expanded (e.g I think the stdlib should include a `heap` implementation). But it only works if there are contributors interested in maintaining the modules. For some of the current modules, that's not the case. Take a look at `xmldom` and `xmldomparser`. They are used in `0.2%` and `0.5%` of projects, they haven't been touched in a long time (`xmldomparser` hasn't had a commit in over 2 years), they use outdated naming conventions and so on. Since obviously nobody cares about these modules, why are they included in the standard library? Another advantage of Nimble packages is that they are updated independently. Updating the stdlib is an all-or-nothing kind of thing. Either you deal with all the breaking changes in the compiler and stdlib at once, or you don't update at all. This usually means that stdlib modules are very limited in what kind of breaking changes can be done - this is why we have `parseopt`/`parseopt2`, `re`/`nre` and so on. Because of this, bad design choices in standard libraries has a tendency to stay around for ever (examples of this can be found in any popular language).
Re: Statistics for standard library usage
> I am against reducing standard library. Fully agreed! To the contrary, I would prefer to expand it.
Re: Statistics for standard library usage
@GULPF That's very interesting, actually! I never thought I'm SO mainstream using so many macros.
Re: Statistics for standard library usage
Very interesting!
Statistics for standard library usage
Since there's been some take about reducing the standard library, and because I was curious, I decided to generate some statistics for usage of the standard library in GitHub projects. I used the top 1000 most starred Nim projects on GitHub. * * * **Results:** [https://gist.github.com/GULPF/6d49e74af9992f8fc65476a9264488a0](https://gist.github.com/GULPF/6d49e74af9992f8fc65476a9264488a0) Some observations: * `re` is used significantly more than `nre` * `parseopt` is used more than `parseopt2` * About ~25% of the projects use the `macros` module * `posix` is used about twice as much as `winlean` (not surprising) Note: I based the list of stdlib modules on [this](https://nim-lang.org/docs/theindex.html) listing, which includes some modules that are not listed on the main documentation page.
