Re: Nim vs D
I like D quite a bit, and there's clearly been some convergent evolution (UFCS) with Nim. The D story with respect to memory management is still unfolding. As was already said, I think the language could have been designed to make GC easier, perhaps separating **ref** and **ptr** like Nim. Then it might not be in the current mess with the GC. I like Nim the language more. I'd like it if Nim modules could be locally imported, like D ones, but I'm pretty sure Araq doesn't like that idea. IMO, the main advantage of D vs Nim is that the language is already well past it's '1.0' release and has a larger community. It feels like a safer bet for many industrial users, though not as safe as any mainstream language. Nim is still more of a work in progress than D, [parts](file://Users/bpr/Applications/Nim/doc/html/manual.html#generics-vtable-types) of the manual aren't implemented, the OO and async stories are unfolding, the community is tiny, and most discussion appears to happen on the IRC. It's still an 'early adopter' language.
Re: Tutorials and documentation for a nim project
I've used Nim's Docgen to generate a tutorial + documentation. End result is [here](https://mratsim.github.io/Arraymancer/) which I'm quite happy with for the time invested. A few remarks to ease its use: * It would be nice to prepend the autogenerated "API" doc with rst tutorials. As a workaround, I include a .nim file [full of commentaries](https://github.com/mratsim/Arraymancer/blob/087b5c82addf90fd3a569364613e89b683f011a1/docs/autogen_nim_API.nim) * I cannot seem to display images this way. * [See source switch](https://nim-lang.org/docs/docgen.html#related-options-see-source-switch) doesn't work. * It ignores `docSeeSrcUrl`. [Issue raised](https://github.com/nim-lang/Nim/issues/6071). Even if I don't specify it does a wrong link. * It seems non-trivial to link against a specific github commit * The `edit` link links to a devel branch (hardcoded). I used find+replace to fix "See source issues" but it would be nice if automated.
Re: Generic Pointer Question
Something like the following should work (warning, untested code): type Val* = object mvSize*: csize #*< size of the data item mvData*: pointer#*< address of the data item proc valToString*(v: Val): string = result = newString(v.mvSize) copyMem(addr result[0], v.mvData, v.mvSize)
Re: Generic Pointer Question
The [NESM](https://github.com/xomachine/NESM) library may also give you a hint. It [de]serializes strings into|from structure you described. [Documentation here](https://xomachine.github.io/NESM/) So the code will be: from nesm import serializable serializable: type Value = string # Actual memory representation of serialized string equals to your structure ... # The "datasource" value is a stream that provides bytes of your structure ... # Lets assume it declared and configured over here (e.g. you may use newStringStream for string input) let val = Value.deserialize(datasource) echo val The underlying implementation of the deserialize proc uses copyMem and is being generated at compile time.
Re: Generic Pointer Question
> I guess I could add a NUL character to the end of the input strings, which > would allow me to cast to cstring on retrieval. If you can add the '0' character with which C strings are generally terminated, then you do not need alloc() and copyMem(). Bytes sequences terminated with '0' looks like regular C strings, so you should be able to cast to a C string and assign to a regular Nim string like "var str: string = $cast[cstring](myByteData)". The $ converts from cstring to nim string.
Re: Generic Pointer Question
Hi Stefan, the database I connected to was populated using strings, so no NUL terminators. If I have control of database load, I guess I could add a NUL character to the end of the input strings, which would allow me to cast to cstring on retrieval. Thank you for the response...it was helpful! Now I need to learn about alloc() and copyMem().
Re: Nim vs D
This recursion unpacking/unrolling trick that gcc does (at call-site if insulated by call via volatile function ptr, and always inside the recursive impl) is, in my experience, a rare compiler optimization, but maybe it will catch on. clang does _neither_. If you `objdump -D` the executable (or disassemble in `gdb`/etc.) you will see the single `callq` to Fibonacci at the entry with the full N and a pair of `callq` inside the impl. So with clang the full [1.618**n work](https://stackoverflow.com/questions/360748/computational-complexity-of-fibonacci-sequence) happens. On my i7-6700K Linux with gcc-7.1.0&clang-4.0.1, I get a time ratio of about 15.3 to 1 (53.5s/3.5s). For what it's worth, it's likely the compiler writer guys think of this as mostly a "remove a few funcalls here and there" type optimization..maybe use SSE/AVX/branch predictor a bit better, maybe save a little dynamic stack usage, etc. This doubly recursive Fibonacci approach just happens to be about as sensitive as possible to that particular optimization because it can save exponentially many funcalls.
Re: Arraymancer - A n-dimensional array / tensor library
`get_data_ptr` is now public . For now, I will add the neural network functionality directly in Arraymancer. The directory structure will probably be: * src/arraymancer ==> core Tensor stuff * src/autograd ==> automatic gradient computation (i.e. [Nim-rmad](https://github.com/mratsim/nim-rmad) ported to tensors) * src/neuralnet ==> neural net layers This mirrors [PyTorch's tree](https://github.com/pytorch/pytorch/tree/master/torch) I made this choice for the following reasons: * It's easier for me to keep track of one repo, refactor code, document and test. * I'm focusing on deep learning * It's much easier to communicate about one single package (and attracts new people to Nim ). * Data scientists are used to have deep learning in a single package (tensor + neural net interface): Tensorflow, Torch/PyTorch, Nervana Neon, MxNet ... * Nim's `DeadCodeElim` will ensure that unused code will not be compiled. If the tensor part (without the NN) get even 0.1% of Numpy popularity and people start using it in several packages that means: * It's a rich man problem! * We get new devs and input for scientific/numerical Nim. * We can reconsider splitting as we will know actual expectations. * We can even build a "scinim" community which drives all key scientific nim packages. In the mean time I think it's best if I do what is easier for me and worry about how to scale later.
Re: Arraymancer - A n-dimensional array / tensor library
I've been following this for a while on GitHub and I think it is a very impressive project. Nim would be a great language for scientific computing, but it needs to have the numerical libraries and this is an excellent first step in creating them. A couple of questions. First, are you planning to add neural network functionality directly to Arraymancer? Surely that would be something better suited for a separate, specialised library? A second, more general, question I have is whether you'd consider making the [get_data_ptr](https://github.com/mratsim/Arraymancer/blob/master/src/arraymancer/data_structure.nim#L62) proc public. It would be nice to be able to integrate your tensors with wrappers for existing numerical software written in C and we'd need access to the raw data for that.
Re: Tutorials and documentation for a nim project
@euant no worries ! @Araq I had to update to the latest devel. It didn`t work with my nim install from the middle of May. I checked solutions that could be used to upload to Github Pages easily and was recommended [Couscous.io](http://couscous.io/). Workflow seems to be: * Have a directory of Markdown files * `couscous preview` to check rendering on local machine * `couscous deploy` to commit on `gh-pages`
Re: Generic Pointer Question
What do you desire as the result? When it is a Nim string, then you could create a new Nim String (newString(size)) of at least the desired size, and fill in the characters. To access the source characters, you may temporary cast your data to a cstring and access the cstring with the [] operator. But is you want a plain C string with 0 termination -- well that is what we may get from the OS, you should not desire that. Otherwise you may need something like alloc() to allocate untraced memory. If you think copying characters one by one is too slow -- something like low level copyMem() may be possible also, ask again...
Generic Pointer Question
I'm testing the LMDB package and I'm stuck. When I retrieve key/value pairs they are saved to MDB_val structures: type Val* = object mvSize*: csize #*< size of the data item mvData*: pointer#*< address of the data item I tried casting the generic pointer (mvData) to a cstring, but the data is not zero terminated so I get garbage attached. Knowing the starting address and number of bytes, what is the best way to capture the data (which is a string in my instance)? Thanks!
Re: Nim vs D
I am using mingw64/clang64 in Msys2. But I found that the exe file build by clang runs very slower than exe by gcc. The nim is a fresh build from github source yesterday $ nim Nim Compiler Version 0.17.1 (2017-07-07) [Windows: amd64] Copyright (c) 2006-2017 by Andreas Rumpf for the mentioned nim file (a.nim) proc fibonacci(n: int): float = if n < 2: result = float(n) else: result = fibonacci(n-1) + fibonacci(n-2) echo fibonacci(50) first, for cc = gcc in nim.cfg $ gcc --version gcc.exe (Rev2, Built by MSYS2 project) 6.2.0 Copyright (C) 2016 Free Software Foundation, Inc. $ nim c -d:release --out:a_gcc.exe a.nim $ du -b ./a_gcc.exe 457546 ./a_gcc.exe $ time ./a_gcc.exe 12586269025.0 real0m5.164s user0m0.015s sys 0m0.015s Then use cc = clang in nim.cfg $ clang --version clang version 3.9.1 (tags/RELEASE_391/final) Target: x86_64-w64-windows-gnu Thread model: posix InstalledDir: E:\msys64\mingw64\bin $ nim c -d:release --out:a_clang.exe a.nim $ du -b ./a_clang.exe 423480 ./a_clang.exe $ time ./a_clang.exe 12586269025.0 real2m2.055s user0m0.000s sys 0m0.015s
