date:20180321

Hey I have another good one in the category "ignore the compiler and just go 
back to the code".


import db_mysql, tables

type XchgSym = tuple[xchg: string, sym: string]
var xchg_symbol2id = initTable[XchgSym, int]()

# real code:
#let id = db.tryInsertId(sql(qs), m.xchg, m.sym)
# simulation of the result for a stateless environment:
type RetTofTryInsert = type(tryInsertId(open("", "", "", ""), sql""))
let id: RetTofTryInsert = 1
xchg_symbol2id[(xchg:"", sym:"")] = id


change the last line to: 


xchg_symbol2id[(xchg:"", sym:"")] = int(id)


and now it builds.

the compiler output is unhelpful:


comp.nim(11, 15) Error: type mismatch: got (Table[comp.XchgSym, 
system.int], tuple[xchg: string, sym: string], RetTofTryInsert)
but expected one of:
proc `[]=`(s: var string; x: Slice[int]; b: string)
proc `[]=`[T](s: var seq[T]; x: Slice[int]; b: openArray[T])
proc `[]=`[I: Ordinal; T, S](a: T; i: I; x: S)
proc `[]=`[Idx, T](a: var array[Idx, T]; x: Slice[Idx]; b: openArray[T])
proc `[]=`[Idx, T](a: var array[Idx, T]; x: Slice[int]; b: openArray[T])
proc `[]=`[A](t: var CountTable[A]; key: A; val: int)
proc `[]=`[A](t: CountTableRef[A]; key: A; val: int)
proc `[]=`[A, B](t: TableRef[A, B]; key: A; val: B)
proc `[]=`[A, B](t: var Table[A, B]; key: A; val: B)
proc `[]=`[A, B](t: OrderedTableRef[A, B]; key: A; val: B)
proc `[]=`[A, B](t: var OrderedTable[A, B]; key: A; val: B)


Because the type it expects, is in fact not listed here. otherwise we'd have my 
tuple showing up.

The red tilda in VS code (and the column number in the compiler output) points 
to the first [ bracket of the key.

The only clue that got me to save the day, was that it was not showing 
RetTofTryInsert but i64, (I introduced RetTofTryInsert just for the sake of the 
argument). In fact the problem was on the other side of the assignment. Because 
it's NOT an assignment, it's an []= operator and that is very surprising. I 
have no i64 in my types, so changing to int was the fix.

Re: Nim syntax Quiz

2018-03-21 Thread Araq

Unfair quoting IMO, I said

> Or better yet, you ask a beginner who can read a tutorial again to look at 
> some valid syntax. **I 'm not a fan of pathos.**

A friendlier tone would have produced a friendlier answer. But I'm changing 
this to "just always be nice anyway", trying to listen to complaints. 
Independent of the tone I tried my best to improve the situation.

> Because that's what we do, we copy paste a compiler's complaint in google and 
> see if we're not faced with a common mistake, that is solved by checking the 
> first answer on SO as first result in the google response. I've been out of 
> misery in 10 seconds like that for lots of things. Never with nim.

"Nim is not Google friendly" is definitely Nim's problem number one for 
learners, agreed.

Re: Nim syntax Quiz

I'm also a beginner and I'm also exasperated by error messages just like you. 
Let's take C#, it's a language that was designed on purpose to be beginner 
friendly, and the compiler outputs messages that are incredible and spot on. 
This makes it frustration-free, and that's one key to adoption.

nim has a steep entry slope, I'm climbing it slowly but it's not without 
frequent requestioning "should I just scrap my proto and move to C# and redo it 
all in 2 days ?"

@Araq your attitude doesn't help with this feeling I'm afraid :'( you often are 
rough with your tone, and stuff like "a beginner who can read a tutorial again" 
that's... hmm

Ok, so the last way too long-lasting head scratching I had (I'll start 
collecting them like you @Allin lol) was when using ".items" and ".keys" 
iterators, it gives out weird errors about fields. I mentioned it in this SO 
question 
[https://stackoverflow.com/q/49407160/893406](https://stackoverflow.com/q/49407160/893406).
 I'm trying to flood SO with everything I can ask that meets the SO standards. 
Because right now, google returns few useful stuff for common beginners 
queries, or the compiler's error messages

Because that's what we do, we copy paste a compiler's complaint in google and 
see if we're not faced with a common mistake, that is solved by checking the 
first answer on SO as first result in the google response. I've been out of 
misery in 10 seconds like that for lots of things. Never with nim. I hope this 
improves.

Re: how to read/write object from/to binary file?

While doofenstein makes some good points especially about fragility with regard 
to adding new fields and old data files, there is some real efficiency charm to 
a "just memory map and go" pure binary native data approach. It's not always 
fun to be parsing data files again and again. So, I would also add another way 
to read the data. Note that you should probably use the `{.packed.}` pragma, 
note the use of `ptr UncheckedArray` which should really alert the reader to be 
careful, note the commented out inference of the number of objects in the file. 
My example code assumes you wrote out at least two objects from code like you 
had above (with a `{.packed.}` pragma). It will probably crash if the file is 
too short, etc. Just to give you a flavor. 


import memfiles
type
  MyType* {.packed.} = object
somefield*: array[10, int16]
  MyTypes* = ptr UncheckedArray[MyType]
var t: MyType
var f = memfiles.open("/tmp/tmp.mytype")
# infer n object from file size (may need packed); Do own range checking!
# let n = f.size / sizeof(MyType)
let myobs = cast[MyTypes](f.mem)
echo myobs[0].somefield[1]
echo myobs[1].somefield[0]

Re: how to read/write object from/to binary file?

2018-03-21 Thread doofenstein

Because it's very dangerous.

Simply having a compiler which puts structures in another layout or adding a 
field to the object is enough to render all your previously generated files 
useless.

If you just want to save and load data, I would rather serialise 
it([https://nim-lang.org/docs/marshal.html](https://nim-lang.org/docs/marshal.html)
 ). If you want something more compact then e.g. JSON or YAML take a look at 
msgpack: 
[https://nimble.directory/pkg/msgpack4nim](https://nimble.directory/pkg/msgpack4nim)
 (note: I haven't used msgpack with Nim, but I had good experiences with it 
with other programming languages)

Re: OrderedTable is not an ordered table

Thanks for the compliments. You should probably learn about backward shift 
deletion. It's a little tricky the first time you see it, but ultimately it's 
better than tombstones and a bunch of ad hoc garbage collection rules. You can 
look at `lib/pure/collections/tables.nim`. There are really only a few 
improvements to Nim's `Table` that I think might be good.

We could do Robin-Hood reorganization on inserts, being sure to retain linear 
probing and backward shift deletion. In my tests, Robin Hood costs about 10% 
more time on workloads which are mostly successful lookup but saves about 50% 
of the time for workloads that are mostly failed lookups. (Note that every 
insert of a novel element starts with an unsuccessful lookup.) So, Robin Hood 
never adds much on average and is often a net win on "many workloads" and as a 
bonus squashes the variance of lookup times.

The second idea to speed up might be storing the hash codes in a separate array 
from the keys so that a single cache line can fit a larger collision cluster. 
That speeds up searches, but it also slows down inserts and deletes which need 
to hit twice as many memory spots. So, your mileage may vary even more than RH.

Finally, Tables could also elide saving the hash code for `SomeInteger`-like 
"cheaply hashable/comparable" keys. Unlike the prior two ideas, the speed up 
from this would not be workload dependent. However, to do it there needs to be 
either a separate occupied bit vector (two cache lines again, like a segregated 
hash code array) or a special reserved key value signifying "empty". The 
reserved value is the efficient way to go and just one is usually not too 
onerous (zero, -1 or -2**31, etc.), **but** it 's not an easy thing to default. 
So, that expands the API to the user - for just some type classes of key - to 
specify the empty value. So, a specialized name like `SomeIntTable` analogous 
to the specialized histogram `CountTable` is probably warranted.

Re: How to (de)serialize inherited object?

Probably there's no easy solution of this task

[Same for c++ on Stack 
Overflow](https://stackoverflow.com/questions/3268801/how-do-you-de-serialize-a-derived-class-from-serialized-data)

[Comprehensive 
guide](https://isocpp.org/wiki/faq/serialization#serialize-inherit-no-ptrs)

Re: how to read/write object from/to binary file?

Thank you, it works. It strange that there is no possibility to make some cast 
from string of seq[byte] directly to object

Re: How to (de)serialize inherited object?

It would be too easy! 


import streams, nesm

serializable:
  type
TA = object of RootObj
TB = object of TA
  f: int

var
  a: ref TA
  b: ref TB

new(b)
a = b

echo stringify(serialize(a))



lib/nim/system.nim(2833, 7) Error: unhandled exception: kind(thetype[1]) == 
nnkEmpty
 Inheritence not supported in serializable

Re: OrderedTable is not an ordered table

@cblake omg such a concentration of goodness in this answer, thank you for the 
tombstone warning. If I understand what you mean, I gave it some thought at the 
time, and concluded it would be an idea to "garbage collect" if some metric 
goes red, which is the solution google dense map appears to have chosen (auto 
rehash at 50% deleted). In the meantime, I did a much simpler mitigation, line 
832 explains: I don't mark all deletion as deleted if the condition allows for 
return to clean/empty state. I'm relieved to see that nim is in the hands of 
better experts than the feeling it gives out.

Re: How to (de)serialize inherited object?

2018-03-21 Thread mashingan

Try [nesm](https://github.com/xomachine/NESM)

Re: how to read/write object from/to binary file?

Have a look at this:


import streams
type
  MyType* = object
somefield*: array[10, int16]
var t1, t2: MyType
t1.somefield[0] = 1
t2.somefield[0] = 2


var f = newFileStream("/tmp/tmp.mytype", fmWrite)
if not f.isNil:
  f.write t1
  f.write t2
f.flush

var r1, r2: MyType
var f2 = newFileStream("/tmp/tmp.mytype", fmRead)
discard f2.readData(r1.addr, r1.sizeof)
discard f2.readData(r2.addr, r2.sizeof)
echo r1.somefield
echo r2.somefield

Is there any way to create template with await?

2018-03-21 Thread slangmgh

We can create template with like this: 


template withFile(name: string, body: untyped) =
   let fh = open(name, ...)
   defer: close(fh)
   block:
  body


But when template contain await, the compiler will complain the 'await' is 
undeclared identifier. The code like this: 


template withDb(body: untyped) =
   let db = await open(...)
   defer: close(db)
   block:
  body


Because the template is hygiene, and await is not removed by macro async, so 
the compiler will find undeclared 'await', but is there any way to make the 
code above works? Thanks.

Re: how to read/write object from/to binary file?

Sorry, it works correctly. But how to read it back? File contains several 
instances of MyType, so I need to take subpart of file. I've tried this 


let data = readFile("/tmp/tmp.mytype")
let ololo = cast[MyType](data[0 ..

How to (de)serialize inherited object?

Hello guys,

Is there any way to serialize object including its runtime type data? Both 
[marshal](https://nim-lang.org/docs/marshal.html) and 
[msgpack4nim](https://github.com/jangko/msgpack4nim#ref-types) don't support it 
out-of-box.


import streams, msgpack4nim

type
  TA = object of RootObj
  TB = object of TA
f: int

var
  a: ref TA
  b: ref TB

new(b)
a = b

echo stringify(pack(a))
#produces "[ ]" or "{ }"
#not "[ 0 ]" or '{ "f" : 0 }'


I know about object variants but if I use them I will lose the elegance of 
dynamic dispatch and will have to write those ugly


case obj.kind
  of kndA:
# 50 lines of code
  of kndB:
# another 50 loc
  # ... etc


One possible solution is to write a macro that will pack class name together 
with the data, like


genPackers(TA, TB, TC, ...)
# transforms into:
method pack*(a: ref TA): string =
  pack "TA"  # specify runtime type as string
  pack a

method pack*(b: ref TB): string =
  pack "TB"
  pack b

proc unpack*(data: string): ref TA =
  var typ: string = unpack(data)
  case typ
of "TA":
  result = unpack[TA](data)
of "TB":
  result = unpack[TB](data)
# ...



This will probably work, but is there a better solution, without strings of 
class names and lising all possible classes? Kinda


proc pack(anything: ref TA): string =
  pack(anything.runtimeType)  # pack internal type info
  pack[anything.runtimeType](anything)  # pack the object with respect to 
its runtime type

proc unpack(data: string): ref TA =
  let runtimeType = unpack[typeInfo](data)  # unpack runtime type
  result = unpack[runtimeType](data)  # unpack data according to that 
runtime type

Re: how to read/write object from/to binary file?

I do 


t.somefield[0] = 255
t.somefield[1] = 255



to check

how to read/write object from/to binary file?

I can do something like this: 


type
  MyType* = object
somefield*: array[10, int16]
var t: MyType
var f = newFileStream("/tmp/tmp.mytype", fmWrite)
if not fff.isNil:
  f.write t


but result file contains garbage, size of file is 20 bytes.

So question is how to read/write object from/to file?

Re: OrderedTable is not an ordered table

@lightness1024 - fair criticism of C++ STL.

modulo prime reduction of the hash code to a table address/array index is slow 
because division is slow. In my tests using modulo can sometimes **triple** the 
entire lookup time for simple integer keys due to the slowness of division! I 
recommend you consult Knuth 's Art Of Computer Programming Volume 3 (section 
6.4 in my 1998 edition, but his treatment mostly dates back to the 60s). He 
motivates why prime may be a good choice and then immediately moves on to a 
multiplicative hash. If you really need to mix the bits a multiply & shift is 
much faster than modulo. While the upper bits of the half-product do have more 
entropy, masking seems about as good in my experience and the code is slightly 
simpler due to the close relation of the mask and table size.

Knuth goes on at some length about choosing multipliers - about 10X as much 
length as the module prime. For some reason (perhaps an overzealous affection 
for number theory by CS theorists that write algo books or maybe just 
simplicity?) "modulo prime" got repeated more (everywhere - in 
classes/lectures/textbooks/code, etc.). modulo prime sadly "stuck" in 
programming culture needlessly. Python even has some array of primes less than 
powers of two. The pedagogical situation seems to be improving in the past 
decade or so.

In the context of Nim's Table, for integers the absolute most trivial identity 
hash is used. If that is not good enough, a user should override system hash 
functions with their own better scrambled hash code producer. Believe it or not 
(I always recommend testing on your own real data..), the trivial hash is often 
plenty fast. While adequacy of randomization is data dependent, with linear 
probing and modern CPU cache prefetching even large collision clusters can be 
scanned quickly (especially compared to many "improved randomness" mitigations).

Here are two examples of improved randomness failing. You could use a super 
expensive cryptographic hash function to get super random hash code bits, but 
the time you spend doing that would likely be much more than the extra time 
from scanning bigger clusters. Another example is double hashing which hops all 
over the table randomly in a very cache unfriendly way. The clusters are 
smaller, but the number of cache misses goes way up. To avoid dealing with 
hardware variation, people, including smart researchers, often use 
**machine-neutral surrogates** for actual time like  "number of comparisons" or 
"cluster sizes". This can create all sorts of confusion and cross-talk when the 
surrogates are not very faithful to real machines.

Also, this is slightly off-topic for Nim, but I looked at your package and you 
probably have a subtle bug. I don't see you counting "tombstone" or deleted 
markers - usually desirable to reorganize the table if there are too many. See 
[this thread](https://forum.nim-lang.org/t/829) for details. You also might 
like to read the [thread on hash functions](https://forum.nim-lang.org/t/1730).

How to call a macro from a template with a constructed identifier argument?

2018-03-21 Thread rpowers

I have a macro that creates a type and associated functions, and I'm calling 
that from a template. I want to use identifier construction to append a suffix 
to the template parameter, but the whole backticked AST (instead of the 
constructed identifier) gets passed in to the macro. Is there some way to 
generate an identifier before the macro is evaluated? I've included a 
simplified example below.


macro defthing(name: untyped): untyped =
return parseStmt("let $1 = 1000".format($name.ident))

template calldefthing(name: untyped): untyped {.dirty.} =
defthing name # argument is an identifier
defthing `name Extra` # argument is not an identifier, I'd like to 
evaluate the `name Extra` before the macro is called

calldefthing x

Re: OrderedTable is not an ordered table

@cblake - ok perfect. also naming things search-aware is important so I concur.

The python's OrderedDict: I didn't know that, it's a good thing nim is not
alone then. And in fact the presence of this is enough to not go and rename it.
(less disturbance)

In C++ the set/map is usually implemented with a red black tree. It fits a
category which the language lawyers call a "node base container". Sorts are
log(N) and happens at each operation that must trigger a rebalancing of the
tree. (insert/remove)

The undered_set/unordered_map stuff are hash tables, implemented with an
index(hashes)-array of pointers to linked lists. This allows for stable memory
positioning of the values during the needs for rehashings. (decoupling index
from values) Which allows operations to be "non iterator invalidating".

The hashes are not bound to the table's size, which means that they pass
through a modulo operator before being used as indices. The statistics say that
modulo is unfair in most cases, but the most fair for prime numbers. Which
means that the "buckets" (hash table reservation size) are maintained and
resized by jumps to the next reasonable prime number.

A problem of this scheme is that memory gets fragmented horribly, requiring
allocators for serious work. I have written an open address hash map (here
[https://sourceforge.net/projects/cgenericopenaddresshashmap/](https://sourceforge.net/projects/cgenericopenaddresshashmap/))
that allows to keep everything in a flat buffer. This has always better
performance from small types.

The nim implementation appears to require powers of two for the table size,
which means that the modulo must be done by discarding the most significant
bits of the hashes, which loses entropy compared to the prime numbers solution.

Re: OrderedTable is not an ordered table

@lightness1024 - good point. Another argument in favor of "KeyOrderedTable" 
would be that it would probably show up in searches of the more generic 
"OrderedTable". I don't think Nim has a key-ordered collection (EDIT - in the 
stdlib) right now, but Araq has said he has a B-Tree waiting in the wings.

Re: OrderedTable is not an ordered table

Hey that makes sense though. Maybe adding a SortedTable will be enough to solve 
this confusion. The mere presence of an OrderedTable and a SortedTable will 
pick reader's curiosity and will naturally force us to look into why there are 
two concepts that sounds similar.

Re: How to cross-compile a Nim executable for Android

2018-03-21 Thread mashingan

you forgot to link libm, add option `-lm` at very end.

Re: How to cross-compile a Nim executable for Android

2018-03-21 Thread alexsad

Hi

finelly is works!

I tried the way suggested Tetralux via make_standalone_toolchain.py and then 
clang.

Simple echo works, but I tried simple pow from math and clang doesn't compile 
it:


import math

var a = pow(5.09, 2.33)



clang -I ../sysroot/usr/include -I C:/Nim/lib -fPIE -pie -o 
C:/Users/uname/Documents/nimapps/jnijava/hello.bin 
C:/Users/uname/Documents/nimapps/jnijava/nimcache/*.c

...and get error:


\nimcache\hello.c:function NimMainModule: error: undefined reference to 
'pow'
clang50.exe: error: linker command failed with exit code 1 (use -v to see 
invocation)

Re: OrderedTable is not an ordered table

2018-03-21 Thread Araq

An OrderedTable is an ordered table. But it is not a _sorted_ table. IMO, but I 
'm not a native speaker.

OrderedTable is not an ordered table