writeFile with iterator

[Garry_Galler]

How can I make this code work? 


proc enumerableRepeat[T](value: T, n: int = -1): iterator(): T =
  result = iterator(): T {.closure.} =
var i = 0
while n == -1 or i < n:
  yield value
  i += 1


proc writeFile[T](filePath,iter:iterator():T) =
  var file = open(filePath,mode=fmWrite)
  for str in iter: file.writeLine(str)
  file.close()

writeFile("iter.txt",enumerableRepeat("a",100))

Re: writeFile with iterator

[Garry_Galler]

**OderWat**,Thank you for editing the code :)

And it should be a little slower 


import times
import threadpool
{.experimental.}

var w = ["aa", "bb","cc"]

var startTime = epochTime()
parallel:
  for i,str in w:
spawn writeFile("dict_" & $i & ".txt",enumerableRepeat(str, 3_000_000))

echo "Time: ",epochTime() - startTime," seconds"
#Time: 2.846011525369249 seconds


than this? 


proc writeFile2(filePath,text:string) =
  var file = open(filePath,mode=fmWrite)
  for str in 1..3_000_000: file.writeLine(text)
  file.close()

var w = ["aa", "bb","cc"]

var startTime = epochTime()
parallel:
  for i,str in w:
spawn writeFile2("dict_" & $i & ".txt",str)

echo "Time: ",epochTime() - startTime," seconds"
#Time: 2.264008925878443 seconds


PS: I wanted to do something like how in C # 


Parallel.For(0, 3,  i=> {File.WriteAllLines(@"dict" + i + 
".txt",Enumerable.Repeat(w[i],300));});

But there is a running total 00:00:01.1200015

Re: writeFile with iterator

[Garry_Galler]

**cblake**

I did not know that --opt: size so affects the speed. Now I will take this into 
account. Thank you. Replacing writeLine to write really accelerated the code is 
almost 1.5 times.

PS: Nim really fast. You just have to know how to use :) 

Re: writeFile with iterator

[OderWat]

Modifying where the line ending is added can be even more optimized when you 
change the parameter to the iterator such that the string is only changed once. 
E.G.: `enumerableRepeat(str & "\n", 3_000_000)`

This would not even be faster if that was done more efficient than introducing 
a string copy.

That using WriteLine() is slower is a pity though. Here we see how simple 
things can change the outcome a lot. I wildly guess it calls "write()" twice, 
which is causing more overhead than the string implementation of Nim which 
copies the string to add the line ending.

Re: writeFile with iterator

[cblake]

Also, in terms of optimization just replacing: 


for str in iter(): file.writeLine(str)


with 


for str in iter(): file.write(str & "\n")


yielded a 1.3X speed-up (I.e., it took the time down to 0.14 seconds).

Re: writeFile with iterator

[OderWat]

Try this fixed version:


proc enumerableRepeat[T](value: T, n: int = -1): iterator(): T =
  result = iterator(): T {.closure.} =
var i = 0
while n == -1 or i < n:
  yield value
  i += 1


proc writeFile[T](filePath: string, iter: iterator():T) =
  var file = open(filePath, mode=fmWrite)
  for str in iter(): file.writeLine(str)
  file.close()

writeFile("iter.txt", enumerableRepeat("a", 100))


  1. You accidentally made `filePath` an iterator too.
  2. You need the `()` when you call the iterator.

Re: writeFile with iterator

[cblake]

Garry: --opt:size (and things like gcc -Os) instructs compilers to choose 
trade-offs to make object code smaller rather than faster. E.g., on Oderwat's 
code, I get 0.19 seconds with just -d:release (with gcc-6.1 on Linux x86_64, 
i7-6700k @4.5GHz). If I additionally specify --opt:size the time rises to 0.28 
seconds, almost 1.5X slower. The program text segment is 80KB in the fast case 
but 33KB in the slow case. So, all is behaving as expected/requested.

Re: writeFile with iterator

[OderWat]

So why not using C# then? (evil smile)

Compiling with "-d:release" gives me "Time: 0.6296679973602295 seconds" 
(against about 2 seconds without it)

Re: writeFile with iterator

[cblake]

Yeah...I thought about \n higher up but didn't want to change the semantics too 
much. With that change my time goes down to 0.10 seconds (almost a full 3X 
faster than the original 0.28 --opt:size version on the same machine). About 
55..60% of that 0.10 seconds is all just libc fwrite stuff. The remaining Nim 
probably can't be sped up much more..maybe 10% total runtime at most. 
file.write is just a very thin wrapper for stdC fwrite. You can see if you look 
at lib/system/sysio.nim (search for proc.*write.*File).

Re: writeFile with iterator

[Garry_Galler]

> So why not using C# then? (evil smile)

C # is bound to NET platform. And this is his main, but in many cases - a 
significant minus. Nim - self-sufficient, self-contained - and that I like.

> That is 0.77 using this exact code

I also compiled with the -d: release and --opt: size. Apparently, you have a 
much faster computer. I have a 4-core, but rather the old generation.

Ok. Let us assume that the code as fast as in C#. In addition, C # uses to 
restart the cached version of the program. I have run-time C # code is 
restarted. The first time it runs about 4 seconds.