How to get just the first N bytes with httpclient ?

[void09]

How to get just the first N bytes with http(async?)client for a server that 
does not support range requests ? I did not find any obvious way to do it with 
httpclient methods. I'm thinking something like httpAsyncClient that closes the 
connection when the stream is > N bytes, with 0ms poll, but I do not know how 
to write this.

How to get just the first N bytes with httpclient ?

[void09]

I'd like a confirmation if that's really true or not. If so, my question 
changes to just "How to get the first N bytes.."

How to get just the first N bytes with httpclient ?

[void09]

I also got this solution from someone in the community: 


Not sure which one is "better"

Wave: illegal capture ' ' because 'weaveParallelForSection' has the calling convention:

[void09]

So I have this piece of code that errors with: : illegal capture 'idsBuf' 
because 'weaveParallelForSection' has the calling convention:  
 The same code runs fine if not in a proc. 
Don't know much about weave, read only so much as to put this together. I 
cannot figure out what it wants from me. Please help fix.

Wave: illegal capture ' ' because 'weaveParallelForSection' has the calling convention:

[void09]

Title should be "Weave: ". Reposting for readibility:

So I have this piece of code that errors with:

: illegal capture 'idsBuf' because 'weaveParallelForSection' has the calling 
convention: 

The same code runs fine if not in a proc. Don't know much about weave, read 
only so much as to put this together. I cannot figure out what it wants from 
me. Please help fix. 

Wave: illegal capture ' ' because 'weaveParallelForSection' has the calling convention:

[void09]

Thank you for the comprehensive response ! I had wrongly assumed that the 
explicit capture is not needed because the code worked without it, if put 
outside proc. But since it didn't work with code in the proc, I should have 
thought something needs to be done different.

The workflow here is simple. Retrieve a html page with a certain key passed in 
the URL, parse the html to find a certain string, put the string in an sqlite 
database. The limit here is the connection/http latency. The other parts are 
super fast.

I don't think async can be used here, unless I simulated some kind of thread 
pool to manage the number of parallel jobs, which defeats the purpose I think. 
I also do not know how to work with sync/spawn. This library was recommended 
when I asked in the community chat. I now understand it is overkill for my 
needs, as in, I do not use/need any of its fancy features, but then again, the 
parallelFor was simple enough to use, vs learning the others.

How to draw images on collectionView elements (rects), with nimx ?

[void09]

I need help, issue described here: 
 The documentation is too slim and 
I have no experience with guis at all. Please assist.

Custom header values inside multi-part form ?

[void09]

Trying to implement a few IPFS API calls, specifically this one: 


Basically I need to implement the "nocopy" option of the above linked API call, 
which in curl looks like this: curl 
'' -F 
'file=@SOMEFILE;headers="Abspath: /home/magik6k/full/path/to/SOMEFILE"' (from 
)

I have tried various things, wasted half a day on this, and still no success. I 
am not good with http and its workings. Please provide httpclient code for the 
equivalent curl command. No, I can't use libcurl because it can't be used 
asynchronously in nim, afaik.

thank you

Custom header values inside multi-part form ?

[void09]

A little update after getting the idea that I can easily debug with nc: curl 
"correct" request:


--c6d4d41bf1581405
Content-Disposition: form-data; name="file"; filename="1.txt"
Content-Type: text/plain
Abspath: /home/sgm/1.txt

123

--c6d4d41bf1581405--


Run

nim result, with the code below:


--4292486321577947087
Content-Disposition: form-data; name="file"; filename="1.txt"
Content-Type: text/plain

123

--4292486321577947087
Content-Disposition: form-data; name="headers"

Abspath: /home/sgm/1.txt
--4292486321577947087--

Run

Custom header values inside multi-part form ?

[void09]

Alright so constructing the http body by hand is not possible if you want to 
post a file, cause big files will take some time to load in memory, and eat 
that memory.

I managed to implement a dirty hack in httpclient to insert my headers when 
building it.


MultipartEntry* = object
name, content: string
headers*: seq[tuple[name: string, value: string]]

proc format(entry: MultipartEntry, boundary: string): string =
  result = "--" & boundary & httpNewLine
  result.add("Content-Disposition: form-data; name=\"" & entry.name & "\"")
  if entry.isFile:
result.add("; filename=\"" & entry.filename & "\"" & httpNewLine)
result.add("Content-Type: " & entry.contentType & httpNewLine)
for header in entry.headers:
  result.add(header.name & ": " & header.value)
result.add(httpNewLine)
  else:
for header in entry.headers:
  result.add(header.name & ": " & header.value)
result.add(httpNewLine)
result.add(httpNewLine & httpNewLine & entry.content)

proc add*(p: MultipartData, name, content: string, headers: 
seq[tuple[name:string, value: string]] = @[("","")], filename: string = "",
  contentType: string = "", useStream = true) =
...
  var entry = MultipartEntry(
name: name,
content: content,
headers: headers,

proc addFiles*(p: MultipartData, xs: openArray[tuple[name, file: string]], 
headers: seq[tuple[name:string, value:string]] = @{"":""},
   mimeDb = newMimetypes(), useStream = true):
   MultipartData {.discardable.} =


Run

It would be nice if someone could rewrite it properly and submit the changes to 
nim git. 

Any libraries for dictionary coding (compression) of array structures ?

[void09]

I have started my second project in nim, mostly for learning purposes, and in 
both I have encountered the need to save memory when storing sequential data 
with many repeating symbols.  
(storing unique/deduplicated value in a dictionary + an array with the indexes 
of the values pointing to that dictionary to represent the original sequence) I 
have googled a bit (+ github and nimble dir) and couldn't find anything to help 
with that, maybe I am searching for the wrong words ? Started writing my own 
now, in just 30 lines of code, which works well enough for the purposes I have, 
but I'd rather use something that exists already, better optimized for speed.

Any libraries for dictionary coding (compression) of array structures ?

[void09]

No, it's not at all the easiest way to go at it. I don't see what I can use 
from there, even conceptually. This is the barebones code I wrote for this - 
 It has some shortcomings: The algorithm to 
generate it is probably not optimal, checking if each new entry already exists 
by iterating over the existing data. Hashtable might make more sense in some 
scenarios.

The data structure does not support recursion (to be able to encode seq of 
seq..). Perhaps this can be done with object variant, although it's not 
immediately obvious to me how.

It uses uint32 for index size, instead of bits of appropiate size packed in a 
bit vector. Alternatively, for data with values repeating a lot one after 
another, a start/end range based encoding might be more optimal

Looking for a pair programming partner / coach

[void09]

A coach is exactly what I have been hoping for ever since I restarted my 
programming hobby with nim (previously pascal). Have 3 awesome projects in 
mind, but currently working on a simple cli torrent client (and associated 
lib), which nim ecosystem is missing, and also a good scope for learning: 
sockets, async, network protocol parsing, many algorithms and such. Although I 
am not very skilled, I am rather motivated to learn "proper" programming.

Discord has been of great help, but sometimes I wish I could discuss the 
particulars of my project in greater detail, nim server's attention span is 
rather limited for this purpose. In exchange for some tutoring and spoon 
feeding/code review, nim community will hopefully benefit from (at least) 2 
interesting libraries published to nimble directory. And a happy coder.

You can hit me up on discord (preferably) voidwalker#1983 or matrix 
@voidwalker09:matrix.org or e-mail sgmihai at "don't be evil" company

Is flags={SocketFlag.Peek} buggy or am I missing something ?

[void09]

I have this code with asyncSocket on linux:


while true:
  var msgSize = ""
  while msgSize.len == 0:
  try:
msgSize = await asocket.recv(4, flags={SocketFlag.Peek})
await sleepAsync(200)
  except ...
try:
  resp = await asocket.recv(4 + msg2Int(msgSize))
(proces message...)

Run

This works as expected, until I get a larger message (16kB, confirmed with 
wireshark that I actually receive it). What happens is that it gets stuck on 
recv the first 4 bytes in peek mode, it stays at 0, even though I have not 
gotten the data out of the socket buffer. If I remove the Peek flag, and adjust 
the code (remove +4 from the second recv size), I get the expected data.

 :


MSG_PEEK
This flag causes the receive operation to return data from the beginning of 
the receive queue without removing that data from the queue. Thus, a subsequent 
receive call will return the same data.

Any idea what could be going on ?

What would be the bast way to get parsing and validation of parameters (option:value)

[void09]

Need to parse and validate user supplied parameters for video encoders, in the 
form of "\--option1:value --option2 --option3:value", which are unique for each 
encoder we have defined. option fields must be in a set of predefined such 
fields, while values have to be in a certain range for integers, have to be of 
certain specific values for strings, or maybe be a valid file path.

 only does the parsing which is not 
that hard, and not that useful to me without validation, as I'd need to extract 
the data from its object anyway after parsing. The other command line libs in 
nimble that I looked at all have parsing, but no validation options.

 \- this is kind of nice, but 
only does validation on object fields, and I am not sure I want an object with 
tens or 100 fields. I would have to look into how to handle iterating over 
fields. Worth keeping in mind.

 with the help of sealmove, I do have a 
working implementation using this. Downside is, binarylang was used for binary 
streams, not string parsing, author believes it's not a proper context to 
utilize it in.

I am curious how would you implement this. Without using 100 line long case 
statements, for each option.

Bonus features would be the ability to "inherit" these values:constraints from 
an encode type and extend them, like with classes. So I don't have to repeat 
100 lines for each slightly different version or a fork that changed an option 
and so on.

A nice thing to have would be runtime loading of these values, so they don't 
need to be known at compile time. but not strictly necessary.

What would be the bast way to get parsing and validation of parameters (option:value)

[void09]

Hm no, in this case I want to simply ensure the parameters that I get, from any 
external source, are valid in their context. They won't be used as parameters 
in a nim program, but to eventually execute an external binary. So cligen is 
not of any help.

Araq's way, "default design procedural programming", is not what I had in mind. 
I believe that if nim has the features needed to abstract these concepts to a 
more declarative syntax, then it should be used. Much more readable and sane to 
reason about. After all, this is a rather general use case I can see being 
needed (to validate cli params of a nim program, or external program you want 
to launch), I am surprised I couldn't find something readily available for the 
task.

Still, expanding on Aarq's suggestion, using templates and macros, I believe a 
small DSL could be built to define such constructs. Something like:


enumType fields:
  "param1" {validExpression} defaultValue
  "param2" {validExpression} defaultValue


Run

and some associated procs that would generate a Table of the key value pairs 
(if valid) from the input string, and maybe the associated enum types for them, 
if needed/designed like that. Could also have the "inherit" keyword, so as to 
copy the list of statement from another structure previously defined, and 
expand/modify it.

Optimise async request / code [For Beginners]

[void09]

As far as number of lines go, you can combine the creation of http client, 
request, and checking for content in one line. Also, although you made an async 
proc, you use waitFor in a for loop, for each future, which will make the 
program behave just like doing non async http requests, it will make each http 
request sequentially, only start a new one after the future/proc is complete. 
To avoid this you can replace the for loop with "waitFor all futures" 


So the code would look like:


import std/[httpclient, asyncdispatch, strutils]

const urls = @[
  "https://www.google.com";,
  "https://www.yahoo.com";]

proc getContent(url: string): Future[void] {.async.} =
  echo url & " " & $(await 
newAsyncHttpClient().getContent(url)).contains("tag")

var futures : seq[Future[void]]

for url in urls: futures.add(getContent(url))

waitFor all futures

Run

Question from a complete Newbie

[void09]

I tried your code and it works with both H and HH, and shows the correct 
result, according to the format:  
 Don't see a problem, check your code again.

Help with generics and typeclass problem

[void09]

type
  Units* = SomeUnsignedInt
  BitVecS*[S:static int, B:Units = uint] = object
Base*: array[S div (sizeof(B) * 8) + int(S mod (sizeof(B) * 8) != 0), 
B] #8,16,32,64
  BitVec*[B:Units] = object
Base: seq[B]
size: int
  AnyBitVec[S:static int, B:Units] = (BitVecS[S, B] | BitVec[B])

proc `[]`*[S,B](bv:AnyBitVec[S, B], i:int): int =
  return (bv.Base[i div (B.sizeof * 8)] shr (i and (B.sizeof * 8 - 1)) and 
1).int

func newBitVector*[B](size: int, init = 0): BitVec[B] {.inline.} =
  var blocks = size div (sizeof(B) * 8) + int(size mod (sizeof(B) * 8) != 0)
  result.Base = newSeqOfCap[B](blocks)
  result.Base.setlen(blocks)
  result.size = size

var bv1: BitvecS[20240001]

var y = newBitVector[uint32](1000)
discard y[0] #this fails

Run

I am trying to write a bitvector lib that has a common implementation for 
static (array) and dynamic (seq) ones, with user defineable base unit of 
storage. For static one I need S (static size) and B (base unit), while for the 
dynamic one I only need B. I tried like in the code above, but when accessing 
any procs on the dynamic one (BitVec), I get "Error: cannot instantiate: 'S'". 
I thought this would work since BitVec doesn't have/need S, but no. I got 
around it by adding the S parameter to the type, and passing a random number as 
generic parameter when calling procs on it, and while this works, it's 
obviously not pretty, and makes me believe there has to be a way to make this 
work cleanly. I was told to use concepts, but I couldn't make that work either, 
and I really don't understand how they work even after reading the doc. But I'd 
like not to use them unless absolutely necessary. Also, on a side note, the 
default generic parameter type I have on BitVecS (B:Units = uint) seems to 
actually work, although I was told it shouldn't. Any idea why it works (in this 
context)?

Help with generics and typeclass problem

[void09]

mratsim that might be true in general, but I want to have user-pickable uint 
base type for the array/seq, so openArray[byte] won't do. Also I want to have 
max performance, and one extra proc call is not ideal.

Help with generics and typeclass problem

[void09]

So I don't open another topic, I'd like to know if it's possible to use 
std/math ceilDiv inside a (generic) type definition, like here: 
 this does not compile, even though const a 
= ceildiv(..) works, so ceildiv is good for compile time. If I write my own 
ceildiv without generics and use that, it works

Help with generics and typeclass problem

[void09]

You are right mratsim, it's the nim compiler that sigsegv.. which is obviously 
a bug, no doubt about it :) I will report it.

Speaking of bugs and types, I almost forgot this little code snippet when I was 
trying to figure out concepts: 

Code behaves correctly but for some reason the `[]` proc is very slow. I get 
~45 second runtime vs ~4ms what it normally should be, for 4 million 
operations. 10.000x slower. While the []= proc is as fast as it's supposed to 
be.

I lack the skills to figure out the root cause of this, maybe you or someone 
else with some time can try and see if this is a nim bug, or there is something 
wrong with my code in this context. But it looks like a bug. Using nim-devel.

Need suggestions for time scheduling async tasks

[void09]

I am writing a client/crawler/scraper library for a certain website, hopefully 
my first nimble package. The site has some rate limiting which I haven't figure 
out exactly yet, but will try to do. Once I do, I want to have a request 
scheduler so that all requests (async) done to the website are automatically 
delayed by the right amount of time once I reach the rate limit threshold value 
(eg, 1000 requests per hour, 30 requests per minute etc), so that it doesn't 
trip the rate limiting. I am thinking a template to wrap the procedures call 
bodies in and send it to the schedueler would be a proper interface. Or maybe I 
should just compute the required sleep time value (if needed) at each call, and 
make a proc that wraps the call to the proc, adding a sleepAsync call before it 
? I feel this is a rather common use case, and we should have a library for 
this. I could only find this after 5 minutes of searching: 
 \- but I can't say I understand how it 
works/how to use it after reading the readme.

Need suggestions for time scheduling async tasks

[void09]

Thank you for the suggestion, unfortunately the way it works is not clear to 
me, and how I should implement it for my use case. I realized the actual 
requests are done by a single procedure in my code, and I can calculate a sleep 
delay at the start of it, given the times previous requests were made and the 
ratelimit constraints. It will do. Although a more generic queue-like (outside 
of asyncdispatch) method would have been preferable.

nim merch

[void09]

I agree that we should have more or less nerdy nim t-shirts, caps etc. 
officially. So people of the nim almost secret by obscurity society can 
recognize each other. Would certainly be more useful stuff to buy than Araq's 
book.

Wishlist: Ideal UI library for Nim

[void09]

I believe the right approach for this is something like Lazarus' LCL, having a 
unified interface, with different platform-specific implementations of various 
wideget sets. Here is what they have working so far: 
 I have 
found a nim project that managed to use it somehow (no idea about the details, 
seems to have used some automated binding tool): 
 A native port is probably difficult, since 
it relies so much on pascal classes, which do not translate well to c++ 
classes. And nim doesn't even have classes.

But I would very much like to see something like this, having one code base 
that can support native gui on all platforms, and able to select gtk/qt, etc at 
compiletime, if one works better than the other on a certain platform. It's 
probably what keeps the freepascal language alive these days. I had a very easy 
time writing gui programs in Lazarus, as a total noob.

High level TUI framework (or wrapper of)

[void09]

Already posted here 
(), but trying the 
forum as well for more visibility. I've been wanting to make a tui app for some 
time, planned on using nimwave, but on closer look I was surprised to see that 
there's basically no widgets implemented in it, and you have to do things more 
or less by hand (I need a table, drop down, text input, and more).

Rust has ratatui, go has bubbletea and a ton of others, same with python, and 
it seems nim has no high level tui building framework. Also nimwave doesn't 
seem to be used in any projects except the author's one. I saw nimbox (termbox 
wrapper), which maybe might suffice, but I am having trouble installing it for 
archlinux, and termbox2 seems to be the one to use now, which is not wrapped. 
There's also nim-notcurses, but notcurses only has some basic widgets. Ncurses 
does have widgets but they're in a separate library which is not wrapped.

Because of this I am contemplating learning another language for this project, 
go or rust. Since it would seem easier than writing your own TUI framework in 
nim. I found these two very nice projects written in C++: 
  
Does anyone have the ability to make a nim wrapper for any of them ?

It seems to me that this is quite an essential library to have, big negative 
point for this lack. Interesting that there's so many gui libraries attempts, 
yet nothing serious for tui. 

High level TUI framework (or wrapper of)

[void09]

I was under the impression that termbox/2 has some simple widgets, like a 
scrollable table, seeing  made with it. But 
it seems it draws everything manually, and for that I could use the already 
existing nim lib illwill. So I can scratch the temrboxes from the list of 
candidates, my project idea is a bit too complex to be drawing everything by 
hand. And also counter productive to reinvent the wheel, as a principle.

So I'd like to know if some c++ people can maybe wrap one of those projects 
mentioned above, or maybe we should find a good candidate to be ported to nim, 
from a similar language that doesn't make heavy use of classes ? Like rust or 
go. 

High level TUI framework (or wrapper of)

[void09]

Thank you enthus1ast, somehow I missed that project. I believe it is enough to 
get me started on it in nim. Although everything is rather primitive, no auto 
sizing based on terminal resize, no scrollable list for table. Do you think you 
have time in the near future to make the table scrollable, or should I look 
into it ?

High level TUI framework (or wrapper of)

[void09]

@oakes It would be great if you could provide a one widget example/template. 
Not sure how it would differ from illwillwidgets mentioned above.

High level TUI framework (or wrapper of)

[void09]

Well, I decided I will learn rust and attempt my project there. Nim ecosystem 
(libs and potential manpower, as well as velocity of growth) too sparse for the 
job unfortunately. And also because my poor coding speed and knowledge can't 
make up for the lacking ecosystem. Will still use Nim for simpler stuff that is 
doable in it. I'm thinking to write a post about my experience with Nim, all 
the goods and the bads.

Newbie Here! Can I have some feedback on my code?

[void09]

Since the strings use are constant, better to define them before the code, it's 
cleaner imho.

You can use the contains proc with sets of letters, like this, instead of your 
loops:
let shouting = phrase.contains({'A'..'Z'}) and not 
phrase.contains({'a'..'z'}

This part is redundant " elif contains(PossibleQuestion,"?"):" since all you 
care is if the question mark is at the end, so you can just use the endsWith ? 
You can ommit the type declaration here, it gets inferred from the value "var 
qualifier = true" You don't need to specify range to iterate all chars in a 
string, just for i in stringVar. This was the best I could do, solved it 
previously:  
Compare with my solution and see if your logic can be shortened, your code is 
way too convoluted. 

Enabling compression on socket send/receive.

[void09]

What would be the most simple/elegant way to have (de)compression enabled on 
all data sent/received over a socket ? In go it appears that it's as simple as: 
.. code-block:: go

> conn, err := net.Dial("tcp", "localhost:8080")
> 
> `if err != nil ...`
> 
> gzipWriter := gzip.NewWriter(conn) message := "Hello, this is compressed 
> data!" _, err = gzipWriter.Write([]byte(message))

similary in rust: .. code-block:: rust

> let stream = TcpStream::connect("127.0.0.1:8080")?; let mut encoder = 
> GzEncoder::new(stream, Compression::default());
> 
> let message = "Hello, this is compressed data!"; 
> encoder.write_all(message.as_bytes())?;

Well, these examples use a separate construct/writer for encapsulating the 
compression operations, but perhaps there's ways around to make it transparent, 
so that you'd use the standard syntax to write to sockets and get compressed 
data.

Could we have something like this in nim as well, given that it doesn't seem to 
support interfaces ? Inherit from Socket type maybe and add custom procs that 
wrap around the existing send/recv ? Use socketstreams instead of socket to 
write, but nim-lang/zip module only seems to expose a filestream ? zippy also 
doesn't support streams yet.

Enabling compression on socket send/receive.

[void09]

already saw that example that is not relevant to the topic. if compression then 
send compress(string) else send string. I was looking for a more generic 
socket-centric solution.

Enabling compression on socket send/receive.

[void09]

I tried to get a proc as field in my client object, so i can swap it at runtime 
without having if else everywhere.

I have these definitions:


proc sendProcU(nntp: NNTP | AsyncNNTP, data: string) {.multisync.} =
  await nntp.sock.send(data)
  
  NNTPClientBase*[SocketType] = ref object
...
sendProc*: proc(nntp: NNTP | AsyncNNTP, data: string)
  
  NNTP*  = NNTPClientBase[Socket]
  AsyncNNTP* = NNTPClientBase[AsyncSocket]

proc new(T: typedesc[NNTP | AsyncNNTP], host: string, port: Port, tlsPort = 
port, user, pass: string, tls: bool): T =
  result = T(host: host, port: port, tlsPort: tlsPort, user: user, pass: 
pass)
  when T is NNTP:
result.sock = newSocket()
result.sendProc = (proc (nntp: NNTP, data: string))sendProcU
  else:
result.sock = newAsyncSocket()
result.sendProc = (proc (nntp: AsyncNNTP, data: string))sendProcU


Run

This results in the error: ` Error: generic instantiation too nested` at the 
last line where the sendProc field is assigned. Any idea what I am doing wrong 
and how to fix it ?

Enabling compression on socket send/receive.

[void09]

bump. still no solution found yet

Enabling compression on socket send/receive.

[void09]

Thank you, I guess that is an acceptable workaround while keeping the same 
syntax of nntp.sendProc(), but the reason I wanted to store the procedure as a 
field in the object, is to avoid the if compress: sendCompressed else: 
sendUncompressed. Would avoid extra code and runtime overhead for all those 
ifs, at every command sent. And same with receive. So I'd like someone to help 
me, maybe I messed up the syntax somewhere, maybe the use of multisync + 
generic requires workaround etc.

Enabling compression on socket send/receive.

[void09]

I am trying to write a NNTP client for nim, for some reason this basic ancient 
protocol is not implemented in any lib I could find. NNTP protocol allows for 
gzip or deflate compression. I know an if boolean for every send is not a big 
deal, but if it can be done better, by swapping assigned procs in the object, 
why not. It's also much cleaner and logical code. So please assist with this. 
Problem might come from bad syntax, or some complication from generics or 
multisync.

Enabling compression on socket send/receive.

[void09]

not sure what you mean there Araq. That is what I want to accomplish here, 
write 3 procs, sendUncompressed, sendGzip, sendDeflate, and when compression is 
enabled, just do client.sendProc = sendGzip or whatever. And all socket sends 
are handled by client.sendProc(data: string). I am stuck at this for 9 days. 
Help please?

Enabling compression on socket send/receive.

[void09]

Thank you Clonk, but like I said, I wanted to avoid the "ifs", and case is just 
an alternative if. Already know to implement it like this. My question was how 
I can implement it with storing the procs instead, like I said 3 times already 
:)