Re: [Chicken-users] u8vector to numbers bignum
On Thu, May 28, 2015 at 08:18:14PM -0700, chi wrote: Also, a number-u8vector function would be nice. Converting a number to a hex string, then taking every 2 characters of that and converting that back to a number, for each element of the u8vector, just to keep me from accessing the number's bytes directly, just strikes me as terribly roundabout. (define (number-u8vector num) (let ((s (number-string num #x10))) (let ((v (make-u8vector (/ (string-length s) 2 (let loop ((s s) (i 0)) (when ( i (u8vector-length v)) (u8vector-set! v i (string-number (substring s 0 2) #x10)) (loop (substring s 2) (+ i 1 v))) vs (define (numbers-u8vector num) (blob-u8vector (somehow-cast-to-blob (subvector (somehow-cast-to-vector num) 1 Here you go. It's nasty, low-level and prone to break if ever the representation is going to change (which it did a few times, so it only works with the latest release of the numbers egg). (define (bignum-u8vector/host-byte-order num) (unless (bignum? num) (error Argument is not a bignum! num)) (let* ((word-size (cond-expand (64bit 8) (else 4))) ; Is there a better way? (bignum-contents (##sys#slot num 1)) ; Digits preceded by sign word (bignum-digits (substring bignum-contents word-size))) (blob-u8vector/shared (string-blob bignum-digits The result data will be in little endian order of the words (least significant bignum digit first), but the bytes within the word will be in the host endian order. This means the order is the REVERSE of the slow procedure you quoted above. Plus, it will include spurious zero octets (because it serializes whole machine words): #;6 (bignum-u8vector/host-byte-order #xffaa) #u8(170 0 0 0 0 0 0 0 0 255 0 0 0 0 0 0) Cheers, Peter signature.asc Description: Digital signature ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
chi scripsit: Ironically, that is exactly what the SRFI-27 module implements. http://wiki.call-cc.org/eggref/4/srfi-27#random-sources Could make an insecure random source, or even a totally non-random random source I imagine. I had no idea it was so comprehensively extended. -- John Cowan http://www.ccil.org/~cowanco...@ccil.org If you have ever wondered if you are in hell, it has been said, then you are on a well-traveled road of spiritual inquiry. If you are absolutely sure you are in hell, however, then you must be on the Cross Bronx Expressway. --Alan Feuer, New York Times, 2002-09-20 ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
On Thu, May 28, 2015 at 9:09 PM, John Cowan co...@mercury.ccil.org wrote: Peter Bex scripsit: If this is such an important feature it may make more sense to include a proper PRNG. Different applications will want fast crude random-ish numbers, PRNGs, cryptographic PRNGs, or full quantum randomness, with tradeoffs for speed and quality. Since that's so, I'd rather require programmers to make the choice up front rather than fall back on some dubious OS version that does who-knows-what. Not all applications require cryptographically secure real random number generators. In fact, I'd argue that most random usage is not for that sort of thing. It will be used in tutorials, or in games for a purpose such as selecting between several available sounds. Or procedural generation if it doesn't totally suck. If you *are* writing something that really requires a better quality pseudorandom generator, odds are that you will have to import one anyway. I have no idea of what rand() uses internally, and much less what is Chicken is actually calling on Win32. So one thing that could be of value is not relying on the OS implementation and instead providing our own (Mersenne Twister?). This would remove a core dependency and guarantee consistency when running on all operating systems. Even if it is consistently crappy(which MT doesn't appear to be, even if it should not be used for crypto). That would be preferable, so you'd deal with it in the beginning, instead of when porting to another OS, when you've already forgotten all about the random generator. For reference: (random N) http://api.call-cc.org/doc/extras#def:random procedure Returns a pseudo-random integer in [0, N-1]. N is an integer. On Windows, N and the random value are exact integer. *Warning*: This procedure uses *rand(3)* internally and exhibits its deficiencies, including low quality pseudo-randomness: - On Windows and Solaris, only 32768 unique random values can be generated in the range [0, N-1]. If N = 32768, there will be gaps in the result set. - On Mac OS X, Windows and some other platforms, little variance in output is seen with nearby seeds. Since the random generator is seeded with current-seconds at startup, new processes may see similar or identical random sequences for up to a minute. - On Linux, *rand(3)* is an alias to *random(3)*, which provides output of reasonable quality. — Stephen ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
(define (bignum-u8vector/host-byte-order num) (unless (bignum? num) (error Argument is not a bignum! num)) (let* ((word-size (cond-expand (64bit 8) (else 4))) ; Is there a better way? (bignum-contents (##sys#slot num 1)) ; Digits preceded by sign word (bignum-digits (substring bignum-contents word-size))) (blob-u8vector/shared (string-blob bignum-digits That's pretty fascinating. I wonder if you could even just C_copy_memory the bignum-contents+word into an existing blob of large enough size, instead of that substring/string-blob. Assuming you know your number is of fixed but 64 bit size, that is. (1024 bit for instance). If you didn't want native order, instead of C_copy_memory you could do a word based converter to network byte order called I dunno, C_copy_flipwords. Incidentally, chicken.h defines C_WORD_SIZE, and I think it's a fairly safe assumption even though the core never explicitly exports that value into scheme. You could probably do this pretty confidently: (define word-octets (/ (foreign-value C_WORD_SIZE integer) 8) ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
On Thu, May 28, 2015 at 8:59 AM, Peter Bex pe...@more-magic.net wrote: Yeah, the numbers random implementation is shitty, which is why I decided to omit it from my port to CHICKEN core (CHICKEN 5's random is still fixnum only). Personally, I think it would be better if we simply got rid of all random support in core, because there's no way this is ever going to satisfy everyone. Besides, the standard C library functions are deeply flawed, especially on some platforms like OS X. I think that if you keep removing features from core, we will soon run out of them. When I use “stock random functions in any language I don’t have high expectations, but for sure, I always expect a random function to be readily available. Regards, Michele ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
On Thu, May 28, 2015 at 10:34:55PM +0200, Michele La Monaca wrote: On Thu, May 28, 2015 at 8:59 AM, Peter Bex pe...@more-magic.net wrote: Yeah, the numbers random implementation is shitty, which is why I decided to omit it from my port to CHICKEN core (CHICKEN 5's random is still fixnum only). Personally, I think it would be better if we simply got rid of all random support in core, because there's no way this is ever going to satisfy everyone. Besides, the standard C library functions are deeply flawed, especially on some platforms like OS X. I think that if you keep removing features from core, we will soon run out of them. That's silly rethoric, considering we are on the verge of merging in a huge new feature (full numeric tower) and have a lot more new things planned in the roadmap: http://wiki.call-cc.org/chicken-5-roadmap We want a well-defined, coherent core rather than a core with features from all over the place in various degrees of disrepair. When I use “stock random functions in any language I don’t have high expectations, but for sure, I always expect a random function to be readily available. If this is such an important feature it may make more sense to include a proper PRNG. Anyway, this is just my opinion, I don't expect the entire core team to agree with me. Cheers, Peter signature.asc Description: Digital signature ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
On Thu, May 28, 2015 at 11:23 PM, Peter Bex pe...@more-magic.net wrote: On Thu, May 28, 2015 at 10:34:55PM +0200, Michele La Monaca wrote: On Thu, May 28, 2015 at 8:59 AM, Peter Bex pe...@more-magic.net wrote: Yeah, the numbers random implementation is shitty, which is why I decided to omit it from my port to CHICKEN core (CHICKEN 5's random is still fixnum only). Personally, I think it would be better if we simply got rid of all random support in core, because there's no way this is ever going to satisfy everyone. Besides, the standard C library functions are deeply flawed, especially on some platforms like OS X. I think that if you keep removing features from core, we will soon run out of them. That's silly rethoric, considering we are on the verge of merging in a huge new feature (full numeric tower) and have a lot more new things planned in the roadmap: http://wiki.call-cc.org/chicken-5-roadmap Adding new (great!) features it's not a good reason to remove useful one. Of course, my sentence was an exaggeration to stress the point and give you heads-up. Regards, Michele ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
On May 27, 2015, at 3:53 PM, chi chic...@verge.info.tm wrote: How would I convert a u8vector to a bignum? I'd like a good large random number, and there's srfi 27 for decent random sources, and there's 'numbers' for bignum support, but srfi-27 only produces fixnums or u8vectors. The integer result range extends to that of the limit parameter. Ex: #;1 (use numbers srfi-27) ; loading /usr/local/lib/chicken/7/numbers.import.so ... ; loading /usr/local/lib/chicken/7/srfi-27.import.so ... ; loading /usr/local/lib/chicken/7/srfi-4.import.so … ... ; loading /usr/local/lib/chicken/7/srfi-4-checks.so ... ; loading /usr/local/lib/chicken/7/srfi-4-errors.so .. #;2 (random-integer (expt 2 64)) 15816285179193592418 #;3 (bignum? 15816285179193592418) #t Logically I can't imagine a bignum isn't represented under the hood by a block of bytes somehow, so it should be easy to turn a random byte vector into a random bignum? numbers itself only provides (random ...) which uses rand(3), and that's not what I would call a good random number. Random sources don't really seem like something a bignum library should worry about, anyway. I could always do (number-string (u8vec_to_hex (random-u8vector #x20)) #x10) but having a double sized hexadecimal intermediate isn't terribly appealing, especially for a random number I might have to generate for hundreds of peers a second in a high volume scenario. (Okay now I really /am/ being optimistic.) Looking at the numbers source, it seems to be analagous to a vector whose first element is a magic number indicating it's a bignum, and the rest of the vector is the digits. I'm not qualified to say how hard that would be to turn a vector into that, but it really doesn't seem like it would be too much trouble for someone who was. Of course it would be seen as native endian, but since it's a random number it doesn't matter too much. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
On Thu, May 28, 2015 at 3:59 PM, Peter Bex pe...@more-magic.net wrote: Yeah, the numbers random implementation is shitty, which is why I decided to omit it from my port to CHICKEN core (CHICKEN 5's random is still fixnum only). This is surprisingly hard to do well - the Chibi SRFI-27 implementation had a high bug frequency, and I still haven't done a proper analysis of the distribution for different ranges. -- Alex ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
On 05/28/2015 05:09 PM, John Cowan wrote: Different applications will want fast crude random-ish numbers, PRNGs, cryptographic PRNGs, or full quantum randomness, with tradeoffs for speed and quality. Ironically, that is exactly what the SRFI-27 module implements. http://wiki.call-cc.org/eggref/4/srfi-27#random-sources Could make an insecure random source, or even a totally non-random random source I imagine. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
On 05/28/2015 12:55 PM, Kon Lovett wrote: The integer result range extends to that of the limit parameter. Ex: Yeah sorry, I didn't see that the first time, my bad. (random-integer large-bignum) would produce a large bignum just fine. Problem solved! But did you take a look at the (random-large-integer) module in -numbers? What they do is multiply a bignum by a bignum of its digit size (max of a fixnum), then add random to the new empty digit, and then iterate. I don't want to prematurely optimize or anything, but that's a REALLY slow way to build up a sequence of digits in a bignum. Both the guarantee of the multiple being digit-aligned, and the knowledge of how many digits beforehand, are completely lost with that method, which has to assume the multiple could be anything, and can't pre-allocate anything. A u8vector-number or perhaps set-bignum-digit! function could build the resulting bignum much faster, if the numbers module supported anything like that. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
Also, a number-u8vector function would be nice. Converting a number to a hex string, then taking every 2 characters of that and converting that back to a number, for each element of the u8vector, just to keep me from accessing the number's bytes directly, just strikes me as terribly roundabout. (define (number-u8vector num) (let ((s (number-string num #x10))) (let ((v (make-u8vector (/ (string-length s) 2 (let loop ((s s) (i 0)) (when ( i (u8vector-length v)) (u8vector-set! v i (string-number (substring s 0 2) #x10)) (loop (substring s 2) (+ i 1 v))) vs (define (numbers-u8vector num) (blob-u8vector (somehow-cast-to-blob (subvector (somehow-cast-to-vector num) 1 ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
Peter Bex scripsit: If this is such an important feature it may make more sense to include a proper PRNG. Different applications will want fast crude random-ish numbers, PRNGs, cryptographic PRNGs, or full quantum randomness, with tradeoffs for speed and quality. Since that's so, I'd rather require programmers to make the choice up front rather than fall back on some dubious OS version that does who-knows-what. -- John Cowan http://www.ccil.org/~cowanco...@ccil.org If you understand, things are just as they are. if you do not understand, things are just as they are. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
On Wed, May 27, 2015 at 03:53:39PM -0700, chi wrote: How would I convert a u8vector to a bignum? I'd like a good large random number, and there's srfi 27 for decent random sources, and there's 'numbers' for bignum support, but srfi-27 only produces fixnums or u8vectors. I don't know anything about the srfi-27 egg, but after a quick look, there seems to be an undocumented srfi-27-numbers module. Since srfi-27 depends on the numbers egg, this should probably do the trick. Logically I can't imagine a bignum isn't represented under the hood by a block of bytes somehow, so it should be easy to turn a random byte vector into a random bignum? numbers itself only provides (random ...) which uses rand(3), and that's not what I would call a good random number. Random sources don't really seem like something a bignum library should worry about, anyway. Yeah, the numbers random implementation is shitty, which is why I decided to omit it from my port to CHICKEN core (CHICKEN 5's random is still fixnum only). Personally, I think it would be better if we simply got rid of all random support in core, because there's no way this is ever going to satisfy everyone. Besides, the standard C library functions are deeply flawed, especially on some platforms like OS X. I could always do (number-string (u8vec_to_hex (random-u8vector #x20)) #x10) but having a double sized hexadecimal intermediate isn't terribly appealing, especially for a random number I might have to generate for hundreds of peers a second in a high volume scenario. (Okay now I really /am/ being optimistic.) Looking at the numbers source, it seems to be analagous to a vector whose first element is a magic number indicating it's a bignum, and the rest of the vector is the digits. That's correct: it's a record object which has 'bignum as its type tag and is followed by a string which contains the binary data. The first C_word of that string is actually 1 or 0 indicating its sign (1 if negative). Cheers, Peter signature.asc Description: Digital signature ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] u8vector to numbers bignum
Personally, I think it would be better if we simply got rid of all random support in core, because there's no way this is ever going to satisfy everyone. Besides, the standard C library functions are deeply flawed, especially on some platforms like OS X. Emphatic +1. -- John Cowan http://www.ccil.org/~cowanco...@ccil.org My corporate data's a mess! It's all semi-structured, no less. But I'll be carefree / Using XSLT On an XML DBMS. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
