Re: [PD] Redis sync, async and subscriber client: Puredis
Ok, I came up with a generic csv loader for Redis datatypes. it is plugged into the [puredis] object with the csv method, it receives a filepath and a datatype. Making it possible to build Strings, Lists, Hashes, Sets ans SortedSets from csv files. I updates the pd-help files to match a well as the docs (the Readme). https://github.com/lp/puredis#readme 2011/7/26 Hans-Christoph Steiner h...@at.or.at Where I could see it used it with large data sets like I did in this project: http://at.or.at/hans/**terrenatale/ http://at.or.at/hans/terrenatale/ The data is used to generate the large scale events like the dropping of coins for each country, the layout of static representing population, etc. I suppose that I would need to convert the data to a specific format for redis anyway, so just supporting CSV would be fine. I ended up using scripts and Pd to process a lot of the data. What would make puredis more interesting is if it meant that it would be easy to process large sets of data into something useful for generating sound and video. .hc On Jul 26, 2011, at 2:54 PM, Louis-Philippe wrote: before jumping on the puredis-loader implementation, here is how I see the specs, please tell me if I get what you think: __**_ Strings: string.csv: KEY, VALUE key1, value1 key2, value2 [lpuredis string string.csv] - creates a String for each line __**_ List and Sets: list.csv: KEY, VALUE1, VALUE2, ... key1, item1, item2, item3 key2, item1, item2, item3 [lpuredis list list.csv] - creates a List for each line [lpuredis set list.csv] - creates a Set for each line __**_ Sorted Sets: zset.csv: KEY, VALUE1, SCORE1, VALUE2, SCORE2 key1, item1, 1, item2, 2, item3, 3 key2, item1, 4, item2, 5, item3, 6 [lpuredis zset zset.csv] - creates a Sorted Set for each line __**__ Hashes hash.csv: KEY, KEY1, VALUE1, KEY2, VALUE2, KEY3, VALUE3 key1, hkey1, value1, hkey2, value2, hkey3, value3 key2, hkey1, value1, hkey2, value2, hkey3, value3 or even: KEY, FIELD1, FIELD2, FIELD3 key1, item1, item2, item3 key2, item1, item2, item3 [lpuredis hash hash.csv] - creates a Hash for each line __**_ For SQL file I am clueless, but I guess it will be similar... JSON would be the easiest to translate but would it be of any use to pd users? L-P --**--** If you are not part of the solution, you are part of the problem. ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management - http://lists.puredata.info/listinfo/pd-list
Re: [PD] Redis sync, async and subscriber client: Puredis
Thanks for the cue about the Library template, I will look into it and will probably integrate it. About Puredis intended usage... for my part I have in mind to integrate pd in a larger system using Redis as my data's rosetta stone. For Queue systems, Data Persistence, Pub/Sub IPC... I got used to the way you can manipulate large data sets inside Redis and tought it would match nicely with pd, and after implementating it I am even more excited about its potential. L-P 2011/7/25 Hans-Christoph Steiner h...@at.or.at That's a great manifesto, sounds like an interesting object. Something like [pool] but with more options. What do you have in mind to use it for? On the exciting topic of build systems, I recommend using the Library Template for this. It'll save you the headache of making a build system for GNU/Linux, Windows, etc. http://puredata.info/docs/developer/LibraryTemplate .hc On Jul 25, 2011, at 9:27 AM, Louis-Philippe wrote: Hi Pure Data Peoples, I'm still quite new to the Pure Data world, so please be indulgent to my first hacked external offering: puredis. Any of you heard of Redis? http://antirez.com/post/redis-manifesto.html for those of you who have not, it is: - NoSQL Key-Value Server - Fast - A DSL for abstract data types: Lists, Hashes, Sets, SortedSets - A Pub/Sub server Puredis brings Redis to the Pure Data world: [puredis] - Synchronous Blocking Redis client [apuredis] - Asynchronous Non-Blocking Redis client [spuredis] - Redis Pub/Sub Subscriber See https://github.com/lp/puredis#readme for build instructions and usage. Feel free to constructively criticize if you feel the pd integration lacks some fundamental functionality. ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management - http://lists.puredata.info/listinfo/pd-list I have always wished for my computer to be as easy to use as my telephone; my wish has come true because I can no longer figure out how to use my telephone. --Bjarne Stroustrup (creator of C++) ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management - http://lists.puredata.info/listinfo/pd-list
Re: [PD] Redis sync, async and subscriber client: Puredis
I would probably do the asset loading from outside pd, most programming languages have a redis client library. For this kind of job Ruby, Python, Perl, Lua or even V8 could do it. I think this kind of script would be trivial to implement. Something could also be achieved from within Redis with Lua, using the soon to be stable Scripting Branch http://antirez.com/post/an-update-on-redis-and-lua.html http://antirez.com/post/an-update-on-redis-and-lua.htmlBut as the Scripting Engine blocks Redis, I would not do anything to heavy in there, still it would probably be fast and convenient. Also, having the pdlua external loading stuff in redis from inside of PD could also be an option to consider. I'm sure these suggestions trigger more implementations ideas for the experienced pd users. Having a puredis-asset loader object as part of the puredis library could also be a possibility, given a CSV, SQL or JSON file path, the object could make sure redis is preset the right way. what do you think? 2011/7/26 Hans-Christoph Steiner h...@at.or.at How would you dump large data sets into redis? That's what I would likely use it for: taking data from CSV or SQL files and then using it within Pd. .hc On Jul 26, 2011, at 9:37 AM, Louis-Philippe wrote: Thanks for the cue about the Library template, I will look into it and will probably integrate it. About Puredis intended usage... for my part I have in mind to integrate pd in a larger system using Redis as my data's rosetta stone. For Queue systems, Data Persistence, Pub/Sub IPC... I got used to the way you can manipulate large data sets inside Redis and tought it would match nicely with pd, and after implementating it I am even more excited about its potential. L-P 2011/7/25 Hans-Christoph Steiner h...@at.or.at That's a great manifesto, sounds like an interesting object. Something like [pool] but with more options. What do you have in mind to use it for? On the exciting topic of build systems, I recommend using the Library Template for this. It'll save you the headache of making a build system for GNU/Linux, Windows, etc. http://puredata.info/docs/developer/LibraryTemplate .hc A cellphone to me is just an opportunity to be irritated wherever you are. - Linus Torvalds ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management - http://lists.puredata.info/listinfo/pd-list
Re: [PD] Redis sync, async and subscriber client: Puredis
I like the idea of a puredis-loader object. It could either be blocking, and that's just a known thing to workaround, i.e. load the file at patch startup. That's a good place to start at least, then things like threading, background loading, etc. could be explored later. .hc On Jul 26, 2011, at 1:22 PM, Louis-Philippe wrote: I would probably do the asset loading from outside pd, most programming languages have a redis client library. For this kind of job Ruby, Python, Perl, Lua or even V8 could do it. I think this kind of script would be trivial to implement. Something could also be achieved from within Redis with Lua, using the soon to be stable Scripting Branch http://antirez.com/post/an-update-on-redis-and-lua.html But as the Scripting Engine blocks Redis, I would not do anything to heavy in there, still it would probably be fast and convenient. Also, having the pdlua external loading stuff in redis from inside of PD could also be an option to consider. I'm sure these suggestions trigger more implementations ideas for the experienced pd users. Having a puredis-asset loader object as part of the puredis library could also be a possibility, given a CSV, SQL or JSON file path, the object could make sure redis is preset the right way. what do you think? 2011/7/26 Hans-Christoph Steiner h...@at.or.at How would you dump large data sets into redis? That's what I would likely use it for: taking data from CSV or SQL files and then using it within Pd. .hc On Jul 26, 2011, at 9:37 AM, Louis-Philippe wrote: Thanks for the cue about the Library template, I will look into it and will probably integrate it. About Puredis intended usage... for my part I have in mind to integrate pd in a larger system using Redis as my data's rosetta stone. For Queue systems, Data Persistence, Pub/Sub IPC... I got used to the way you can manipulate large data sets inside Redis and tought it would match nicely with pd, and after implementating it I am even more excited about its potential. L-P 2011/7/25 Hans-Christoph Steiner h...@at.or.at That's a great manifesto, sounds like an interesting object. Something like [pool] but with more options. What do you have in mind to use it for? On the exciting topic of build systems, I recommend using the Library Template for this. It'll save you the headache of making a build system for GNU/Linux, Windows, etc. http://puredata.info/docs/developer/LibraryTemplate .hc A cellphone to me is just an opportunity to be irritated wherever you are. - Linus Torvalds Man has survived hitherto because he was too ignorant to know how to realize his wishes. Now that he can realize them, he must either change them, or perish.-William Carlos Williams ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management - http://lists.puredata.info/listinfo/pd-list
Re: [PD] Redis sync, async and subscriber client: Puredis
before jumping on the puredis-loader implementation, here is how I see the specs, please tell me if I get what you think: ___ Strings: string.csv: KEY, VALUE key1, value1 key2, value2 [lpuredis string string.csv] - creates a String for each line ___ List and Sets: list.csv: KEY, VALUE1, VALUE2, ... key1, item1, item2, item3 key2, item1, item2, item3 [lpuredis list list.csv] - creates a List for each line [lpuredis set list.csv] - creates a Set for each line ___ Sorted Sets: zset.csv: KEY, VALUE1, SCORE1, VALUE2, SCORE2 key1, item1, 1, item2, 2, item3, 3 key2, item1, 4, item2, 5, item3, 6 [lpuredis zset zset.csv] - creates a Sorted Set for each line Hashes hash.csv: KEY, KEY1, VALUE1, KEY2, VALUE2, KEY3, VALUE3 key1, hkey1, value1, hkey2, value2, hkey3, value3 key2, hkey1, value1, hkey2, value2, hkey3, value3 or even: KEY, FIELD1, FIELD2, FIELD3 key1, item1, item2, item3 key2, item1, item2, item3 [lpuredis hash hash.csv] - creates a Hash for each line ___ For SQL file I am clueless, but I guess it will be similar... JSON would be the easiest to translate but would it be of any use to pd users? L-P ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management - http://lists.puredata.info/listinfo/pd-list
Re: [PD] Redis sync, async and subscriber client: Puredis
Where I could see it used it with large data sets like I did in this project: http://at.or.at/hans/terrenatale/ The data is used to generate the large scale events like the dropping of coins for each country, the layout of static representing population, etc. I suppose that I would need to convert the data to a specific format for redis anyway, so just supporting CSV would be fine. I ended up using scripts and Pd to process a lot of the data. What would make puredis more interesting is if it meant that it would be easy to process large sets of data into something useful for generating sound and video. .hc On Jul 26, 2011, at 2:54 PM, Louis-Philippe wrote: before jumping on the puredis-loader implementation, here is how I see the specs, please tell me if I get what you think: ___ Strings: string.csv: KEY, VALUE key1, value1 key2, value2 [lpuredis string string.csv] - creates a String for each line ___ List and Sets: list.csv: KEY, VALUE1, VALUE2, ... key1, item1, item2, item3 key2, item1, item2, item3 [lpuredis list list.csv] - creates a List for each line [lpuredis set list.csv] - creates a Set for each line ___ Sorted Sets: zset.csv: KEY, VALUE1, SCORE1, VALUE2, SCORE2 key1, item1, 1, item2, 2, item3, 3 key2, item1, 4, item2, 5, item3, 6 [lpuredis zset zset.csv] - creates a Sorted Set for each line Hashes hash.csv: KEY, KEY1, VALUE1, KEY2, VALUE2, KEY3, VALUE3 key1, hkey1, value1, hkey2, value2, hkey3, value3 key2, hkey1, value1, hkey2, value2, hkey3, value3 or even: KEY, FIELD1, FIELD2, FIELD3 key1, item1, item2, item3 key2, item1, item2, item3 [lpuredis hash hash.csv] - creates a Hash for each line ___ For SQL file I am clueless, but I guess it will be similar... JSON would be the easiest to translate but would it be of any use to pd users? L-P If you are not part of the solution, you are part of the problem. ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management - http://lists.puredata.info/listinfo/pd-list
[PD] Redis sync, async and subscriber client: Puredis
Hi Pure Data Peoples, I'm still quite new to the Pure Data world, so please be indulgent to my first hacked external offering: puredis. Any of you heard of Redis? http://antirez.com/post/redis-manifesto.html for those of you who have not, it is: - NoSQL Key-Value Server - Fast - A DSL for abstract data types: Lists, Hashes, Sets, SortedSets - A Pub/Sub server Puredis brings Redis to the Pure Data world: [puredis] - Synchronous Blocking Redis client [apuredis] - Asynchronous Non-Blocking Redis client [spuredis] - Redis Pub/Sub Subscriber See https://github.com/lp/puredis#readme for build instructions and usage. Feel free to constructively criticize if you feel the pd integration lacks some fundamental functionality. ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management - http://lists.puredata.info/listinfo/pd-list
Re: [PD] Redis sync, async and subscriber client: Puredis
On Mon, Jul 25, 2011 at 09:27:39AM -0400, Louis-Philippe wrote: I'm still quite new to the Pure Data world, so please be indulgent to my first hacked external offering: puredis. Any of you heard of Redis? http://antirez.com/post/redis-manifesto.html Hi, Very cool, thanks for sharing! Chris. --- http://mccormick.cx ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management - http://lists.puredata.info/listinfo/pd-list
Re: [PD] Redis sync, async and subscriber client: Puredis
That's a great manifesto, sounds like an interesting object. Something like [pool] but with more options. What do you have in mind to use it for? On the exciting topic of build systems, I recommend using the Library Template for this. It'll save you the headache of making a build system for GNU/Linux, Windows, etc. http://puredata.info/docs/developer/LibraryTemplate .hc On Jul 25, 2011, at 9:27 AM, Louis-Philippe wrote: Hi Pure Data Peoples, I'm still quite new to the Pure Data world, so please be indulgent to my first hacked external offering: puredis. Any of you heard of Redis? http://antirez.com/post/redis- manifesto.html for those of you who have not, it is: - NoSQL Key-Value Server - Fast - A DSL for abstract data types: Lists, Hashes, Sets, SortedSets - A Pub/Sub server Puredis brings Redis to the Pure Data world: [puredis] - Synchronous Blocking Redis client [apuredis] - Asynchronous Non-Blocking Redis client [spuredis] - Redis Pub/Sub Subscriber See https://github.com/lp/puredis#readme for build instructions and usage. Feel free to constructively criticize if you feel the pd integration lacks some fundamental functionality. ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management - http://lists.puredata.info/listinfo/pd-list I have always wished for my computer to be as easy to use as my telephone; my wish has come true because I can no longer figure out how to use my telephone. --Bjarne Stroustrup (creator of C++) ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management - http://lists.puredata.info/listinfo/pd-list
