[nodejs] Celery-equivalent in Node?

[Victor Hooi]

Hi,

I'm starting with Node.JS, and I'm writing an app that will schedule tasks 
to run at pre-set times.

Some tasks are one-offs, some are recurring - they involve things like 
checking if a host is up, or grepping a remote file, or checking for a 
running process.

So basically, I'm looking for the equavilent of Celery but in Node.

I did find Kue (https://github.com/learnboost/kue), which just from reading 
the homepage seems like the closest match, and is in pure JS.

There's also coffee-resque (https://github.com/technoweenie/coffee-resque), 
however, I'm not sure how active that is, and I'd need to setup Resque as 
well.

Do others have thoughts or experiences in job queue systems in Node? What 
worked well and didn't?

Cheers,
Victor

-- 
-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[nodejs] Browserable REST API framework?

[Victor Hooi]

Hi,

I'm currently using the Django Rest Framework 
(http://django-rest-framework.org/), which is, as the name suggests, a REST 
framework for Django.

It has a very neat browserable API - you can see a demo here:

http://restframework.herokuapp.com/

Basically, if you go to the REST endpoints with a browser, it offers a nice 
HTML interface, and if it's a writeable endpoint, you can also use a 
webform to POST/PUT etc.

It's quite nice for discovery, and also great for testing/debugging.

I'm just curious if anybody knows of something similar or equivalent in 
Node.js?

Cheers,
Victor

-- 
-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[nodejs] Using Node.js for managing image file workflow?

[Victor Hooi]

Hi,

I'm new to Node, and was hoping to use Node.js for a a small internal 
webapp for managing workflow for product photos.

The image files are RAW camera files, and stored on a local NAS device.

The webapp should:

   - Have a concept of workflow, and be able to jump back/forth between 
   states, and handle error states.
   - Watch certain directories for image files, and react to new files 
   being added, or existing files being moved/removed.
   - Send out emails in response to events.
   - Scan photos for QR barcodes, and generate events based on these.
   - Rename photos based on user-defined batch patterns in response to 
   events.

Questions:


   1. Is Node.js a suitable tool for something like this? Why or why not?
   2. Any libraries to help manage the workflow? I could only find 
   node-workflow (http://kusor.github.io/node-workflow/) - curious for 
   anybody's experiences with this? Alternatives?
   3. Likewise for file watching? I saw many wrappers for fs.watch (e.g. 
   https://github.com/mikeal/watch), as well as some alternatives (e.g. 
   https://github.com/paulmillr/chokidar) - any advice for somebody new to 
   Node?
   4. Apart from using a NAS with a network filesystem, are there any other 
   alternatives stores I can use for the image files?
   
   I'm open to other alternatives here. I'm worried that the system will 
   get confused or lose track of files.
   
   The Node.js docs also mention that watching files on network file 
   systems might be unreliable 
   (http://nodejs.org/api/fs.html#fs_fs_watch_filename_options_listener). Are 
   there more robust solutions?
   5. Any other tips/suggestions for this project?

Cheers,
Victor

-- 
-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[nodejs] Any library for mirroring website (ala HTTrack) within Node?

[Victor Hooi]

Hi,

Do you know of any Node libraries or packages that would help with mirroring a 
website offline?

I'm basically looking for something like HTTrack/WinHTTrack 
(http://www.httrack.com), but for use within Node.

Cheers,
Victor

-- 
-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[nodejs] Issues installing "logparser" module from NPM

[Victor Hooi]

Hi,

I was trying to the logparser module from NPM:

https://npmjs.org/package/logparser

However, I got the following error complaining about the 
logparser/bin/parselog file?

$ npm install logparser
> npm http GET https://registry.npmjs.org/logparser
> npm http 200 https://registry.npmjs.org/logparser
> npm http GET https://registry.npmjs.org/logparser/-/logparser-0.0.6.tgz
> npm http 200 https://registry.npmjs.org/logparser/-/logparser-0.0.6.tgz
> npm ERR! Error: ENOENT, chmod 
> '/Users/victor/code/bextrack/node_modules/logparser/bin/parselog'
> npm ERR! If you need help, you may report this log at:
> npm ERR! 
> npm ERR! or email it to:
> npm ERR! 
> npm ERR! System Darwin 12.4.0
> npm ERR! command "/usr/local/Cellar/node/0.10.20/bin/node" 
> "/usr/local/bin/npm" "install" "logparser"
> npm ERR! cwd /Users/victor/code/bextrack
> npm ERR! node -v v0.10.20
> npm ERR! npm -v 1.3.11
> npm ERR! path 
> /Users/victor/code/bextrack/node_modules/logparser/bin/parselog
> npm ERR! code ENOENT
> npm ERR! errno 34
> npm ERR!
> npm ERR! Additional logging details can be found in:
> npm ERR! /Users/victor/code/bextrack/npm-debug.log
> npm ERR! not ok code 0


I've uploaded the debug NPM log to here:

https://gist.github.com/7000640

I don't understand enough of how NPM works under the hood to decipher all 
of the logfile - however, it seems to be failing on some kind of uninstall, 
which is weird? 

However, I did download the tarball 
(https://registry.npmjs.org/logparser/-/logparser-0.0.6.tgz) manually, and 
it doesn't seem to be complete, somehow? 

There isn't a bin/parselog file, but there is bin/parselog.js?

Also, as a sidenote, I wasn't able to find the sourcecode for this module 
on Github, or elsewhere, I'm guessing the author's just uploaded the 
package to NPM and that's it.

Cheers,
Victor

-- 
-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[nodejs] How to do a PostgreSQL COPY with Node.JS?

[Victor Hooi]

Hi,

PostgreSQL offers a COPY command, to move data efficient between the 
database and files.

So basically, you can bulk-export a SQL command into a local CSV command.

I come from the Python world, and there's the psycopg2 library, which 
offers copy_to() method to dump a SQL command straight into a local 
file-like object:

http://initd.org/psycopg/docs/usage.html#using-copy-to-and-copy-from

I'm new to Node.js, and I'm wondering if there's something similar for 
Node.JS - perhaps that can stream the resulting CSV file back?

I had a look at the node-postgres library:

https://github.com/brianc/node-postgres

and from what I can tell, it seems to operate on individual rows for a 
query result.

Is there an efficient way of bulk dumping a PostgreSQL table into Node.JS?

Cheers,
Victor

-- 
-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[nodejs] Re: How to do a PostgreSQL COPY with Node.JS?

[Victor Hooi]

Hi,

I saw the author also has this repo:

https://github.com/brianc/node-pg-copy-streams

I'm curious if anybody has experience with it, or how it compares to the 
node-postgres Client that greelgorke mentioned?

Cheers,
Victor

On Monday, 18 November 2013 19:57:35 UTC+11, greelgorke wrote:
>
> my 10 secs on that: 
> https://github.com/brianc/node-postgres/wiki/Client#wiki-bulk-load
>
> Am Montag, 18. November 2013 01:43:54 UTC+1 schrieb Victor Hooi:
>>
>> Hi,
>>
>> PostgreSQL offers a COPY command, to move data efficient between the 
>> database and files.
>>
>> So basically, you can bulk-export a SQL command into a local CSV command.
>>
>> I come from the Python world, and there's the psycopg2 library, which 
>> offers copy_to() method to dump a SQL command straight into a local 
>> file-like object:
>>
>> http://initd.org/psycopg/docs/usage.html#using-copy-to-and-copy-from
>>
>> I'm new to Node.js, and I'm wondering if there's something similar for 
>> Node.JS - perhaps that can stream the resulting CSV file back?
>>
>> I had a look at the node-postgres library:
>>
>> https://github.com/brianc/node-postgres
>>
>> and from what I can tell, it seems to operate on individual rows for a 
>> query result.
>>
>> Is there an efficient way of bulk dumping a PostgreSQL table into Node.JS?
>>
>> Cheers,
>> Victor
>>
>

-- 
-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[nodejs] Node.JS for sysadmin scripting or non-web stuff - books, guides, examples?

[Victor Hooi]

Hi,

I'm curious on people's experiences with using Node.JS for sys-admin 
scripts or non-Web glue stuff - i.e. the sorts of things you might use Perl 
or Python for (e.g. log parsing, manipulating config files etc.)

Is Node.JS suitable for these things?

Are there any books or guides out there on these sorts of things? (Most of 
the books I've seen are geared to Node.JS purely as a web app language).

Or any detailed examples people have posted up of what they use it for?

Cheers,
Victor

-- 
-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[nodejs] Web Scraping Frameworks for Node.JS? (e.g. like Python's Scrapy)

[Victor Hooi]

Hi,

I'm wondering if anybody knows of any web-scraping frameworks in Node.JS?

Previously, there was node.io (https://github.com/chriso/node.io), however, 
the project was recently discontinued.

Googling for Node.JS and web scraping, most of the guides online just talk 
about using requests and cheerio - it works, but you need to handle a whole 
bunch of things yourself (throttling, distributing jobs, configuration, 
managing jobs etc.).

On the Python side, I know of Scrapy (https://github.com/scrapy/scrapy), 
which is using Twisted for asynchronicity

On the Ruby side, Nokogiri (http://nokogiri.org/) is meant to be good, 
although I haven't dived into it much.

Is there anything equivalent in the Node world? Or what approaches are 
people using to tackle this problem?

Cheers,
Victor

-- 
-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [nodejs] Web Scraping Frameworks for Node.JS? (e.g. like Python's Scrapy)

[Victor Hooi]

Hi,

If you look at the docs and feature list for Scrapy, you'll see it has a
whole bunch of scraping features more than just selecting a DOM element.
E.g.:

http://doc.scrapy.org/en/latest/intro/overview.html#what-else

So for us, this would also cover handling things like throttling,
configuration, distributing jobs, managing jobs etc.

PhantomJS seems to be the way to go, from other people's comments.

Most of the full-featured Node frameworks seem to be inactive. E.g.:

https://github.com/chriso/node.io
https://github.com/sylvinus/node-crawler
https://github.com/mape/node-scraper

Only one I've found which is still actively maintained, which Matthew Page
above mention is Noodle:

https://github.com/dharmafly/noodle

Cheers,
Victor


On Sat, Jan 25, 2014 at 3:56 PM, Jamie Popkin  wrote:

> At the risk of sounding stupid... Can't you just use jQuery? It's got
> everything you need for fetching and parsing web content.
>
> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "nodejs" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/nodejs/0E76dy0mgwI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> nodejs+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--- 
You received this message because you are subscribed to the Google Groups 
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.