Re: powerschool webscraper?

[David Rysdam]

David Rysdam  writes:
> However, there's no change detection. I guess I have to have memorized
> all the grades, outstanding assignments, etc for each child so I can see
> what's different when I check the next time. "Huh...it's 90% now. Was it
> 91% yesterday? Maybe I should click through and see if he flunked
> something."
>
> I figured I'd do a simple little script that would download the page and
> just do a diff. Not so fast--it's hidden behind javascript.

So I solved this today (or I think I did--have to actually get some
output before I'm sure, but It Should Work(tm)). Nobody really cares
about the solution per se, but when/if any of you get some kids with
grades hidden inside PowerSchool, here's what I did:

They have a way to email results out. On the downside, you can't ask it
only for new stuff--it email blasts you every class, every day for every
child. (Clearly nobody at PowerSchool has any kids in school.) On the
upside, it's plain text and gives a dump of all the assignments and
their grades (for now...).

I created a separate email account just for receiving these
messages. Then I have a script that canonicalizes each one (by child and
class) and does a diff against a git repository, mailing me if anything
happened.

The extra nice part of this is that I can easily add another
subscriber. My wife, obvs, but also the children themselves because
apparently *that's the only way they know what they got* (for at least
some assignments). 

Maybe the Hour of Code will create a populace that demands a more
programmable Internet...
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: (was: powerschool webscraper?)

[Ben Scott]

On Nov 3, 2014 5:44 PM, "Joshua Judson Rosen"  wrote:
> I was pretty frustrated when I saw the hoops one needs to jump through
> to make blinking text[1], these days, since the browsers
> finally neutered both  and text-decoration:blink.

Firefox used to have an "about:config" option that could (re)enable
blinking, although I don't know if it still does.

But blinking text should be easy to do when you're reimplementing entire UI
concepts in HTML/CSS because all the cool kids are doing it that way these
days.

-- Ben
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

(was: powerschool webscraper?)

[Joshua Judson Rosen]

Ben Scott  writes:
>
> On Thu, Oct 30, 2014 at 8:29 AM, David Rysdam  wrote:
> >
> > Why does an extremely simple, automatable task like "check if
> > posted grades have changed" require a human being to spend valuable time
> > poking buttons (or programming a very faithful simulation thereof)?
>
>   Because software these days is mostly about chasing buzzwords and
> fashion trends, and not about information.
>
>   The current crop of crap appeals to the same mentality that thought
> the  tag was a good idea.
>
>   (Aside: I've just discovered (for myself) that if you search for
> "blink tag", Google makes all matching occurrences blink.)

I, actually, really wanted to use  or it's CSS equivalent, recently.
Mere weeks ago.

It was in an embedded HTML-based UI with minimal content, where the
prime goal was just to get the user's attention by screaming
"OMG SOMETHING IS BROKEN DOESN'T IT LOOK BROKEN TO YOU!!!" at them.

Blinking text is a lousy way to convey information that someone actually
needs to parse, but a screenful of blink is great way to look broken.

I was pretty frustrated when I saw the hoops one needs to jump through
to make blinking text[1], these days, since the browsers
finally neutered both  and text-decoration:blink.

Thanks to everyone who just couldn't restrain themselves--
you ruined it for us all, guys. I'm waiting for Mozilla to
save us from UPPERCASE YELLING BY FORCIBLY downcasing everything


Footnotes:

[1]: http://www.jwz.org/jwz.css?ver=1
 cf. http://www.jwz.org/blog/2013/08/a-light-has-gone-out-on-the-web/

-- 
"'tis an ill wind that blows no minds."
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[Ben Scott]

On Thu, Oct 30, 2014 at 8:24 AM, David Rysdam  wrote:
> I don't need or even want a GUI. I want to run it as a cron
> job and email myself a result when there is one.

  Have the script start a VNC or null or other similar X server to host the GUI.

  Ugly, but it's the ultimate implementation of "simulate what a human
with a browser would do".

  It would be better if the world weren't ugly, but yet, here we are.

-- Ben
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[Ben Scott]

On Thu, Oct 30, 2014 at 8:29 AM, David Rysdam  wrote:
> It's also making me depressed how terrible the internet has
> become.

  Yes.

> Why does an extremely simple, automatable task like "check if
> posted grades have changed" require a human being to spend valuable time
> poking buttons (or programming a very faithful simulation thereof)?

  Because software these days is mostly about chasing buzzwords and
fashion trends, and not about information.

  The current crop of crap appeals to the same mentality that thought
the  tag was a good idea.

  (Aside: I've just discovered (for myself) that if you search for
"blink tag", Google makes all matching occurrences blink.)

-- Ben
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[Jeffry Smith]

https://github.com/baryon5/powerschool

A notification service and client for PowerSchool

https://github.com/ksdtech/powerschool-web

Custom html and javascript for PowerSchool

https://github.com/zacharyliu/mcvsd-powerschool-notify

MCVSD PowerSchool notification system, powered by node.js


Looks like lots of folks are extending PowerSchool - I've never used
it, but hopefully something above will work.

jeff

On Thu, Oct 30, 2014 at 9:24 AM, David Rysdam  wrote:
> Tom Buskey  writes:
>> Powerschool also has an API and there are tools that hook into it.  For
>> example: https://github.com/powerapi/powerapi-php
>
> Ah, this is interesting.
> ___
> gnhlug-discuss mailing list
> gnhlug-discuss@mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Fwd: powerschool webscraper?

[David Berube]

Meant to send this to the list...

-- Forwarded message --
From: David Berube 
Date: Thu, Oct 30, 2014 at 10:06 AM
Subject: Re: powerschool webscraper?
To: David Rysdam 




On Thursday, October 30, 2014, David Rysdam  wrote:
>
> I had this problem with htmlunit too. It wanted to fire up a browser
> GUI. No, bad tool!
>
> Probably there are no GUI-less, javascript-capable, scriptable tools out
> there. There's probably some way to cobble one together now that guile
> supports javascript, but like I said: time.
>
>
PhantomJS.

David Berube
Berube Consulting


-- 

Take it easy,

David Berube
Berube Consulting
http://berubeconsulting.com
P.O Box 1746
Concord, NH 03302
United States
Tel: (603) 574-4766



-- 

Take it easy,

David Berube
Berube Consulting
http://berubeconsulting.com
P.O Box 1746
Concord, NH 03302
United States
Tel: (603) 574-4766
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[Cole Tuininga]

On 10/30/2014 08:24 AM, David Rysdam wrote:
> Tom Buskey  writes:
>> I'd look at Selenium .  I heard about it from
>> our QA guys for automation.
> 
> It's cool, but not really what I want. I'm not automating this because I
> care how a particular browser works. I'm automating it because I want it
> automated. I don't need or even want a GUI. I want to run it as a cron
> job and email myself a result when there is one.
> 
> I had this problem with htmlunit too. It wanted to fire up a browser
> GUI. No, bad tool!

I'm pretty sure that you can run Selenium headless.  I don't know much
about the details, but I know that part of the functional and
integration test suite we run at $DayJob is using selenium to test a
javascript driven web site, without having to pop up a browser...

-- 
Cole Tuininga
Lead Developer
co...@code-energy.com
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[Larry & Brigitte Cook]

On 10/30/2014 8:24 AM, David Rysdam wrote:
> I had this problem with htmlunit too. It wanted to fire up a browser
> GUI. No, bad tool!

Look at httpunit.  I recall using it once and it did not require a 
browser.  Not sure if it has JavaScript support.

Larry
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[David Rysdam]

Tom Buskey  writes:
> Powerschool also has an API and there are tools that hook into it.  For
> example: https://github.com/powerapi/powerapi-php

Ah, this is interesting.
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[Tom Buskey]

Elinks?  It has some javascript support that calls Mozilla's engine.  It
has a text interface.  I've used lynx -dump in the past, but that doesn't
have java script.

I've seen OpenOffice automated.  A kluge would be to fire up selenium,
print/save to PDF, compare PDFs.  FWIW I have a postscript differ.

Powerschool also has an API and there are tools that hook into it.  For
example: https://github.com/powerapi/powerapi-php

On Thu, Oct 30, 2014 at 8:29 AM, David Rysdam  wrote:

> David Rysdam  writes:
> > I had this problem with htmlunit too. It wanted to fire up a browser
> > GUI. No, bad tool!
> >
> > Probably there are no GUI-less, javascript-capable, scriptable tools out
> > there. There's probably some way to cobble one together now that guile
> > supports javascript, but like I said: time.
>
> It's also making me depressed how terrible the internet has
> become. We've made a thing to run on computer that computers themselves
> can't use. Why does an extremely simple, automatable task like "check if
> posted grades have changed" require a human being to spend valuable time
> poking buttons (or programming a very faithful simulation thereof)?
>
> We used to be a tool-making species. Then a tool-using species. Now
> we're a tool-used species.
>
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[David Rysdam]

David Rysdam  writes:
> I had this problem with htmlunit too. It wanted to fire up a browser
> GUI. No, bad tool!
>
> Probably there are no GUI-less, javascript-capable, scriptable tools out
> there. There's probably some way to cobble one together now that guile
> supports javascript, but like I said: time.

It's also making me depressed how terrible the internet has
become. We've made a thing to run on computer that computers themselves
can't use. Why does an extremely simple, automatable task like "check if
posted grades have changed" require a human being to spend valuable time
poking buttons (or programming a very faithful simulation thereof)?

We used to be a tool-making species. Then a tool-using species. Now
we're a tool-used species. 
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[David Rysdam]

Tom Buskey  writes:
> I'd look at Selenium .  I heard about it from
> our QA guys for automation.

It's cool, but not really what I want. I'm not automating this because I
care how a particular browser works. I'm automating it because I want it
automated. I don't need or even want a GUI. I want to run it as a cron
job and email myself a result when there is one.

I had this problem with htmlunit too. It wanted to fire up a browser
GUI. No, bad tool!

Probably there are no GUI-less, javascript-capable, scriptable tools out
there. There's probably some way to cobble one together now that guile
supports javascript, but like I said: time.
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[Tom Buskey]

I'd look at Selenium .  I heard about it from
our QA guys for automation.



On Wed, Oct 29, 2014 at 4:35 PM, David Rysdam  wrote:

> Bill Freeman  writes:
> > Not that I know of an existing tool (BeautifulSoup probably doesn't deal
> > with the JavaScript), but it seems like you want to capture the eventual
> > DOM.  You need something that you can trigger when the page has
> > settled.
>
> Nope. I thought this too, but it's not true. I have absolutely no
> compunction about actually parsing the HTML--I would just grep for
> whatever keyword worked and jam a POST in the web form for my
> username/password, BeautifulSoup be damned.
>
> However, it's something else. I can't think of the term now, but it
> isn't a matter finding something in the page I can use. I have to
> actually simulate a web browser to run the Javascript so I communicate
> with the server and get the right security tokens and whatever.
> ___
> gnhlug-discuss mailing list
> gnhlug-discuss@mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[David Rysdam]

Bill Freeman  writes:
> Not that I know of an existing tool (BeautifulSoup probably doesn't deal
> with the JavaScript), but it seems like you want to capture the eventual
> DOM.  You need something that you can trigger when the page has
> settled. 

Nope. I thought this too, but it's not true. I have absolutely no
compunction about actually parsing the HTML--I would just grep for
whatever keyword worked and jam a POST in the web form for my
username/password, BeautifulSoup be damned.

However, it's something else. I can't think of the term now, but it
isn't a matter finding something in the page I can use. I have to
actually simulate a web browser to run the Javascript so I communicate
with the server and get the right security tokens and whatever. 
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[Bill Freeman]

There is a "Save back to file from DOM" firefox add on.  I don't know
whether it requires that the page was loaded from a file.

On Wed, Oct 29, 2014 at 3:21 PM, Bill Freeman  wrote:

> And further reading suggests that DOM Inspector's save to file
> functionality has been removed.  You might be able to get an old version,
> though that could require an old version of firefox.
>
> On Wed, Oct 29, 2014 at 3:18 PM, Bill Freeman  wrote:
>
>> Not that I know of an existing tool (BeautifulSoup probably doesn't deal
>> with the JavaScript), but it seems like you want to capture the eventual
>> DOM.  You need something that you can trigger when the page has settled.  A
>> manual way might be to use firebug (assuming that firefox works with the
>> site), although I'm not sure that saving the DOM to a file is built in.
>> The Greasemonkey add on for firefox may let you automate it more.  If you
>> can't save to a file, you might be able to post it to a local server,
>> though cross site scripting protections in your browser could get in the
>> way.  One approach would be to pop up an extra div with the extracted stuff
>> that would be suitable for copy from the browser and paste to a file.
>>
>> A little more googling lead me to the "DOM Inspector" firefox add on,
>> which might do the trick without fiddling.
>>
>> On Wed, Oct 29, 2014 at 2:56 PM, David Rysdam  wrote:
>>
>>> The Milford school district uses this "PowerSchool" web thing so parents
>>> can check on kids' grades (middle school and up). It's pretty useful,
>>> for those teachers that update at a reasonable frequency.
>>>
>>> However, there's no change detection. I guess I have to have memorized
>>> all the grades, outstanding assignments, etc for each child so I can see
>>> what's different when I check the next time. "Huh...it's 90% now. Was it
>>> 91% yesterday? Maybe I should click through and see if he flunked
>>> something."
>>>
>>> I figured I'd do a simple little script that would download the page and
>>> just do a diff. Not so fast--it's hidden behind javascript.
>>>
>>> I know this would be trivial for someone who has done this. Heck, it
>>> would probably be trivial for me if I didn't have 100 other projects
>>> going and I had time to learn the tools (looks like "htmlunit" in this
>>> case). However, it suddenly occurred to me that someone may have
>>> *already* done this, maybe even for Milford's particular installation.
>>>
>>> So...have you?
>>> ___
>>> gnhlug-discuss mailing list
>>> gnhlug-discuss@mail.gnhlug.org
>>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>>>
>>
>>
>
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[Bill Freeman]

And further reading suggests that DOM Inspector's save to file
functionality has been removed.  You might be able to get an old version,
though that could require an old version of firefox.

On Wed, Oct 29, 2014 at 3:18 PM, Bill Freeman  wrote:

> Not that I know of an existing tool (BeautifulSoup probably doesn't deal
> with the JavaScript), but it seems like you want to capture the eventual
> DOM.  You need something that you can trigger when the page has settled.  A
> manual way might be to use firebug (assuming that firefox works with the
> site), although I'm not sure that saving the DOM to a file is built in.
> The Greasemonkey add on for firefox may let you automate it more.  If you
> can't save to a file, you might be able to post it to a local server,
> though cross site scripting protections in your browser could get in the
> way.  One approach would be to pop up an extra div with the extracted stuff
> that would be suitable for copy from the browser and paste to a file.
>
> A little more googling lead me to the "DOM Inspector" firefox add on,
> which might do the trick without fiddling.
>
> On Wed, Oct 29, 2014 at 2:56 PM, David Rysdam  wrote:
>
>> The Milford school district uses this "PowerSchool" web thing so parents
>> can check on kids' grades (middle school and up). It's pretty useful,
>> for those teachers that update at a reasonable frequency.
>>
>> However, there's no change detection. I guess I have to have memorized
>> all the grades, outstanding assignments, etc for each child so I can see
>> what's different when I check the next time. "Huh...it's 90% now. Was it
>> 91% yesterday? Maybe I should click through and see if he flunked
>> something."
>>
>> I figured I'd do a simple little script that would download the page and
>> just do a diff. Not so fast--it's hidden behind javascript.
>>
>> I know this would be trivial for someone who has done this. Heck, it
>> would probably be trivial for me if I didn't have 100 other projects
>> going and I had time to learn the tools (looks like "htmlunit" in this
>> case). However, it suddenly occurred to me that someone may have
>> *already* done this, maybe even for Milford's particular installation.
>>
>> So...have you?
>> ___
>> gnhlug-discuss mailing list
>> gnhlug-discuss@mail.gnhlug.org
>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>>
>
>
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[Bill Freeman]

Not that I know of an existing tool (BeautifulSoup probably doesn't deal
with the JavaScript), but it seems like you want to capture the eventual
DOM.  You need something that you can trigger when the page has settled.  A
manual way might be to use firebug (assuming that firefox works with the
site), although I'm not sure that saving the DOM to a file is built in.
The Greasemonkey add on for firefox may let you automate it more.  If you
can't save to a file, you might be able to post it to a local server,
though cross site scripting protections in your browser could get in the
way.  One approach would be to pop up an extra div with the extracted stuff
that would be suitable for copy from the browser and paste to a file.

A little more googling lead me to the "DOM Inspector" firefox add on, which
might do the trick without fiddling.

On Wed, Oct 29, 2014 at 2:56 PM, David Rysdam  wrote:

> The Milford school district uses this "PowerSchool" web thing so parents
> can check on kids' grades (middle school and up). It's pretty useful,
> for those teachers that update at a reasonable frequency.
>
> However, there's no change detection. I guess I have to have memorized
> all the grades, outstanding assignments, etc for each child so I can see
> what's different when I check the next time. "Huh...it's 90% now. Was it
> 91% yesterday? Maybe I should click through and see if he flunked
> something."
>
> I figured I'd do a simple little script that would download the page and
> just do a diff. Not so fast--it's hidden behind javascript.
>
> I know this would be trivial for someone who has done this. Heck, it
> would probably be trivial for me if I didn't have 100 other projects
> going and I had time to learn the tools (looks like "htmlunit" in this
> case). However, it suddenly occurred to me that someone may have
> *already* done this, maybe even for Milford's particular installation.
>
> So...have you?
> ___
> gnhlug-discuss mailing list
> gnhlug-discuss@mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

Re: powerschool webscraper?

[Matt Minuti]

I found myself wanting a similar scraper a couple years ago for looking for
job postings hidden behind similar javascript silliness that took forever
to manually check. Never got around to it, though.

"Selenium" looks to be a possible alternative to htmlunit, might be easier
to use?

I'm still interested in what you come up with :)

On Wed, Oct 29, 2014 at 2:56 PM, David Rysdam  wrote:

> The Milford school district uses this "PowerSchool" web thing so parents
> can check on kids' grades (middle school and up). It's pretty useful,
> for those teachers that update at a reasonable frequency.
>
> However, there's no change detection. I guess I have to have memorized
> all the grades, outstanding assignments, etc for each child so I can see
> what's different when I check the next time. "Huh...it's 90% now. Was it
> 91% yesterday? Maybe I should click through and see if he flunked
> something."
>
> I figured I'd do a simple little script that would download the page and
> just do a diff. Not so fast--it's hidden behind javascript.
>
> I know this would be trivial for someone who has done this. Heck, it
> would probably be trivial for me if I didn't have 100 other projects
> going and I had time to learn the tools (looks like "htmlunit" in this
> case). However, it suddenly occurred to me that someone may have
> *already* done this, maybe even for Milford's particular installation.
>
> So...have you?
> ___
> gnhlug-discuss mailing list
> gnhlug-discuss@mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/

powerschool webscraper?

[David Rysdam]

The Milford school district uses this "PowerSchool" web thing so parents
can check on kids' grades (middle school and up). It's pretty useful,
for those teachers that update at a reasonable frequency.

However, there's no change detection. I guess I have to have memorized
all the grades, outstanding assignments, etc for each child so I can see
what's different when I check the next time. "Huh...it's 90% now. Was it
91% yesterday? Maybe I should click through and see if he flunked
something."

I figured I'd do a simple little script that would download the page and
just do a diff. Not so fast--it's hidden behind javascript.

I know this would be trivial for someone who has done this. Heck, it
would probably be trivial for me if I didn't have 100 other projects
going and I had time to learn the tools (looks like "htmlunit" in this
case). However, it suddenly occurred to me that someone may have
*already* done this, maybe even for Milford's particular installation.

So...have you?
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/