LinkedIn is sensitive to scraping. You need to set user agent headers and
use a proxy.
On Jul 11, 2015 3:25 AM, "Rock Slate" <[email protected]> wrote:
> Hi,
>
> I am trying to scrape a public linkedin profile using scrapy. I started of
> with scrapy shell linkedinprofile. When I try the response.body I get this
> message
> Out[1]: '<html><head>\n<script type="text/javascript">\nwindow.onload =
> function() {\n var newLocat
> ion = "";\n if (window.location.protocol == "http:") {\n var cookies =
> document.cookie.split(";
> ");\n for (var i = 0; i < cookies.length; ++i) {\n if
> ((cookies[i].indexOf("sl=") == 0) && (
> cookies[i].length > 3)) {\n newLocation = "https:" +
> window.location.href.substring(window.lo
> cation.protocol.length);\n }\n }\n }\n\n if (newLocation.length
> == 0) {\n var domain =
> location.host;\n var newDomainIndex = 0;\n if (domain.substr(0, 6)
> == "touch.") {\n newDo
> mainIndex = 6;\n }\n else if (domain.substr(0, 7) == "tablet.") {\n
> newDomainIndex = 7;\n
> }\n if (newDomainIndex) {\n domain =
> domain.substr(newDomainIndex);\n }\n newLocat
> ion = "https://"; + domain +
> "/uas/login?trk=sentinel_org_block&session_redirect=" + encodeURICompon
> ent(window.location)\n }\n window.location.href =
> newLocation;\n}\n</script>\n</head></html>\n'
>
> I dont need to authenticate to view public profile. What am I doing wrong?
>
> --
> You received this message because you are subscribed to the Google Groups
> "scrapy-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/scrapy-users.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
