Re: sqlite3 scrapy

2016-07-07 Thread Thomas Kaufmann 
Am Donnerstag, 7. Juli 2016 16:56:48 UTC+2 schrieb Peter Otten:
> Thomas Kaufmann wrote:
> 
> > Am Donnerstag, 7. Juli 2016 16:05:14 UTC+2 schrieb Thomas Kaufmann:
> >> Hi,
> >> 
> >> I have a little scrapy-script: scrapy gets the links from a webpage; this
> >> works fine. Then I want to write these links in a sqlite3-table. There is
> >> no error-note. But in the database are not any records. What is the
> >> problem here? My code:
> >> 
> >>  # -*- coding: utf-8 -*-
> >>   2
> >>   3 import scrapy
> >>   4 import sqlite3
> >>   5
> >>   6 class MySpider(scrapy.Spider):
> >>   7
> >>   8 name= "frisch2"
> >>   9 allowed_domains = ["frischblau.de"]
> >>  10 start_urls  = ["http://www.frischblau.de/;,]
> >>  11
> >>  12 def parse(self, response):
> >>  13
> >>  14 for href in response.xpath("//ul/li/a/@href"):
> >>  15
> >>  16 url = response.urljoin(href.extract())
> >>  17 yield scrapy.Request(url,
> >>  callback=self.parse_dir_contents) 18
> >>  19 def parse_dir_contents(self, response):
> >>  20
> >>  21 sql  = u' '
> >>  22 conn = sqlite3.connect('frisch.db')
> >>  23 cur  = conn.cursor()
> >>  24 counter = 3;
> >>  25
> >>  26 try:
> >>  27 cur.execute("SELECT SQLITE_VERSION()")
> >>  28 print "-> SQLite version: %s" % cur.fetchone()
> >>  29
> >>  30 item = response.xpath('//a/@href').extract()
> >>  31 for el in item:
> >>  32
> >>  33 #print("--> ", type(el))
> >>  34 sql = "INSERT INTO Frisch VALUES(" + unicode(counter)
> >>  + ", " + "'" + el.strip() + "');" 35
> >>  36 print (sql)
> >>  37 cur.execute(sql)
> >>  38 conn.commit
> >>  39 counter = int(counter)
> >>  40 counter += 1
> >>  41
> >>  42 except sqlite3.Error as e:
> >>  43 print "Error %s:" % e.args[0]
> >>  44 conn.close()
> >>  45
> >>
> >>
> >>
> >>   
> >> 
> >> Thanks for help.
> >> 
> >> o-o
> >> 
> >> Thomas
> > 
> > Hi Chris,
> > 
> > I changed my code in the way you suggested. 
> > There is no effect in the db.
> 
> Did you replace the line
> 
> conn.commit
> 
> with
> 
> con.commit()
> 
> ? If not, can you provide the updated code?
> 
> 
> > I start this script on the commandline with:
> > 
> > tk@Hamlet:~/myscrapy/tutorial/tutorial/spiders$ scrapy crawl webbot
> > 
> > When I call the sql-statement in the db it works.
> > 
> > sqlite> INSERT INTO Frisch VALUES(4, '//:Üs.chi/hhh\n');
> > 
> > sqlite> SELECT * FROM Frisch;
> > 1|Üschi/hhh
> > 2|Üschi/hhh\n
> > 3|//:Üs.chi/hhh\n
> > 4|//:Üs.chi/hhh\n
> > 
> > 
> > 
> > 
> > 
> > 
> >

Thanx a lot Peter. IT WORKS!!!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: sqlite3 scrapy

2016-07-07 Thread Thomas Kaufmann 
Am Donnerstag, 7. Juli 2016 16:52:12 UTC+2 schrieb Chris Angelico:
> On Fri, Jul 8, 2016 at 12:42 AM, Thomas Kaufmann <tok...@gmail.com> wrote:
> > I changed my code in the way you suggested. There is no effect in the db. I 
> > start this script on the commandline with:
> >
> > tk@Hamlet:~/myscrapy/tutorial/tutorial/spiders$ scrapy crawl webbot
> 
> You have some print calls in your code. Are they happening? Do you see
> anything on your console? If not, you probably don't have a database
> issue, you have an invocation issue. If you do, please post the output
> along with the latest code - it might be revealing.
> 
> ChrisA

Hi Chris,

here is my table:

sqlite> CREATE TABLE Frisch(id INTEGER PRIMARY KEY NOT NULL, line TEXT NOT 
NULL);

And here is the output from my script on the console:

2016-07-07 16:59:13 [scrapy] DEBUG: Crawled (200) http://www.frischblau.de/> (referer: None)
2016-07-07 16:59:14 [scrapy] DEBUG: Crawled (200) http://www.frischblau.de/#top> (referer: http://www.frischblau.de/)
-> SQLite version: 3.8.2
INSERT INTO Frisch VALUES (?, ?) (2, u'#top')
INSERT INTO Frisch VALUES (?, ?) (3, u'mailto:i...@frischblau.de')
INSERT INTO Frisch VALUES (?, ?) (4, u'http://www.zukunftscharta.de')
INSERT INTO Frisch VALUES (?, ?) (5, 
u'https://www.microsoft.com/de-de/WindowsForBusiness/End-of-IE-support')
INSERT INTO Frisch VALUES (?, ?) (6, 
u'https://contao.org/de/news/screenguide28-platz-1.html')
INSERT INTO Frisch VALUES (?, ?) (7, u'http://www.ifmo.de')
INSERT INTO Frisch VALUES (?, ?) (8, u'http://www.werksdesign.de')

...







-- 
https://mail.python.org/mailman/listinfo/python-list

Re: sqlite3 scrapy

2016-07-07 Thread Thomas Kaufmann 
Am Donnerstag, 7. Juli 2016 16:05:14 UTC+2 schrieb Thomas Kaufmann:
> Hi,
> 
> I have a little scrapy-script: scrapy gets the links from a webpage; this 
> works fine. Then I want to write these links in a sqlite3-table. There is no 
> error-note. But in the database are not any records. What is the problem 
> here? My code:
> 
>  # -*- coding: utf-8 -*-
>   2 
>   3 import scrapy
>   4 import sqlite3
>   5 
>   6 class MySpider(scrapy.Spider):
>   7
>   8 name= "frisch2"
>   9 allowed_domains = ["frischblau.de"]
>  10 start_urls  = ["http://www.frischblau.de/;,]
>  11 
>  12 def parse(self, response):
>  13 
>  14 for href in response.xpath("//ul/li/a/@href"):
>  15 
>  16 url = response.urljoin(href.extract())
>  17 yield scrapy.Request(url, callback=self.parse_dir_contents)
>  18 
>  19 def parse_dir_contents(self, response):
>  20 
>  21 sql  = u' '
>  22 conn = sqlite3.connect('frisch.db')
>  23 cur  = conn.cursor()
>  24 counter = 3;
>  25 
>  26 try:
>  27 cur.execute("SELECT SQLITE_VERSION()")
>  28 print "-> SQLite version: %s" % cur.fetchone()
>  29 
>  30 item = response.xpath('//a/@href').extract()
>  31 for el in item:
>  32 
>  33 #print("--> ", type(el))
>  34 sql = "INSERT INTO Frisch VALUES(" + unicode(counter) + 
> ", " + "'" + el.strip() + "');"
>  35 
>  36 print (sql)
>  37 cur.execute(sql)
>  38 conn.commit
>  39 counter = int(counter)
>  40 counter += 1
>  41 
>  42 except sqlite3.Error as e:
>  43 print "Error %s:" % e.args[0]
>  44 conn.close()
>  45 
>   
>   
>   
>  
> 
> Thanks for help.
> 
> o-o
> 
> Thomas

Hi Chris,

I changed my code in the way you suggested. There is no effect in the db. I 
start this script on the commandline with: 

tk@Hamlet:~/myscrapy/tutorial/tutorial/spiders$ scrapy crawl webbot

When I call the sql-statement in the db it works.

sqlite> INSERT INTO Frisch VALUES(4, '//:Üs.chi/hhh\n');

sqlite> SELECT * FROM Frisch;
1|Üschi/hhh
2|Üschi/hhh\n
3|//:Üs.chi/hhh\n
4|//:Üs.chi/hhh\n







-- 
https://mail.python.org/mailman/listinfo/python-list

Re: newbie: write content in a file (server-side)

2012-07-30 Thread Thomas Kaufmann 
Am Sonntag, 29. Juli 2012 17:16:11 UTC+2 schrieb Peter Otten:
 Thomas Kaufmann wrote:
 
 
 
  I send from a client file content to my server (as bytes). So far so good.
 
  The server receives this content complete. Ok. Then I want to write this
 
  content to a new file. It works too. But in the new file are only the
 
  first part of the whole content.
 
  
 
  What's the problem.
 
 
 
  Here's my server code:
 
 
 
  while True:
 
  bytes = self.request.recv(4096)
 
  if bytes:
 
  s  = bytes.decode(utf8)
 
  print(s)
 
  li = s.split(~)
 
  with open(li[0], 'w') as fp:
 
  fp.write(li[1])
 
 
 
 - Do you ever want to leave the loop?
 
 
 
 - You calculate a new filename on every iteration of the while loop -- 
 
 probably not what you intended to do.
 
 
 
 - The w argument tells Python to overwrite the file if it exists. You 
 
 either need to keep the file open (move the with... out of the loop) or open 
 
 it with a.
 
 
 
 - You may not receive the complete file name on the first iteration of the 
 
 while loop.
 
 
 
 - The bytes buffer can contain incomplete characters, e. g.:
 
 
 
  data = b\xc3\xa4
 
  data.decode(utf-8)
 
 'ä'
 
  data[:1].decode(utf-8)
 
 Traceback (most recent call last):
 
   File stdin, line 1, in module
 
 UnicodeDecodeError: 'utf8' codec can't decode byte 0xc3 in position 0: 
 
 unexpected end of data


Thanks Peter. It helps;-).
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: newbie: write content in a file (server-side)

2012-07-30 Thread Thomas Kaufmann 
Am Sonntag, 29. Juli 2012 16:16:01 UTC+2 schrieb Thomas Kaufmann:
 Hi,
 
 
 
 I send from a client file content to my server (as bytes). So far so good.
 
 The server receives this content complete. Ok. Then I want to write this 
 content to a new file. It works too. But in the new file are only the first 
 part of the whole content.
 
 
 
 What's the problem. 
 
 
 
 o-o
 
 
 
 Thomas
 
 
 
 Here's my server code:
 
 
 
 
 
 
 
 import socketserver
 
 
 
 class MyTCPServer(socketserver.BaseRequestHandler):
 
 
 
 def handle(self):
 
 
 
 s  = '' 
 
 li = []
 
 addr = self.client_address[0]
 
 print([{}] Connected! .format(addr))
 
 while True:
 
 
 
 bytes = self.request.recv(4096)
 
 if bytes:
 
 s  = bytes.decode(utf8)
 
 print(s)
 
 li = s.split(~)
 
 with open(li[0], 'w') as fp:
 
 fp.write(li[1])
 
 
 
 #... main ..
 
 
 
 if __name__ == __main__:
 
 
 
 server = socketserver.ThreadingTCPServer((, 12345), MyTCPServer)
 
 server.serve_forever()


Thanks a lot. It helps.
-- 
http://mail.python.org/mailman/listinfo/python-list

newbie: write content in a file (server-side)

2012-07-29 Thread Thomas Kaufmann 
Hi,

I send from a client file content to my server (as bytes). So far so good.
The server receives this content complete. Ok. Then I want to write this 
content to a new file. It works too. But in the new file are only the first 
part of the whole content.

What's the problem. 

o-o

Thomas

Here's my server code:



import socketserver

class MyTCPServer(socketserver.BaseRequestHandler):

def handle(self):

s  = '' 
li = []
addr = self.client_address[0]
print([{}] Connected! .format(addr))
while True:

bytes = self.request.recv(4096)
if bytes:
s  = bytes.decode(utf8)
print(s)
li = s.split(~)
with open(li[0], 'w') as fp:
fp.write(li[1])

#... main ..

if __name__ == __main__:

server = socketserver.ThreadingTCPServer((, 12345), MyTCPServer)
server.serve_forever()
-- 
http://mail.python.org/mailman/listinfo/python-list