subject:"Re\: Django database problem"

Re: Django database problem

2018-08-27 Thread Mikko Meronen

Yes it seems that I can influence to the 'jumps' with my management
command. First my bot went through tens of items per loop, however when I
changed the bot to check only 2 items per loop, the 'jumps' got smaller. So
even though the table rows do not accrue (nothing is added) when no new
items are found in a loop, the ID numbers still accrue behind the scenes in
every loop.

So I just gotta figure out if there are any other way than INSERT OR IGNORE
command, or just decrease the sleep time of the bot and keep the checked
item numbers low per a loop.

I actually tried ROLLBACK and some other commands as well, but they killed
the loop.

Thanks for the help ^^

-Mikko


ma 27. elok. 2018 klo 15.01 Andréas Kühne (andreas.ku...@hypercode.se)
kirjoitti:

> I have looked at this several times myself. :-)
>
> Regards,
>
> Andréas
>
>
> Den mån 27 aug. 2018 kl 13:33 skrev Jason :
>
>> oh, good catch.  I didn't think of that, but you're right.
>>
>>
>> https://stackoverflow.com/questions/449346/mysql-auto-increment-does-not-rollback
>>
>> On Monday, August 27, 2018 at 3:50:28 AM UTC-4, Andréas Kühne wrote:
>>>
>>> The management command could be wrapped within a transaction, this would
>>> then in the database create a lot of models, but if the transaction fails -
>>> the rollback will not reset the database id.
>>>
>>> Regards,
>>>
>>> Andréas
>>>
>>>
>>> Den sön 26 aug. 2018 kl 12:29 skrev Mikko Meronen >> >:
>>>
 Hi,

 Thanks for advice ^^

 I have one new concern. Is there a limit for for Django model IDs?

 Now I have automated the data insert with python code loop, but the
 ID-number is not consecutive. It jumps for example from ...56 to 571... and
 from ...587 to 3763... and so forth.

 Do you know why it is doing that?

 I'm not using django custom command (if that makes the difference or
 can be used for my case, I will get familiar with it a bit later), I'm just
 inserting the data to django's database with separate python code.

 -Mikko

 ma 20. elok. 2018 klo 20.41 Jani Tiainen (red...@gmail.com) kirjoitti:

> Hi,
>
> Instead of doing everything by hand I would recommend doing a custom
> management command in Django.
>
> Because management commands live inside Django ecosystem you get full
> benefits of Django. Migrations to setup database tables from model
> definitions. Models and ORM to interact with database. (Model)forms to
> validate your incoming data etc.
>
> Also I really suggest that you do the official tutorial from the docs
> to get hang of basic django concepts.
>
> ma 20. elok. 2018 klo 19.59 Mikko Meronen 
> kirjoitti:
>
>> Hi and thanks for your help.
>>
>> I have tried postgress and now I have a problem using it. Here's my
>> code following an error. I appreciate any help.
>>
>> def do():
>> x = 1
>> while x == 1:
>> create_table()
>> print('finding data')
>> find_data()
>> time.sleep(120)
>>
>> def create_table():
>> c.execute('CREATE TABLE IF NOT EXISTS testi(url TEXT, title TEXT,
>> published TEXT, UNIQUE(url))')
>>
>> def find_data():
>> r = requests.get(homepage)
>> soup = BeautifulSoup(r.text, 'html.parser')
>> results = soup.find_all('data')
>> for result in results:
>> title = result.find('h1').text
>> url = result.find('a')['href']
>> published = result.find('time')['datetime']
>>
>>
>> c.execute("INSERT OR IGNORE INTO testi (url, title,
>> published) VALUES (%s, %s, %s)",
>>   ('url', 'title', 'published'))
>> connection.commit()
>>
>>
>> do()
>>
>> ERROR:
>>
>> Traceback (most recent call last):
>>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line
>> 43, in 
>> do()
>>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line
>> 21, in do
>> find_data()
>>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line
>> 39, in find_data
>> ('url', 'title', 'published'))
>> *psycopg2.ProgrammingError: syntax error at or near "OR"*
>> *LINE 1: INSERT OR IGNORE INTO testi (url, title, published) VALUES
>> (...*
>>
>>
>>
>> -Mikko
>>
>>
>> 2018-08-17 19:09 GMT+03:00 Mikhailo Keda :
>>
>>> You could generate models from sqlite database -
>>> https://docs.djangoproject.com/en/2.1/howto/legacy-databases/
>>> Than switch to PostgresQL
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to

Re: Django database problem

2018-08-27 Thread Andréas Kühne

I have looked at this several times myself. :-)

Regards,

Andréas


Den mån 27 aug. 2018 kl 13:33 skrev Jason :

> oh, good catch.  I didn't think of that, but you're right.
>
>
> https://stackoverflow.com/questions/449346/mysql-auto-increment-does-not-rollback
>
> On Monday, August 27, 2018 at 3:50:28 AM UTC-4, Andréas Kühne wrote:
>>
>> The management command could be wrapped within a transaction, this would
>> then in the database create a lot of models, but if the transaction fails -
>> the rollback will not reset the database id.
>>
>> Regards,
>>
>> Andréas
>>
>>
>> Den sön 26 aug. 2018 kl 12:29 skrev Mikko Meronen > >:
>>
>>> Hi,
>>>
>>> Thanks for advice ^^
>>>
>>> I have one new concern. Is there a limit for for Django model IDs?
>>>
>>> Now I have automated the data insert with python code loop, but the
>>> ID-number is not consecutive. It jumps for example from ...56 to 571... and
>>> from ...587 to 3763... and so forth.
>>>
>>> Do you know why it is doing that?
>>>
>>> I'm not using django custom command (if that makes the difference or can
>>> be used for my case, I will get familiar with it a bit later), I'm just
>>> inserting the data to django's database with separate python code.
>>>
>>> -Mikko
>>>
>>> ma 20. elok. 2018 klo 20.41 Jani Tiainen (red...@gmail.com) kirjoitti:
>>>
 Hi,

 Instead of doing everything by hand I would recommend doing a custom
 management command in Django.

 Because management commands live inside Django ecosystem you get full
 benefits of Django. Migrations to setup database tables from model
 definitions. Models and ORM to interact with database. (Model)forms to
 validate your incoming data etc.

 Also I really suggest that you do the official tutorial from the docs
 to get hang of basic django concepts.

 ma 20. elok. 2018 klo 19.59 Mikko Meronen 
 kirjoitti:

> Hi and thanks for your help.
>
> I have tried postgress and now I have a problem using it. Here's my
> code following an error. I appreciate any help.
>
> def do():
> x = 1
> while x == 1:
> create_table()
> print('finding data')
> find_data()
> time.sleep(120)
>
> def create_table():
> c.execute('CREATE TABLE IF NOT EXISTS testi(url TEXT, title TEXT,
> published TEXT, UNIQUE(url))')
>
> def find_data():
> r = requests.get(homepage)
> soup = BeautifulSoup(r.text, 'html.parser')
> results = soup.find_all('data')
> for result in results:
> title = result.find('h1').text
> url = result.find('a')['href']
> published = result.find('time')['datetime']
>
>
> c.execute("INSERT OR IGNORE INTO testi (url, title,
> published) VALUES (%s, %s, %s)",
>   ('url', 'title', 'published'))
> connection.commit()
>
>
> do()
>
> ERROR:
>
> Traceback (most recent call last):
>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line
> 43, in 
> do()
>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line
> 21, in do
> find_data()
>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line
> 39, in find_data
> ('url', 'title', 'published'))
> *psycopg2.ProgrammingError: syntax error at or near "OR"*
> *LINE 1: INSERT OR IGNORE INTO testi (url, title, published) VALUES
> (...*
>
>
>
> -Mikko
>
>
> 2018-08-17 19:09 GMT+03:00 Mikhailo Keda :
>
>> You could generate models from sqlite database -
>> https://docs.djangoproject.com/en/2.1/howto/legacy-databases/
>> Than switch to PostgresQL
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to django-users...@googlegroups.com.
>> To post to this group, send email to django...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/52f5c754-7310-4baa-8008-edbcea15eafb%40googlegroups.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
>

Re: Django database problem

2018-08-27 Thread Jason

oh, good catch.  I didn't think of that, but you're right.

https://stackoverflow.com/questions/449346/mysql-auto-increment-does-not-rollback

On Monday, August 27, 2018 at 3:50:28 AM UTC-4, Andréas Kühne wrote:
>
> The management command could be wrapped within a transaction, this would 
> then in the database create a lot of models, but if the transaction fails - 
> the rollback will not reset the database id.
>
> Regards,
>
> Andréas
>
>
> Den sön 26 aug. 2018 kl 12:29 skrev Mikko Meronen  >:
>
>> Hi,
>>
>> Thanks for advice ^^
>>
>> I have one new concern. Is there a limit for for Django model IDs?
>>
>> Now I have automated the data insert with python code loop, but the 
>> ID-number is not consecutive. It jumps for example from ...56 to 571... and 
>> from ...587 to 3763... and so forth.
>>
>> Do you know why it is doing that?
>>
>> I'm not using django custom command (if that makes the difference or can 
>> be used for my case, I will get familiar with it a bit later), I'm just 
>> inserting the data to django's database with separate python code.
>>
>> -Mikko 
>>
>> ma 20. elok. 2018 klo 20.41 Jani Tiainen (red...@gmail.com ) 
>> kirjoitti:
>>
>>> Hi,
>>>
>>> Instead of doing everything by hand I would recommend doing a custom 
>>> management command in Django.
>>>
>>> Because management commands live inside Django ecosystem you get full 
>>> benefits of Django. Migrations to setup database tables from model 
>>> definitions. Models and ORM to interact with database. (Model)forms to 
>>> validate your incoming data etc.
>>>
>>> Also I really suggest that you do the official tutorial from the docs to 
>>> get hang of basic django concepts.
>>>
>>> ma 20. elok. 2018 klo 19.59 Mikko Meronen >> > kirjoitti:
>>>
 Hi and thanks for your help.

 I have tried postgress and now I have a problem using it. Here's my 
 code following an error. I appreciate any help.

 def do():
 x = 1
 while x == 1:
 create_table()
 print('finding data')
 find_data()
 time.sleep(120)
 
 def create_table():
 c.execute('CREATE TABLE IF NOT EXISTS testi(url TEXT, title TEXT, 
 published TEXT, UNIQUE(url))')

 def find_data():
 r = requests.get(homepage)
 soup = BeautifulSoup(r.text, 'html.parser')
 results = soup.find_all('data')
 for result in results:
 title = result.find('h1').text
 url = result.find('a')['href']
 published = result.find('time')['datetime']
 

 c.execute("INSERT OR IGNORE INTO testi (url, title, 
 published) VALUES (%s, %s, %s)",
   ('url', 'title', 'published'))
 connection.commit()
 

 do()

 ERROR:

 Traceback (most recent call last):
   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 43, 
 in 
 do()
   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 21, 
 in do
 find_data()
   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 39, 
 in find_data
 ('url', 'title', 'published'))
 *psycopg2.ProgrammingError: syntax error at or near "OR"*
 *LINE 1: INSERT OR IGNORE INTO testi (url, title, published) VALUES 
 (...*



 -Mikko


 2018-08-17 19:09 GMT+03:00 Mikhailo Keda >>> >:

> You could generate models from sqlite database - 
> https://docs.djangoproject.com/en/2.1/howto/legacy-databases/
> Than switch to PostgresQL
>
> -- 
> You received this message because you are subscribed to the Google 
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to django-users...@googlegroups.com .
> To post to this group, send email to django...@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/52f5c754-7310-4baa-8008-edbcea15eafb%40googlegroups.com
>  
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

 -- 
 You received this message because you are subscribed to the Google 
 Groups "Django users" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to django-users...@googlegroups.com .
 To post to this group, send email to django...@googlegroups.com 
 .
 Visit this group at https://groups.google.com/group/django-users.
 To view this discussion on the web visit

Re: Django database problem

2018-08-27 Thread Andréas Kühne

The management command could be wrapped within a transaction, this would
then in the database create a lot of models, but if the transaction fails -
the rollback will not reset the database id.

Regards,

Andréas


Den sön 26 aug. 2018 kl 12:29 skrev Mikko Meronen <
mikkovillemero...@gmail.com>:

> Hi,
>
> Thanks for advice ^^
>
> I have one new concern. Is there a limit for for Django model IDs?
>
> Now I have automated the data insert with python code loop, but the
> ID-number is not consecutive. It jumps for example from ...56 to 571... and
> from ...587 to 3763... and so forth.
>
> Do you know why it is doing that?
>
> I'm not using django custom command (if that makes the difference or can
> be used for my case, I will get familiar with it a bit later), I'm just
> inserting the data to django's database with separate python code.
>
> -Mikko
>
> ma 20. elok. 2018 klo 20.41 Jani Tiainen (rede...@gmail.com) kirjoitti:
>
>> Hi,
>>
>> Instead of doing everything by hand I would recommend doing a custom
>> management command in Django.
>>
>> Because management commands live inside Django ecosystem you get full
>> benefits of Django. Migrations to setup database tables from model
>> definitions. Models and ORM to interact with database. (Model)forms to
>> validate your incoming data etc.
>>
>> Also I really suggest that you do the official tutorial from the docs to
>> get hang of basic django concepts.
>>
>> ma 20. elok. 2018 klo 19.59 Mikko Meronen 
>> kirjoitti:
>>
>>> Hi and thanks for your help.
>>>
>>> I have tried postgress and now I have a problem using it. Here's my code
>>> following an error. I appreciate any help.
>>>
>>> def do():
>>> x = 1
>>> while x == 1:
>>> create_table()
>>> print('finding data')
>>> find_data()
>>> time.sleep(120)
>>>
>>> def create_table():
>>> c.execute('CREATE TABLE IF NOT EXISTS testi(url TEXT, title TEXT,
>>> published TEXT, UNIQUE(url))')
>>>
>>> def find_data():
>>> r = requests.get(homepage)
>>> soup = BeautifulSoup(r.text, 'html.parser')
>>> results = soup.find_all('data')
>>> for result in results:
>>> title = result.find('h1').text
>>> url = result.find('a')['href']
>>> published = result.find('time')['datetime']
>>>
>>>
>>> c.execute("INSERT OR IGNORE INTO testi (url, title,
>>> published) VALUES (%s, %s, %s)",
>>>   ('url', 'title', 'published'))
>>> connection.commit()
>>>
>>>
>>> do()
>>>
>>> ERROR:
>>>
>>> Traceback (most recent call last):
>>>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 43,
>>> in 
>>> do()
>>>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 21,
>>> in do
>>> find_data()
>>>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 39,
>>> in find_data
>>> ('url', 'title', 'published'))
>>> *psycopg2.ProgrammingError: syntax error at or near "OR"*
>>> *LINE 1: INSERT OR IGNORE INTO testi (url, title, published) VALUES (...*
>>>
>>>
>>>
>>> -Mikko
>>>
>>>
>>> 2018-08-17 19:09 GMT+03:00 Mikhailo Keda :
>>>
 You could generate models from sqlite database -
 https://docs.djangoproject.com/en/2.1/howto/legacy-databases/
 Than switch to PostgresQL

 --
 You received this message because you are subscribed to the Google
 Groups "Django users" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to django-users+unsubscr...@googlegroups.com.
 To post to this group, send email to django-users@googlegroups.com.
 Visit this group at https://groups.google.com/group/django-users.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/django-users/52f5c754-7310-4baa-8008-edbcea15eafb%40googlegroups.com
 
 .

 For more options, visit https://groups.google.com/d/optout.

>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users+unsubscr...@googlegroups.com.
>>> To post to this group, send email to django-users@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-users.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-users/CAGD0jjJof7v9nMPF593DJiQHRv6z_%3DmQ_ZtQWob47PBT1gvL9Q%40mail.gmail.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django

Re: Django database problem

2018-08-26 Thread Jason

yeah, there is a limit, and its set by the data type you use for the id 
field.  by default, id is set to IntegerField 

 which 
has a maximum value of 2147483647 

sounds like the issue with the ids jumping around like that is with your 
management command, not django or the db.  

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/2716221c-ccfa-4123-a2f4-f77724ee497a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django database problem

2018-08-26 Thread Mikko Meronen

Hi,

Thanks for advice ^^

I have one new concern. Is there a limit for for Django model IDs?

Now I have automated the data insert with python code loop, but the
ID-number is not consecutive. It jumps for example from ...56 to 571... and
from ...587 to 3763... and so forth.

Do you know why it is doing that?

I'm not using django custom command (if that makes the difference or can be
used for my case, I will get familiar with it a bit later), I'm just
inserting the data to django's database with separate python code.

-Mikko

ma 20. elok. 2018 klo 20.41 Jani Tiainen (rede...@gmail.com) kirjoitti:

> Hi,
>
> Instead of doing everything by hand I would recommend doing a custom
> management command in Django.
>
> Because management commands live inside Django ecosystem you get full
> benefits of Django. Migrations to setup database tables from model
> definitions. Models and ORM to interact with database. (Model)forms to
> validate your incoming data etc.
>
> Also I really suggest that you do the official tutorial from the docs to
> get hang of basic django concepts.
>
> ma 20. elok. 2018 klo 19.59 Mikko Meronen 
> kirjoitti:
>
>> Hi and thanks for your help.
>>
>> I have tried postgress and now I have a problem using it. Here's my code
>> following an error. I appreciate any help.
>>
>> def do():
>> x = 1
>> while x == 1:
>> create_table()
>> print('finding data')
>> find_data()
>> time.sleep(120)
>>
>> def create_table():
>> c.execute('CREATE TABLE IF NOT EXISTS testi(url TEXT, title TEXT,
>> published TEXT, UNIQUE(url))')
>>
>> def find_data():
>> r = requests.get(homepage)
>> soup = BeautifulSoup(r.text, 'html.parser')
>> results = soup.find_all('data')
>> for result in results:
>> title = result.find('h1').text
>> url = result.find('a')['href']
>> published = result.find('time')['datetime']
>>
>>
>> c.execute("INSERT OR IGNORE INTO testi (url, title,
>> published) VALUES (%s, %s, %s)",
>>   ('url', 'title', 'published'))
>> connection.commit()
>>
>>
>> do()
>>
>> ERROR:
>>
>> Traceback (most recent call last):
>>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 43,
>> in 
>> do()
>>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 21,
>> in do
>> find_data()
>>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 39,
>> in find_data
>> ('url', 'title', 'published'))
>> *psycopg2.ProgrammingError: syntax error at or near "OR"*
>> *LINE 1: INSERT OR IGNORE INTO testi (url, title, published) VALUES (...*
>>
>>
>>
>> -Mikko
>>
>>
>> 2018-08-17 19:09 GMT+03:00 Mikhailo Keda :
>>
>>> You could generate models from sqlite database -
>>> https://docs.djangoproject.com/en/2.1/howto/legacy-databases/
>>> Than switch to PostgresQL
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users+unsubscr...@googlegroups.com.
>>> To post to this group, send email to django-users@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-users.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-users/52f5c754-7310-4baa-8008-edbcea15eafb%40googlegroups.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-users@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/CAGD0jjJof7v9nMPF593DJiQHRv6z_%3DmQ_ZtQWob47PBT1gvL9Q%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
>

Re: Django database problem

2018-08-20 Thread Jani Tiainen

Hi,

Instead of doing everything by hand I would recommend doing a custom
management command in Django.

Because management commands live inside Django ecosystem you get full
benefits of Django. Migrations to setup database tables from model
definitions. Models and ORM to interact with database. (Model)forms to
validate your incoming data etc.

Also I really suggest that you do the official tutorial from the docs to
get hang of basic django concepts.

ma 20. elok. 2018 klo 19.59 Mikko Meronen 
kirjoitti:

> Hi and thanks for your help.
>
> I have tried postgress and now I have a problem using it. Here's my code
> following an error. I appreciate any help.
>
> def do():
> x = 1
> while x == 1:
> create_table()
> print('finding data')
> find_data()
> time.sleep(120)
>
> def create_table():
> c.execute('CREATE TABLE IF NOT EXISTS testi(url TEXT, title TEXT,
> published TEXT, UNIQUE(url))')
>
> def find_data():
> r = requests.get(homepage)
> soup = BeautifulSoup(r.text, 'html.parser')
> results = soup.find_all('data')
> for result in results:
> title = result.find('h1').text
> url = result.find('a')['href']
> published = result.find('time')['datetime']
>
>
> c.execute("INSERT OR IGNORE INTO testi (url, title,
> published) VALUES (%s, %s, %s)",
>   ('url', 'title', 'published'))
> connection.commit()
>
>
> do()
>
> ERROR:
>
> Traceback (most recent call last):
>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 43, in
> 
> do()
>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 21, in
> do
> find_data()
>   File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 39, in
> find_data
> ('url', 'title', 'published'))
> *psycopg2.ProgrammingError: syntax error at or near "OR"*
> *LINE 1: INSERT OR IGNORE INTO testi (url, title, published) VALUES (...*
>
>
>
> -Mikko
>
>
> 2018-08-17 19:09 GMT+03:00 Mikhailo Keda :
>
>> You could generate models from sqlite database -
>> https://docs.djangoproject.com/en/2.1/howto/legacy-databases/
>> Than switch to PostgresQL
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-users@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/52f5c754-7310-4baa-8008-edbcea15eafb%40googlegroups.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAGD0jjJof7v9nMPF593DJiQHRv6z_%3DmQ_ZtQWob47PBT1gvL9Q%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHn91oex8cAE1yFshSABt9QZpdiiRVtALtLx0AWDNU84w0%2Bqvw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django database problem

2018-08-20 Thread Mikko Meronen

Hi and thanks for your help.

I have tried postgress and now I have a problem using it. Here's my code
following an error. I appreciate any help.

def do():
x = 1
while x == 1:
create_table()
print('finding data')
find_data()
time.sleep(120)

def create_table():
c.execute('CREATE TABLE IF NOT EXISTS testi(url TEXT, title TEXT,
published TEXT, UNIQUE(url))')

def find_data():
r = requests.get(homepage)
soup = BeautifulSoup(r.text, 'html.parser')
results = soup.find_all('data')
for result in results:
title = result.find('h1').text
url = result.find('a')['href']
published = result.find('time')['datetime']


c.execute("INSERT OR IGNORE INTO testi (url, title,
published) VALUES (%s, %s, %s)",
  ('url', 'title', 'published'))
connection.commit()


do()

ERROR:

Traceback (most recent call last):
  File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 43, in

do()
  File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 21, in
do
find_data()
  File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 39, in
find_data
('url', 'title', 'published'))
*psycopg2.ProgrammingError: syntax error at or near "OR"*
*LINE 1: INSERT OR IGNORE INTO testi (url, title, published) VALUES (...*



-Mikko


2018-08-17 19:09 GMT+03:00 Mikhailo Keda :

> You could generate models from sqlite database -
> https://docs.djangoproject.com/en/2.1/howto/legacy-databases/
> Than switch to PostgresQL
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/52f5c754-7310-4baa-8008-edbcea15eafb%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAGD0jjJof7v9nMPF593DJiQHRv6z_%3DmQ_ZtQWob47PBT1gvL9Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django database problem

2018-08-17 Thread Mikhailo Keda

You could generate models from sqlite database 
- https://docs.djangoproject.com/en/2.1/howto/legacy-databases/
Than switch to PostgresQL

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/52f5c754-7310-4baa-8008-edbcea15eafb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django database problem

2018-08-17 Thread Okware Aldo

Mikko,

Check out this link below, it will help you understand how to connect you
Django project to any DB manager.

https://docs.djangoproject.com/en/2.1/intro/tutorial02/

On Fri, Aug 17, 2018 at 6:55 PM Mikko Meronen 
wrote:

> Hi,
>
> I'm quite new with django and python and I wish someone could help me.
>
> I'm building a webpage where my python program/bot collects data from
> webpages.
>
> Is there a way that I can use my python program to store the data to
> django's database?
>
> At the moment when I run my program, I collect the data to separate sqlite
> database, but I don't know how to collect/connect it to django's sqlite
> database (or models).
>
> - Mikko
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAGD0jjLbzK1XVVSUTD4Z66p_RUoK0pBW%2BrEYtVM_HwAd9d%3DwvQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAMEZma_gWB4gysi%2BBC559jNFiJxvge8QvWzqgDs7mqB2D34QYw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django database problem

2018-08-17 Thread Anirudh Jain

You will have to create a database on your system first (it is better to
use mysql instead of sqlite) and then run commands :-

1. python manage.py makemigrations
2. python manage.py migrate

Also you will have to create forms (forms.ModelForm or forms.Forms) for
taking input.

On Fri, 17 Aug 2018, 21:26 Mikko Meronen, 
wrote:

> Hi,
>
> I'm quite new with django and python and I wish someone could help me.
>
> I'm building a webpage where my python program/bot collects data from
> webpages.
>
> Is there a way that I can use my python program to store the data to
> django's database?
>
> At the moment when I run my program, I collect the data to separate sqlite
> database, but I don't know how to collect/connect it to django's sqlite
> database (or models).
>
> - Mikko
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAGD0jjLbzK1XVVSUTD4Z66p_RUoK0pBW%2BrEYtVM_HwAd9d%3DwvQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAC3mK7dgX__66cbM53OZF%3DSYCHfVQRTe9ghDWWD_4h5cesOgCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django database problem

Re: Django database problem

Re: Django database problem

Re: Django database problem

Re: Django database problem

Re: Django database problem

Re: Django database problem

Re: Django database problem

Re: Django database problem

Re: Django database problem

Re: Django database problem

11 matches

Site Navigation

Mail list logo

Footer information