Re: Using data migration to create initial users errors

[James Schneider]

On Sep 7, 2017 10:36 AM, "Tim Huang"  wrote:

Hi Melvyn,


I am lack of knowledge how to use model manager. Would you mind give a
quick example? Thanks

- timh


There's a fairly substantial doc page for model managers:

https://docs.djangoproject.com/en/1.11/topics/db/managers/

-James

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2Be%2BciW56%3DDBb4gQKFKfPM%2BbKwsQXPX5WfHLRohgukDgf11spQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to pre-load model data?

[James Schneider]

On Sep 9, 2017 8:59 AM, "Derek"  wrote:

"If you dump data directly in to models or the database, then you risk
dumping in data that hasn't been evaluated by your application which may
produce subtle bugs of the worst kind where things work in dev and break in
prod because your input channel is different."

If this is happening its because you have poor validation on your model
clean(s). You should NEVER just rely on a form to check data for you (at
least beyond very basic sanitation); that logic properly belongs with the
model.  We import direct from spreadsheets to models (note - NOT direct to
the database) because we make use of such logic.


That's not what I'm saying. Validation at the model level should apply to
all instances of the model in any situation. Validation at the form level
can be much more context specific. The two levels of validation are not
mutually exclusive, rather they are complementary.

I'm not saying that there shouldn't be model validation. Quite the
opposite, there absolutely should be. I'm saying that the import of the
data should follow the same workflow as an end-user would enter it to
ensure the data appears correct in the context that it is being entered
where possible.

I'm simple cases, a model is only implemented in a single context (in only
one spot in the application and only via one form). Model and form
validation would usually be the same. In advanced cases, the same model can
be referenced via multiple pages, and each page may place a different set
of restrictions on what data may be entered or how it is formatted. This is
typical when a model is related to several other models.

Hypothetically, if you have a Widget model, it may have a long list of
available colors to choose from in a choices= list. The model validation
would be responsible for ensuring that the color selected is one of the
available colors in the entire list of colors available.

However, Widgets may only be available in red and blue when paired with a
FooBar model. The form for FooBarWidgets would then limit the color choices
to just red and blue, and would validate that the color submitted is one of
those. The model validation wouldn't complain, because red and blue are in
the long list of colors available.

Importing the data directly to the model could potentially mean that a
Widget paired with a FooBar can be set to green, which is invalid. The
model validation wouldn't help in this case, because green is technically a
valid color for widgets, but in the context of FooBars, it isn't.

That's why I mentioned that you need to be 150% confident in your data
generation when importing via the ORM directly to avoid "impossible" data
in your models, which is worse than missing data IMO.

-James

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2Be%2BciXetGK9EhQEFyE05g18g8x7gEESRN9vmPw_yNjicC8HbA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Supervisor FATAl Exited too quickly (process log may have details(About inet_http_server and unix_http_server)

[Mario Gudelj]

Does that Guniorn command run directly from the command line? What's
gunicorn.pod? Make sure you can start gunicorn outside of supervisor and
then stick that command into conf file.

Cheers,

M

On Sun, 10 Sep 2017 at 2:28 am, Jonathan Cheng 
wrote:

> I wrote a Django project and I use supervisor with gunicorn
>
>
> /etc/supervisor/conf.d/weather.conf
>
>
>
> [group:weather_station]
> programs=site
>
> [program:site]
> directory=$PROJECT
> command=/home/nhcc/.local/bin/gunicorn -c /$PROJECT/weather_station/
> gunicorn.conf.py -p gunicorn.pod weather_station.wsgi
> autostart=true
> autorestart=true
> stdout_logfile=/var/log/supervisor.log
> environment=my-environment-variable
> ~
>
>
>
>
> `sudo supervisorctl reread`
>
>
> `sudo supervisorctl reload`
>
>
> `sudo supervisorctl status`
>
>
> It showed up the error
>
> > weather_station:site FATAL Exited too quickly (process
> log may have details)
>
>
> So I checked out the log file :
>
> `/var/log/supervisor/supervisord.log`
>
>
> > 2017-09-08 17:15:25,000 CRIT Supervisor running as root (no user in
>> config file)
>> > 2017-09-08 17:15:25,000 WARN Included extra file
>> "/etc/supervisor/conf.d/weather.conf" during parsing
>> >2017-09-08 17:15:25,007 INFO RPC interface 'supervisor' initialized
>> >2017-09-08 17:15:25,008 CRIT Server 'inet_http_server' running without
>> any HTTP
>> > authentication checking
>> > 2017-09-08 17:15:25,008 INFO RPC interface
>> > 'supervisor' initialized
>> > 2017-09-08 17:15:25,008 CRIT Server 'unix_http_server' running without
>> any HTTP authentication checking
>> > 2017-09-08 17:15:25,008 INFO supervisord started with pid 32371
>> > 2017-09-08 17:15:26,013 INFO spawned: 'site' with pid 32447
>> > 2017-09-08 17:15:26,018 INFO exited: site (exit status 127; not
>> expected)
>> > 2017-09-08 17:15:27,022 INFO spawned: 'site' with pid 32448
>> > 2017-09-08 17:15:27,026 INFO exited: site (exit status 127; not
>> expected)
>> > 2017-09-08 17:15:29,032 INFO spawned: 'site' with pid 32449
>> > 2017-09-08 17:15:29,038 INFO exited: site (exit status 127; not
>> expected)
>> > 2017-09-08 17:15:32,043 INFO spawned: 'site' with pid 32451
>> > 2017-09-08 17:15:32,059 INFO exited: site (exit status 127; not
>> expected)
>> > 2017-09-08 17:15:33,060 INFO gave up: site entered FATAL state, too
>> > many start retries too quickly
>>
>>
>>
>
>
>
>
> According to `Included extra file`[the solution](
> https://github.com/Supervisor/supervisor/issues/272)
>
> According to `Server 'unix_http_server' running without any HTTP
> authentication checking`[the solution](
> https://github.com/Supervisor/supervisor/issues/717)
>
> I added two section [unix_http_server] and [inet_http_server]
>
>
>
>[unix_http_server]
> file=/var/run/supervisor.sock   ; (the path to the socket file)
> chmod=0766 ; socket file mode (default 0700)
> ;chown=nobody:nogroup   ; socket file uid:gid owner
> ;username=user  ; (default is no username (open server))
> ;password=123   ; (default is no password (open server))
>
> [inet_http_server] ; inet (TCP) server disabled by default
> port=*:9001; (ip_address:port specifier, *:port for all iface,
> ;username=user  ; (default is no username (open server))
> ;password=123   ; (default is no password (open server))
>
>
>
>
> This is the current  supervisor.conf
>
>
>
> [unix_http_server]
> file=/var/run/supervisor.sock   ; (the path to the socket file)
> chmod=0766 ; socket file mode (default 0700)
>
> [supervisord]
> logfile=/var/log/supervisor/supervisord.log ; (main log file;default
> $CWD/supervisord.log)
> pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default
> supervisord.pid)
> childlogdir=/var/log/supervisor; ('AUTO' child log dir,
> default $TEMP)
>
> [rpcinterface:supervisor]
> supervisor.rpcinterface_factory = supervisor.rpcinterface:
> make_main_rpcinterface
>
> [supervisorctl]
> serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL  for a
> unix socket
>
> [inet_http_server] ; inet (TCP) server disabled by default
> port=*:9001;
>
> [include]
> files = /etc/supervisor/*.conf
> files = /etc/supervisor/conf.d/*.conf
>
>
>
>
>
>
>
>
>
> Then I ran
>
>
>sudo supervisord -c /etc/supervisor/supervisord.conf
> sudo supervisorctl -c /etc/supervisor/supervisord.conf
>
>
>
>
> And I checked the log again
>
> The error is completely be same
>
> Does anyone know why??
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at 

Supervisor FATAl Exited too quickly (process log may have details(About inet_http_server and unix_http_server)

[Jonathan Cheng]

I wrote a Django project and I use supervisor with gunicorn


/etc/supervisor/conf.d/weather.conf



[group:weather_station]
programs=site

[program:site]
directory=$PROJECT
command=/home/nhcc/.local/bin/gunicorn -c /$PROJECT/weather_station/
gunicorn.conf.py -p gunicorn.pod weather_station.wsgi
autostart=true
autorestart=true
stdout_logfile=/var/log/supervisor.log
environment=my-environment-variable
~




`sudo supervisorctl reread`


`sudo supervisorctl reload`


`sudo supervisorctl status`


It showed up the error

> weather_station:site FATAL Exited too quickly (process 
log may have details)


So I checked out the log file :

`/var/log/supervisor/supervisord.log`


> 2017-09-08 17:15:25,000 CRIT Supervisor running as root (no user in 
> config file) 
> > 2017-09-08 17:15:25,000 WARN Included extra file 
> "/etc/supervisor/conf.d/weather.conf" during parsing 
> >2017-09-08 17:15:25,007 INFO RPC interface 'supervisor' initialized 
> >2017-09-08 17:15:25,008 CRIT Server 'inet_http_server' running without 
> any HTTP
> > authentication checking 
> > 2017-09-08 17:15:25,008 INFO RPC interface
> > 'supervisor' initialized 
> > 2017-09-08 17:15:25,008 CRIT Server 'unix_http_server' running without 
> any HTTP authentication checking
> > 2017-09-08 17:15:25,008 INFO supervisord started with pid 32371
> > 2017-09-08 17:15:26,013 INFO spawned: 'site' with pid 32447 
> > 2017-09-08 17:15:26,018 INFO exited: site (exit status 127; not expected)
> > 2017-09-08 17:15:27,022 INFO spawned: 'site' with pid 32448 
> > 2017-09-08 17:15:27,026 INFO exited: site (exit status 127; not expected)
> > 2017-09-08 17:15:29,032 INFO spawned: 'site' with pid 32449 
> > 2017-09-08 17:15:29,038 INFO exited: site (exit status 127; not expected)
> > 2017-09-08 17:15:32,043 INFO spawned: 'site' with pid 32451 
> > 2017-09-08 17:15:32,059 INFO exited: site (exit status 127; not expected)
> > 2017-09-08 17:15:33,060 INFO gave up: site entered FATAL state, too
> > many start retries too quickly
>   
>   
>  




According to `Included extra file`[the 
solution](https://github.com/Supervisor/supervisor/issues/272)

According to `Server 'unix_http_server' running without any HTTP 
authentication checking`[the 
solution](https://github.com/Supervisor/supervisor/issues/717)

I added two section [unix_http_server] and [inet_http_server]


 
   [unix_http_server]
file=/var/run/supervisor.sock   ; (the path to the socket file)
chmod=0766 ; socket file mode (default 0700)
;chown=nobody:nogroup   ; socket file uid:gid owner
;username=user  ; (default is no username (open server))
;password=123   ; (default is no password (open server))

[inet_http_server] ; inet (TCP) server disabled by default
port=*:9001; (ip_address:port specifier, *:port for all iface, 
;username=user  ; (default is no username (open server))
;password=123   ; (default is no password (open server))




This is the current  supervisor.conf



[unix_http_server]
file=/var/run/supervisor.sock   ; (the path to the socket file)
chmod=0766 ; socket file mode (default 0700)

[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default 
$CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default 
supervisord.pid)
childlogdir=/var/log/supervisor; ('AUTO' child log dir, 
default $TEMP)

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:
make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL  for a 
unix socket

[inet_http_server] ; inet (TCP) server disabled by default
port=*:9001;

[include]
files = /etc/supervisor/*.conf
files = /etc/supervisor/conf.d/*.conf









Then I ran

 
   sudo supervisord -c /etc/supervisor/supervisord.conf
sudo supervisorctl -c /etc/supervisor/supervisord.conf




And I checked the log again

The error is completely be same

Does anyone know why??

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/4a1e659d-93be-4420-b68e-cd04d8188961%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to pre-load model data?

[Mark Phillips]

James,

Thanks for your comments.

Unfortunately, the data is not auto generated, so FactoryBoy may not help
much. As far as validation, the data is so simple that the only validation
I have come up with is to insure the names of the metadata and metadata
values are unique. The metadata and metadata values are used to describe
documents imported into the application, so they need to be unique values.
We have defined ~200 metadata names and values so far for the production
system. The are the two models -

class MetaData(models.Model):
metadata_id = models.AutoField(primary_key = True)
name = models.CharField('metadata name', max_length=200, unique=True)
description = models.TextField('description')

def __str__(self):
return self.name

class MetaDataValue(models.Model):
metadata_id = models.ForeignKey(MetaData, on_delete=models.CASCADE,)
value = models.CharField('value', max_length=200, unique=True)

def __str__(self):
return self.value

I could use FactoryBoy to generate a test data for these two models, but
what I really need is a simple way to fill out these two tables in the
database with production data. The same data could be used for testing
purposes.

I have been playing with django-import-export. It is a great solution
except it does not support InlineModelAdmin objects and material design (ie
the import/export buttons don't appear in the admin pages). So I will look
for another django import/exporter.

A spreadsheet seems the easiest way to keep a list of the metadata names
and metadata values for now. I am also looking at a csv -> json converter
to use with "manage.py loaddata".

Still open to other suggestions to simplify this problem!

Thanks again,

Mark

On Fri, Sep 8, 2017 at 9:31 PM, James Schneider 
wrote:

>
>
> On Sep 8, 2017 12:12 PM, "Mark Phillips" 
> wrote:
>
> Thanks for all your replies and helpful suggestions. To answer some of
> your questions -
>
> 1. Where is the data coming from? - It is textual data in a
> spec/spreadsheet. The data is a lot of meta data (name and one or more
> values) for describing attributes of scanned documents. There is what I
> would call a "base set" of data, which means it is what we can think of now
> based on a review of a representative set of documents. However, as new
> documents are imported into the application, there may be other types of
> metadata with multiple values that will have to be created on the fly
> (hence the need for the admin forms to add meta date in the future). The
> import function I need is a one-off function. We need it as we develop the
> models and test them against various documents. Sometimes it is easier to
> just delete the database and rebuild it when we are developing the app than
> to back out certain migrations. So a simple way to populate the metadata
> for development purposes, and then one time when we go into production is
> what we are looking for. Currently, we have 24 metadata names, and each one
> can have one to 20 values.
>
>
> I use a management command to do exactly that: drop the database, create
> the database, run migrations, and populate the database with a fair bit of
> data. I use FactoryBoy to generate hundreds of fake users and other objects
> upon reset.
>
> http://factoryboy.readthedocs.io/en/latest/orms.html
>
> If the files are only being used for development, and should not normally
> be loaded in to the database, then I would definitely recommend a separate
> management command rather than building the fixture/fake data loading
> process directly in to the migrations.
>
>
>
> 2. The manage.py loaddata is an appealing option. However, it will take
> further effort to convert the spreadsheet data to any of the formats for
> this option. I think a csv file reader is more suitable for our purposes.
>
>
> If it is simple data that is being some what auto generated, I'd consider
> writing a FactoryBoy definition so you can generate 1 or thousands of
> entries on demand.
>
>
> 3. I will have to think about the validation concepts. These are simple
> name-value pairs, so it is not clear what I am validating against, unless
> it is detecting duplicates.
>
>
> Forms can do all sorts of validation. Duplicate detection of one of them.
> Value formatting and type checking is another. You may have some integer
> values that should only be within a certain range (ie a person can't be
> 29,000,000 years old, or maybe they can, who knows...). Forms are where
> input is filtered and massaged to match the business logic in your models.
>
> If you dump data directly in to models or the database, then you risk
> dumping in data that hasn't been evaluated by your application which may
> produce subtle bugs of the worst kind where things work in dev and break in
> prod because your input channel is different.
>
>
> 4. I am also looking into a spreadsheet -> csv file > mysql load data as
> perhaps the easiest way 

Re: How to pre-load model data?

[Derek]

"If you dump data directly in to models or the database, then you risk 
dumping in data that hasn't been evaluated by your application which may 
produce subtle bugs of the worst kind where things work in dev and break in 
prod because your input channel is different."

If this is happening its because you have poor validation on your model 
clean(s). You should NEVER just rely on a form to check data for you (at 
least beyond very basic sanitation); that logic properly belongs with the 
model.  We import direct from spreadsheets to models (note - NOT direct to 
the database) because we make use of such logic.

On Saturday, 9 September 2017 06:31:56 UTC+2, James Schneider wrote:
>
>
>
> On Sep 8, 2017 12:12 PM, "Mark Phillips"  > wrote:
>
> Thanks for all your replies and helpful suggestions. To answer some of 
> your questions - 
>
> 1. Where is the data coming from? - It is textual data in a 
> spec/spreadsheet. The data is a lot of meta data (name and one or more 
> values) for describing attributes of scanned documents. There is what I 
> would call a "base set" of data, which means it is what we can think of now 
> based on a review of a representative set of documents. However, as new 
> documents are imported into the application, there may be other types of 
> metadata with multiple values that will have to be created on the fly 
> (hence the need for the admin forms to add meta date in the future). The 
> import function I need is a one-off function. We need it as we develop the 
> models and test them against various documents. Sometimes it is easier to 
> just delete the database and rebuild it when we are developing the app than 
> to back out certain migrations. So a simple way to populate the metadata 
> for development purposes, and then one time when we go into production is 
> what we are looking for. Currently, we have 24 metadata names, and each one 
> can have one to 20 values.
>
>
> I use a management command to do exactly that: drop the database, create 
> the database, run migrations, and populate the database with a fair bit of 
> data. I use FactoryBoy to generate hundreds of fake users and other objects 
> upon reset.
>
> http://factoryboy.readthedocs.io/en/latest/orms.html
>
> If the files are only being used for development, and should not normally 
> be loaded in to the database, then I would definitely recommend a separate 
> management command rather than building the fixture/fake data loading 
> process directly in to the migrations.
>
>
>
> 2. The manage.py loaddata is an appealing option. However, it will take 
> further effort to convert the spreadsheet data to any of the formats for 
> this option. I think a csv file reader is more suitable for our purposes. 
>
>
> If it is simple data that is being some what auto generated, I'd consider 
> writing a FactoryBoy definition so you can generate 1 or thousands of 
> entries on demand.
>
>
> 3. I will have to think about the validation concepts. These are simple 
> name-value pairs, so it is not clear what I am validating against, unless 
> it is detecting duplicates.
>
>
> Forms can do all sorts of validation. Duplicate detection of one of them. 
> Value formatting and type checking is another. You may have some integer 
> values that should only be within a certain range (ie a person can't be 
> 29,000,000 years old, or maybe they can, who knows...). Forms are where 
> input is filtered and massaged to match the business logic in your models. 
>
> If you dump data directly in to models or the database, then you risk 
> dumping in data that hasn't been evaluated by your application which may 
> produce subtle bugs of the worst kind where things work in dev and break in 
> prod because your input channel is different.
>
>
> 4. I am also looking into a spreadsheet -> csv file > mysql load data as 
> perhaps the easiest way to complete this project. The spreadsheet is easy 
> to create and update the metadata with the least effort, and then it is 
> pretty automatic from the spreadsheet to the database. 
>
> I am open to other suggestions!
>
>
> I'd recommend FBoy and eliminate Excel entirely if I could. Otherwise an 
> Excel parser would be my next choice. 
>
> -James
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/ebe085b2-87fc-4a02-aea5-f3e7a54b8e4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Need advice on writing very specific custom user roles

[Ahmad Zoughbi]

Hello,

I need to build custom Django user roles, and i need an advice on the best 
approach (or library that i can use) to achieve that: 


   1. 
   
   There will be 3 different user levels:
   1. 
  
  Author
  2. 
  
  Editor
  3. 
  
  Verification
  2. 
   
   Every user must be limited to a city, based on custom field (New York, 
   Berlin, Amsterdam, etc)
   3. 
   
   Only ‘Author’ can add content, while ‘Editor’ and ‘Verification’ can 
   edit the content. 
   4. 
   
   Author: Can only view/edit the content created by them. 
   5. 
   
   Editor: Can only view/edit the content created by ‘Author’s in their 
   cities. 
   6. 
   
   Verification: Can only view/edit the content created by the ‘Editor’s in 
   their cities. 
   7. 
   
   Author: When creates content, should have the options to: 
   1. 
  
  Save as draft
  2. 
  
  Send to Editor
  8. 
   
   Editor: When edits content, should have the options to:
   1. 
  
  Send back to Author
  2. 
  
  Send to Verification
  9. 
   
   Verification: When edits content, should have the options to:
   1. 
  
  Send back to the Editor
  2. 
  
  Mark as Verified.
  

Scenario:

An admin creates the following users:

Username | Role | City | Content titles written by the user

John | Author | New York | Test1, Test2, Test3

Sam | Author | New York | Test4, Test5, Test6

Kim | Author | Berlin | Test7, Test8, Test9

Adam | Editor | New York

Alex | Editor | Berlin

Stacey | Verification | New York

Anthony | Verification | Berlin

Based on the above scenario, the Authors below should be able to access 
only the titles they created; Additionally: 

Adam: Should be able to view/edit the titles: Test1 - Test6 (Since they 
were written by 2 Authors in his City. 

Alex: Should be able to view/edit the titles: Test7 - Test9 (Since they 
were written by Kim, which in his city)

The same applies for the Verification. 

Finally, If Author sent the content to Editor, the Author should no longer 
be able to edit the content, until it’s sent back to him by the Editor. 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/46db433a-a6c5-4e6b-92f0-e605b8e7b2f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.