[mezzanine-users] Drupal to Mezzanine migration with pandas

2018-01-17 Thread managingwholes . com
Recently I migrated content from Drupal to Mezzanine using jupyter notebook 
and pandas.

After numerous dead ends, fighting with the inherited table structure of 
Mezzanine Pages/RichTextPages, it looked something like this:


   1. install shell_plus jupyter notebook on same (Ubuntu) server as 
   Mezzanine installation, in my virtualenv. Need django_extensions for this. 
   This interfaces directly with Django models (may need to be in the same 
   directory as manage.py), and is a super useful interface to have; you can 
   now manipulate your data, even live, with pandas. This can be run from my 
   laptop via Putty SSH tunneling.
   2. Reproduce mysql database from Drupal on Ubuntu server.
   3. Load relevant data into pandas dataframe using mysql.connector, super 
   fast
   4. was able to clean up some inconsistencies while data was in 
   dataframe, e.g. Drupal uses \n\r\n for paragraph breaks (Mezzanine or 
   tinymce ignores these) which I replaced with html p tags.
   5. Can't use bulk_create() with inherited models. My data was destined 
   for Page and RichTextPage objects. First you must create the RichTextPage 
   objects. I iterated over dataframe rows with itertuples() to create 
   RichTextPage objects, which I saved one at a time. Made sure to have an 
   index in dataframe that corresponded to the id or pk of the RichTextPage 
   objects I created.
   6. This also created Page objects which I then updated individually with 
   the correct title, in_menus, slug, created, site_id, and other fields again 
   using itertuples over the dataframe. 

Somewhat of an involved procedure but it worked well.

-- 
You received this message because you are subscribed to the Google Groups 
"Mezzanine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mezzanine-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [mezzanine-users] Drupal to Mezzanine migration with pandas

2018-01-17 Thread Matt Mansour
Thanks for posting. Glad it worked!

On Wed, Jan 17, 2018 at 11:45 AM,  wrote:

> Recently I migrated content from Drupal to Mezzanine using jupyter
> notebook and pandas.
>
> After numerous dead ends, fighting with the inherited table structure of
> Mezzanine Pages/RichTextPages, it looked something like this:
>
>
>1. install shell_plus jupyter notebook on same (Ubuntu) server as
>Mezzanine installation, in my virtualenv. Need django_extensions for this.
>This interfaces directly with Django models (may need to be in the same
>directory as manage.py), and is a super useful interface to have; you can
>now manipulate your data, even live, with pandas. This can be run from my
>laptop via Putty SSH tunneling.
>2. Reproduce mysql database from Drupal on Ubuntu server.
>3. Load relevant data into pandas dataframe using mysql.connector,
>super fast
>4. was able to clean up some inconsistencies while data was in
>dataframe, e.g. Drupal uses \n\r\n for paragraph breaks (Mezzanine or
>tinymce ignores these) which I replaced with html p tags.
>5. Can't use bulk_create() with inherited models. My data was destined
>for Page and RichTextPage objects. First you must create the RichTextPage
>objects. I iterated over dataframe rows with itertuples() to create
>RichTextPage objects, which I saved one at a time. Made sure to have an
>index in dataframe that corresponded to the id or pk of the RichTextPage
>objects I created.
>6. This also created Page objects which I then updated individually
>with the correct title, in_menus, slug, created, site_id, and other fields
>again using itertuples over the dataframe.
>
> Somewhat of an involved procedure but it worked well.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Mezzanine Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mezzanine-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Mezzanine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mezzanine-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.