Hi,
I am Sudhanshu, a fourth year undergraduate student from IIIT Hyderabad. I
am interested in applying for GSOC 2015 under scrapy. I am specially
interested in the simplifying scrapy addons project.
I spent sometime going over last year's discussions on this topic and on
looking at how Debian handles the dependencies between its packages. So, I
have some ideas and would like to know your views on the same.
1) Debian packages are required to have a control file detailing their
version numbers, dependencies etc. We can make it mandatory for all
extension classes to have a meta data function which will furnish such
information.
2) This meta-data function can thus, be the place where users add their
private configuration options (database passwords, usernames, storage
locations etc).
3) The user will have to list the enabled add-ons in the scrapy.cfg file in
the order of their invocation ( In the later stages this requirement can be
removed and the add-on manager can be made to load the dependencies
automatically).
4) The add-on manager will start instantiating the extensions from top to
bottom ensuring that the dependencies listed in the meta-data are
fulfilled. We can use versioning to ensure extensions can specify the exact
version of another extension they need.
5) To prevent namespace conflict (eg: both extension A and B need
is_enabled setting) we can ask the extension developers to properly scope
their settings. Settings which are defined internally by the module can be
kept in the meta-data area only. Settings which are of a more global nature
(those defined in settings.py or in other extensions) should be used with
proper scoping [self.is_enabled or
addonManager.globalSettings.get("is_enabled") or
firstDependency.get("is_enabled")].
6) If we ask the user to list the extensions in the proper order, we
wouldn't have to deal with circular dependencies at all.
I have used scrapy for some of my projects but haven't played around with
extensions till now, so I might be barking up the wrong tree here. I would
appreciate any feedback/comments/guidance on this topic :)
Thank you.
Regards,
Sudhanshu
irc nick: sudshekhar
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
