Re: Obtaining content from Git

[martin f krafft]

also sprach Russell Keith-Magee  [2014-11-25 01:16 
+0100]:
> Sure - that could work too. Although I'd question why you'd do
> this, rather than just checking out the relevant hash onto the
> filesystem, and doing a direct filesystem read.

Yeah, and I think you have me 99.5% convinced that I want
a filesystem backend.

> And if you're going to be even more ruthless about it, I'd be
> questioning why you've got a live Django server in the loop at all
> - if it's static files, "post processed", then why not pre-render
> the Github files to a "rendered" form (possibly in response to
> a commit trigger), and simply serve the pre-rendered files as
> static content.

Because the static content is used inside dynamic apps, and certain
parts of the website would be offered instead by apps and not static
files at all. As I said in my initial e-mail, we currently have
a Django app with Apache-level overrides of certain static pages
that are trying hard to look just like and integrate well with the
output Django renders and this is ugly.

-- 
@martinkrafft | http://madduck.net/ | http://two.sentenc.es/
 
normaliser unix c'est comme pasteuriser le camembert.
 
spamtraps: madduck.bo...@madduck.net

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20141125054104.GA5859%40fishbowl.rw.madduck.net.
For more options, visit https://groups.google.com/d/optout.


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current)

Re: Obtaining content from Git

[Russell Keith-Magee]

On Mon, Nov 24, 2014 at 7:43 PM, martin f krafft 
wrote:

> also sprach Russell Keith-Magee  [2014-11-24
> 11:38 +0100]:
> > Perhaps I wasn't clear.
>
> No, you were. I am just pushing back a little bit because I come
> from another angle and I greatly appreciate your explanations in
> helping me understand the Django way of thinking. So in the hope
> that you now don't feel exploited… ;)
>
> > You might be able to write a mapping to Storage that did some sort
> > of naming trick (e.g., split off a colon at the end of a filename
> > for read, so you could request "/my/file.txt:deadbeef" to request
> > the "deadbeaf" hash of /my/file.txt") - but my point is that the
> > storage API doesn't contain a *natural* endpoint for version
> > information.
>
> Arguably, the ref/hash/date could come from the config, not from the
> file, and suddenly there'd be a 1:1 mapping between a filesystem
> store and a storage that uses Git directly.
>
>
Sure - that could work too. Although I'd question why you'd do this, rather
than just checking out the relevant hash onto the filesystem, and doing a
direct filesystem read.

> And, even if you did this, what you're going to get is a view of
> > your Git repository that tries *really* hard to make it look like
> > just a normal file system.
>
> Yeah, this is an excellent point, especially since I'd probably not
> allow pushes to that Git repository but instead set up some sort of
> polling or regular interval pulling. Then I might just as well use
> a filesystem.
>
> > def myview(request):
> > with open('filename.txt') as datafile:
> > data = datafile.read()
> > return render(request, 'my template.html', {'data': data})
> >
> > You're possibly getting lost by thinking that this is a "Django" thing -
> it
> > isn't. Basic Python is always an option.
>
> Yeah, but I would like to assume that this has been wrapped for
> Django at least one "canonical" time, with all error handling done
> and proper cache integration taken care of.
>
> Because to just start off with a myview() view like you suggest
> above is quickly going to become a full-time project, if you know
> what I mean. ;)
>

Unless I'm missing some important detail, I don't see why. It's a fairly
simple caching layer, and a fairly simple rendering layer. The only
complexities that I can see are the ones related to accessing the Git
repository - and that's the bit that isn't part of a "common" use case.


> > > Have you seen something like this done? Is this also still best
> > > done in a view?
> >
> > Well... yes, I've seen this done before. It's called GitHub :-)
>
> Hehe, that was funny.
>
> But seriously: forget about all of the Git stuff, let's just look at
> the filesystem… at least in my universe, I'd like to assume that the
> idea of reading marked-up data from disk, processing it, caching it
> and then making the data available to templates is such an intuitive
> and standard thing to do that it could even be in Django core — if
> not there, then there ought to be an existing plugin for this.
>

Again - the reason that this isn't in Django is because it's not really
Django's bailiwick. Response-level caching is something that should be
handled at the web server level, and web servers have various ways to
handle this. And if you're going to be even more ruthless about it, I'd be
questioning why you've got a live Django server in the loop at all - if
it's static files, "post processed", then why not pre-render the Github
files to a "rendered" form (possibly in response to a commit trigger), and
simply serve the pre-rendered files as static content.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAJxq84-HogDjdv_w03qVCnv-KnNd3_9_VEce-ogCbQje8BOXMg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Obtaining content from Git

[reduxionist]

Arg, Google Groups' mobile interface sucks for posting rather than reading. 
As I was trying to say:

On Tuesday, November 25, 2014 2:43:55 AM UTC+7, reduxionist wrote:
>
> >Hence I was thinking: how much trouble would it be to have Django 
> >reach into Git rather than its database and obtain data there to be 
> >filled into template slots? Ideally, there'd be the possibility of 
> >running a filter (e.g. reStructuredText) between Git and the 
> >template rendering. 
>
> >What I envision is a storage layer (with optional caching) that 
> >either fetches from the filesystem (with a Git checkout, using mtime 
> >for cache expiration), or directly from a local Git repo (using 
> >either commit or blob hash for cache expiration). 
>
> >Does anyone know of such a module? 

 
Yes:
https://bitbucket.org/bors_ltd/django-gitstorage
(before you fork to github anew, consider updating this existing, but out 
of date, github mirror: https://github.com/9h37/django-gitstorage)
 

> >Would it be hard to write? Where would one start? 
>

Here's a couple of example projects that do similar things.

A git-backed wiki: 
https://github.com/mgaitan/waliki

A mercurial based latex hosting site:
https://bitbucket.org/Svedrin/webtex
("This is a Django Project providing a WebTeX service. The way this works 
is totally ripped from BitBucket: Users have HG Repos, push their Documents 
into it, and can get a compiled version and other cool stuff over the net.")

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/13c2e5eb-0f34-41c8-a77d-6c8b799f1a96%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Obtaining content from Git

[Tim Chase]

On 2014-11-24 11:49, martin f krafft wrote:
> also sprach Tim Chase  [2014-11-24
> 11:19 +0100]:
> >   urlpatterns = patterns('',
> > url(r'^blob/([0-9a-f]{40})/$',
> > 'myapp.views.render_blob', ...), )
> 
> … this is straight-forward, thanks. I'd probably try not to expose
> the SHAs to the user but request from the Git library a blob
> identified by a path such as /path/to/file which the Git
> library should then map to the blob, given a treeish to use as
> a base (e.g. 'master', this could come from config).

Django certainly gives you the flexibility to do both/either.  You
can even do things like

  url(r'branch/(?P[^/]*)(?P.*)', 
'myapp.views.render_branch_file', ...)

and then in your views.py

  def render_branch_file(request, branch, file_name):
data = some_git_lib.get_file_from_branch(branch, file_name)
...

You might need to tighten up those regular expressions in the urls.py
a bit, or sanitize them in the view to ensure that they don't allow
access to files outside the git tree (you might also want to prevent
access to the .git subdirectory)

> > In theory, you could also set the cache control headers in the
> > response so that they're ludicrously far in the future (assuming
> > you don't plan to change your my_template.html) because the blob
> > shouldn't ever change without having a different SHA1.
> 
> Two problems I see with this:
> 
>   1. If the SHA is not exposed, then a request path (see above) may
>  well return different content on subsequent calls, so
>  expiration-based caching isn't ideal;

You can use the resulting internal sha1 as the "etag" header which
would cache at the object level rather than at the URL level.

>   2. aren't the cache-control headers for the caches downstream?
>  Wouldn't this mean that if 1,000 clients requested a page, the
>  reStructuredText processor would have to do the same work 1,000
>  times? I'd love to cache the result and serve that while the
>  source file's pre-processing mtime hasn't changed.

yes-ish.  If you have a caching proxy such as Varnish in front of your
Django server, it can cache the pages coming out and then serve them
to all requesting clients.  I haven't dug into how that interacts
with "etag" vs. other cache-control headers, but that's where I'd
start researching.  Or just set the other cache-control headers.

-tkc








-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20141124101138.1a76647e%40bigbox.christie.dr.
For more options, visit https://groups.google.com/d/optout.

Re: Obtaining content from Git

[martin f krafft]

also sprach Russell Keith-Magee  [2014-11-24 11:38 
+0100]:
> Perhaps I wasn't clear.

No, you were. I am just pushing back a little bit because I come
from another angle and I greatly appreciate your explanations in
helping me understand the Django way of thinking. So in the hope
that you now don't feel exploited… ;)

> You might be able to write a mapping to Storage that did some sort
> of naming trick (e.g., split off a colon at the end of a filename
> for read, so you could request "/my/file.txt:deadbeef" to request
> the "deadbeaf" hash of /my/file.txt") - but my point is that the
> storage API doesn't contain a *natural* endpoint for version
> information.

Arguably, the ref/hash/date could come from the config, not from the
file, and suddenly there'd be a 1:1 mapping between a filesystem
store and a storage that uses Git directly.

> And, even if you did this, what you're going to get is a view of
> your Git repository that tries *really* hard to make it look like
> just a normal file system.

Yeah, this is an excellent point, especially since I'd probably not
allow pushes to that Git repository but instead set up some sort of
polling or regular interval pulling. Then I might just as well use
a filesystem.

> def myview(request):
> with open('filename.txt') as datafile:
> data = datafile.read()
> return render(request, 'my template.html', {'data': data})
> 
> You're possibly getting lost by thinking that this is a "Django" thing - it
> isn't. Basic Python is always an option.

Yeah, but I would like to assume that this has been wrapped for
Django at least one "canonical" time, with all error handling done
and proper cache integration taken care of.

Because to just start off with a myview() view like you suggest
above is quickly going to become a full-time project, if you know
what I mean. ;)

> > Have you seen something like this done? Is this also still best
> > done in a view?
> 
> Well... yes, I've seen this done before. It's called GitHub :-)

Hehe, that was funny.

But seriously: forget about all of the Git stuff, let's just look at
the filesystem… at least in my universe, I'd like to assume that the
idea of reading marked-up data from disk, processing it, caching it
and then making the data available to templates is such an intuitive
and standard thing to do that it could even be in Django core — if
not there, then there ought to be an existing plugin for this.

Thanks for your help thus far!

-- 
@martinkrafft | http://madduck.net/ | http://two.sentenc.es/
 
don't hate yourself in the morning -- sleep till noon.
 
spamtraps: madduck.bo...@madduck.net

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20141124114316.GA19818%40albatross.lehel.madduck.net.
For more options, visit https://groups.google.com/d/optout.


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current)

Re: Obtaining content from Git

[martin f krafft]

also sprach Tim Chase  [2014-11-24 11:19 +0100]:
>   urlpatterns = patterns('',
> url(r'^blob/([0-9a-f]{40})/$', 'myapp.views.render_blob', ...),
> )

… this is straight-forward, thanks. I'd probably try not to expose
the SHAs to the user but request from the Git library a blob
identified by a path such as /path/to/file which the Git
library should then map to the blob, given a treeish to use as
a base (e.g. 'master', this could come from config).

> In theory, you could also set the cache control headers in the
> response so that they're ludicrously far in the future (assuming
> you don't plan to change your my_template.html) because the blob
> shouldn't ever change without having a different SHA1.

Two problems I see with this:

  1. If the SHA is not exposed, then a request path (see above) may
 well return different content on subsequent calls, so
 expiration-based caching isn't ideal;

  2. aren't the cache-control headers for the caches downstream?
 Wouldn't this mean that if 1,000 clients requested a page, the
 reStructuredText processor would have to do the same work 1,000
 times? I'd love to cache the result and serve that while the
 source file's pre-processing mtime hasn't changed.

-- 
@martinkrafft | http://madduck.net/ | http://two.sentenc.es/
 
"i like wagner's music better than anybody's. it is so loud that one
 can talk the whole time without other people hearing what one says."
-- oscar wilde
 
spamtraps: madduck.bo...@madduck.net

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20141124104910.GA11742%40albatross.lehel.madduck.net.
For more options, visit https://groups.google.com/d/optout.


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current)

Re: Obtaining content from Git

[Russell Keith-Magee]

On Mon, Nov 24, 2014 at 5:06 PM, martin f krafft 
wrote:

> also sprach Russell Keith-Magee  [2014-11-24
> 07:16 +0100]:
> > The right place is in the view.
> […]
> > If you want to do "interesting" things like retrieve a specific
> > file version, you're not going to be able to use the storage API,
> > because most raw
>
> Interesting perspective you have there, which I can follow very
> well. Thanks!
>
> But let me dig a bit further, nevertheless:
>
> > The storage layer is an abstraction to enable reusable apps. If
> > I write a "user profile" app that needs to store an avatar, then
> > I don't care where that avatar comes from, as long as it's
> > available. It could be on a local filesystem; it could be on
> > a cloud file store (like S3). The storage API lets me do that.
>
> It could come from Git! After all, Git is really a database by
> itself, just like the filesystem could be viewed as a database. So
> somewhere in my idealist brain there's this theory that I could just
> add a Git-storage-plugin to the storage layer and that would cause
> content to be fetched from Git auto-magically if the
> request-for-data by the view (not the HTTP request) matches certain
> criteria and those data are available in Git. If not, then the
> lookup goes to the next storage plugin.
>
> So my view might call some sort of lookup() function, e.g.
>
>   lookup_description(event_id)
>
> which just returns a string. Ordinarily, this string comes from
> pgsql at the moment. What I'd like to see is a way to override this
> function so that instead the data are fetched from
>   git.example.org/myrepo.git, branch=live, file=events/$event_id
>
> Obviously, I could just as well call the Git-specific stuff straight
> from the view, but that would inevitably tie the view to Git. Maybe
> this is what I want, I am just trying to explain how I approached
> this and what led me to ask the question the way I did.
>

Perhaps I wasn't clear. Of course an avatar could come from a git store. My
point is that it's not a natural mapping. A file system is a mapping
between a path name and a binary blob. Git has an additional layer - the
path name, *plus* a historical version (referenced either by a hash, or a
date). And you can't arbitrarily write to any point in a Git tree. Ok...
you can if you're willing to have orphan nodes, but if you're thinking
about version control in the traditional sense, a git repository can only
be written to the "end" of history, but read from any point in it's history.

You might be able to write a mapping to Storage that did some sort of
naming trick (e.g., split off a colon at the end of a filename for read, so
you could request "/my/file.txt:deadbeef" to request the "deadbeaf" hash of
/my/file.txt") - but my point is that the storage API doesn't contain a
*natural* endpoint for version information.

And, even if you did this, what you're going to get is a view of your Git
repository that tries *really* hard to make it look like just a normal file
system. All the interesting features will, by necessity of the interface,
be buried. If you actually want to do anything interesting based on the
fact that you're using a git store under the hood, you're better served
working closer to the metal.

> If you're just looking to read the *current* value in a git
> > repository, then just use normal filesystem storage over a normal
> > git checkout.
>
> Yeah, this might well be the best option, also in terms of
> performance, unless I want to keep track of blob hashes to avoid
> doing the whole branch→tree→blob lookup every time, which the
> filesystem "caches" for me.
>
> Do you know of a simple example that fetches data from the
> filesystem? I am being lazy here as I am sure there's plenty, so
> feel free to RTFM me! ;) However, maybe you know a very good example
> that makes everything so clear and then I'd really appreciate this
> over wading through the various means. I know Django has provisions
> for serving static files, but somehow it seems like that's not what
> I want… (since the files are not actually static and a given path
> could return different data on subsequent calls)
>
> A simple example that fetches data from the filesystem? Sure:

def myview(request):
with open('filename.txt') as datafile:
data = datafile.read()
return render(request, 'my template.html', {'data': data})

You're possibly getting lost by thinking that this is a "Django" thing - it
isn't. Basic Python is always an option.

Furthermore, I'd actually like to post-process the data. The Git
> repo would contain reStructuredText files and I'd like to render
> them before filling the result into template slots.
>
> This makes me think that there ought to be a cache involved at this
> level. Sure, I could make a simple expiration-based cache, but I'd
> really rather make the cache depend on the mtime of the underlying
> source file on the filesystem (but include the 

Re: Obtaining content from Git

[Tim Chase]

On 2014-11-24 10:06, martin f krafft wrote:
> It could come from Git! After all, Git is really a database by
> itself, just like the filesystem could be viewed as a database.
[snip]
> Furthermore, I'd actually like to post-process the data. The Git
> repo would contain reStructuredText files and I'd like to render
> them before filling the result into template slots.

You'd still have to sort out how to know which blob you want to
return/render.  If you have the hash-id of it, you can do a direct
lookup.  Throwing together some pseudo-code, your urls.py might look
something like

  urlpatterns = patterns('',
url(r'^blob/([0-9a-f]{40})/$', 'myapp.views.render_blob', ...),
)

and then in myapp/views.py you'd have something like

  from django.shortcuts import render
  import some_git_lib
  import your_markdown_lib

  def render_blob(request, sha):
try:
  blob = some_git_lib.fetch_blob(sha)
except some_git_lib.SomeException:
  raise # do some actual error handling/reporting here
else:
  rendered_markdown = your_markdown_lib.htmlize(blob)
  return render(request, "my_template.html", {"data": blob})

Only you know how the sha1 of the blob is determined, but the rest
should give you a pretty straight-forward implementation.  In theory,
you could also set the cache control headers in the response so that
they're ludicrously far in the future (assuming you don't plan to
change your my_template.html) because the blob shouldn't ever change
without having a different SHA1.

-tkc



-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20141124041927.20723b62%40bigbox.christie.dr.
For more options, visit https://groups.google.com/d/optout.

Re: Obtaining content from Git

[martin f krafft]

also sprach Russell Keith-Magee  [2014-11-24 07:16 
+0100]:
> The right place is in the view.
[…]
> If you want to do "interesting" things like retrieve a specific
> file version, you're not going to be able to use the storage API,
> because most raw

Interesting perspective you have there, which I can follow very
well. Thanks!

But let me dig a bit further, nevertheless:

> The storage layer is an abstraction to enable reusable apps. If
> I write a "user profile" app that needs to store an avatar, then
> I don't care where that avatar comes from, as long as it's
> available. It could be on a local filesystem; it could be on
> a cloud file store (like S3). The storage API lets me do that.

It could come from Git! After all, Git is really a database by
itself, just like the filesystem could be viewed as a database. So
somewhere in my idealist brain there's this theory that I could just
add a Git-storage-plugin to the storage layer and that would cause
content to be fetched from Git auto-magically if the
request-for-data by the view (not the HTTP request) matches certain
criteria and those data are available in Git. If not, then the
lookup goes to the next storage plugin.

So my view might call some sort of lookup() function, e.g.

  lookup_description(event_id)

which just returns a string. Ordinarily, this string comes from
pgsql at the moment. What I'd like to see is a way to override this
function so that instead the data are fetched from
  git.example.org/myrepo.git, branch=live, file=events/$event_id

Obviously, I could just as well call the Git-specific stuff straight
from the view, but that would inevitably tie the view to Git. Maybe
this is what I want, I am just trying to explain how I approached
this and what led me to ask the question the way I did.

> If you're just looking to read the *current* value in a git
> repository, then just use normal filesystem storage over a normal
> git checkout.

Yeah, this might well be the best option, also in terms of
performance, unless I want to keep track of blob hashes to avoid
doing the whole branch→tree→blob lookup every time, which the
filesystem "caches" for me.

Do you know of a simple example that fetches data from the
filesystem? I am being lazy here as I am sure there's plenty, so
feel free to RTFM me! ;) However, maybe you know a very good example
that makes everything so clear and then I'd really appreciate this
over wading through the various means. I know Django has provisions
for serving static files, but somehow it seems like that's not what
I want… (since the files are not actually static and a given path
could return different data on subsequent calls)



Furthermore, I'd actually like to post-process the data. The Git
repo would contain reStructuredText files and I'd like to render
them before filling the result into template slots.

This makes me think that there ought to be a cache involved at this
level. Sure, I could make a simple expiration-based cache, but I'd
really rather make the cache depend on the mtime of the underlying
source file on the filesystem (but include the post-processing step
in between).

Have you seen something like this done? Is this also still best done
in a view?

Thanks a lot for your patience and helpful replies!

-- 
@martinkrafft | http://madduck.net/ | http://two.sentenc.es/
 
review of a chemistry paper:
  "paper should be greatly reduced or completely oxidized."
-- frank vastola
 
spamtraps: madduck.bo...@madduck.net

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20141124090640.GA30500%40albatross.lehel.madduck.net.
For more options, visit https://groups.google.com/d/optout.


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current)

Re: Obtaining content from Git

[Russell Keith-Magee]

On Mon, Nov 24, 2014 at 1:45 PM, martin f krafft 
wrote:

> also sprach Russell Keith-Magee  [2014-11-24
> 00:42 +0100]:
> > The problem is that there isn't one. There's several :-)
> >
> > I'm aware of at least 3:
> >
> > https://pypi.python.org/pypi/GitPython/0.3.2.1
> > https://pypi.python.org/pypi/pygit2
> > https://pypi.python.org/pypi/dulwich
>
> I am aware of those and I was wondering more about the glue on the
> Django-side of things, i.e. how the storage plugin/layer would work.
>
> I am pretty inexperienced with Django, hence I find it hard to even
> figure out where the sensible place to do this would be.
>

The right place is in the view.

At the simplest level, a view is just a function that converts a request
into a response. "Normal" Django template rendering (i.e., return
render(request, "mytemplate.html", {'object': obj}) ) hides a lot of detail
- but all it's really doing is opening a file on disk (from one of a couple
of a known locations - the template directories), parsing that template,
rendering it with a context, and returning it to the user.

A simple "hello world" view will develop "obj" by doing database queries -
but the information can come from any other data source that is relevant.
You might hit a non-relational store (such as Redis), or a search engine
(like ElasticSearch), or do some sort of calculation based on the GET/POST
arguments provided. In your case, you'll be taking the GET/POST values,
plus the URL parameters, and using them to identify a "resource" in your
Git tree; your git calls will extract that resource; and then you can
either return that resource literally as your response (with an appropriate
content type), or pass the resource to a template to be rendered.

Thinking about the storage layer is going deeper than you need to go. The
storage layer is an abstraction to enable reusable apps. If I write a "user
profile" app that needs to store an avatar, then I don't care where that
avatar comes from, as long as it's available. It could be on a local
filesystem; it could be on a cloud file store (like S3). The storage API
lets me do that.

The storage API isn't there to abstract all possible concepts of storage
that might exist, and just because you'll be accessing a notional file
system, doesn't mean that access needs to be done through the storage API -
it's fine to just open the file.

While you probably *could* write a storage plugin that would read a git
repository, that's not really what the storage API is for.

If you're just looking to read the *current* value in a git repository,
then just use normal filesystem storage over a normal git checkout. If you
want to do "interesting" things like retrieve a specific file version,
you're not going to be able to use the storage API, because most raw
filesystems don't have a concept of versioning.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAJxq848%2B46c7Hz1P9VCaiOoZqyoNekDGDdrH0iiDUaXUpCKcNw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Obtaining content from Git

[martin f krafft]

also sprach Russell Keith-Magee  [2014-11-24 00:42 
+0100]:
> The problem is that there isn't one. There's several :-)
> 
> I'm aware of at least 3:
> 
> https://pypi.python.org/pypi/GitPython/0.3.2.1
> https://pypi.python.org/pypi/pygit2
> https://pypi.python.org/pypi/dulwich

I am aware of those and I was wondering more about the glue on the
Django-side of things, i.e. how the storage plugin/layer would work.

I am pretty inexperienced with Django, hence I find it hard to even
figure out where the sensible place to do this would be.

-- 
@martinkrafft | http://madduck.net/ | http://two.sentenc.es/
 
"there are two major products that come out of berkeley: lsd and unix.
 we don't believe this to be a coincidence."
 -- jeremy s. anderson
 
spamtraps: madduck.bo...@madduck.net

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20141124054512.GA14198%40fishbowl.rw.madduck.net.
For more options, visit https://groups.google.com/d/optout.


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current)

Re: Obtaining content from Git

[Russell Keith-Magee]

On Sun, Nov 23, 2014 at 5:28 AM, martin f krafft 
wrote:

> Hello,
>
> we have a Django project with a few pages that come from Git.
> Currently, Apache rewrite rules serve those files statically (and
> they make use of the same template/CSS as Django does, so the user
> does not actually notice), and it's a massive hack and pain to keep
> up-to-date.
>
> Hence I was thinking: how much trouble would it be to have Django
> reach into Git rather than its database and obtain data there to be
> filled into template slots? Ideally, there'd be the possibility of
> running a filter (e.g. reStructuredText) between Git and the
> template rendering.
>
> I've seen http://luispedro.org/software/git-cms, but that does way
> more than just sourcing from Git. And it's not immediately obvious
> to me how it even does the Git interaction.
>
> What I envision is a storage layer (with optional caching) that
> either fetches from the filesystem (with a Git checkout, using mtime
> for cache expiration), or directly from a local Git repo (using
> either commit or blob hash for cache expiration).
>
> Does anyone know of such a module? Would it be hard to write? Where
> would one start?
>

The problem is that there isn't one. There's several :-)

I'm aware of at least 3:

https://pypi.python.org/pypi/GitPython/0.3.2.1
https://pypi.python.org/pypi/pygit2
https://pypi.python.org/pypi/dulwich

These are all Python wrappers around the Git API - and, in the case of
dulwich, a full re-implementation of git in Python. Using these APIs, you
should be able to retrieve an file with an arbitrary hash, the same as you
would at the command line.

I haven't used any of them myself, so I can't comment on which one is
best/most stable/most reliable/fastest, but:

 * Yes, they exist

 * Yes, you could use these modules to retrieve information from the git
checkout as part of a view

 * Yes, that data could then be passed to a template, either directly, or
after post-processing with a filter of some kind.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAJxq849yt%2BU99gWy66uGD_%2BC2%2BuKZoQ2OTAop8UWuLGa9BvOkQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Obtaining content from Git

[Aaron C. de Bruyn]

I'm not aware of any such module, but there are a few python 'git'
modules for reading/writing, etc...

As for how much trouble it would be, that all depends on how familiar
you are with git, python, Django, and the template system.

One way you might be able to achieve what you want is to create your
own template tag.  (The documentation is here:
https://docs.djangoproject.com/en/dev/howto/custom-template-tags/#writing-custom-template-tags)

You could pass a few parameters to your tag--I'm not sure what you're
looking for, but maybe:

{% gitfile "myfolder/myfile.txt" "d4ea4eea799" %}

And your tag could load data from "myfolder/myfile.txt" at revision
"d4ea4eea799".

There are a few exampled in the Django code itself
(https://github.com/django/django/tree/master/django/templatetags) as
well as plenty of samples on the internet for writing your own
template tag.

-A

On Fri, Nov 21, 2014 at 2:29 PM, martin f krafft  wrote:
> Hello,
>
> we have a Django project with a few pages that come from Git.
> Currently, Apache rewrite rules serve those files statically (and
> they make use of the same template/CSS as Django does, so the user
> does not actually notice), and it's a massive hack and pain to keep
> up-to-date.
>
> Hence I was thinking: how much trouble would it be to have Django
> reach into Git rather than its database and obtain data there to be
> filled into template slots? Ideally, there'd be the possibility of
> running a filter (e.g. reStructuredText) between Git and the
> template rendering.
>
> I've seen http://luispedro.org/software/git-cms, but that does way
> more than just sourcing from Git. And it's not immediately obvious
> to me how it even does the Git interaction.
>
> What I envision is a storage layer (with optional caching) that
> either fetches from the filesystem (with a Git checkout, using mtime
> for cache expiration), or directly from a local Git repo (using
> either commit or blob hash for cache expiration).
>
> Does anyone know of such a module? Would it be hard to write? Where
> would one start?
>
> Thanks,
>
> --
> @martinkrafft | http://madduck.net/ | http://two.sentenc.es/
>
> mulutlitithtrhreeaadededd s siigngnatatuurere
>
> spamtraps: madduck.bo...@madduck.net
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/20141121222930.GA15206%40fishbowl.rw.madduck.net.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAEE%2BrGqxe5NGvQU6Coam0%2Bev5AjG1GdONt7cdhj9h%2BUshEHSgg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.