I’ve updated the pystache-django-native template loaders to properly handle partial template loading. Had to dig through the pystache source a bit to see the right way to do it.
Category: Django
Django IntegrityError On PostgreSQL
The answer is here:
http://www.chrisspen.com/blog/handling-postgresql-integrity-errors-in-django.html
I was seeing IntegrityErrors while trying to unit test uniqueness constraints within a database table (it’s not strictly necessary, I suppose, but I was just being thorough, so much for trying to do the right thing):
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
$ ./nosetests -v model_tests.py
[...]
======================================================================
ERROR: test_of_some_sort (model_tests.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/lib/python2.7/django/db/models/manager.py", line 149, in create
return self.get_query_set().create(**kwargs)
File "/lib/python2.7/django/db/models/query.py", line 394, in create
obj.save(force_insert=True, using=self.db)
File "/lib/python2.7/django/db/models/base.py", line 537, in save
force_update=force_update, update_fields=update_fields)
File "/lib/python2.7/django/db/models/base.py", line 632, in save_base
result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
File "/lib/python2.7/django/db/models/manager.py", line 215, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "/lib/python2.7/django/db/models/query.py", line 1641, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/lib/python2.7/django/db/models/sql/compiler.py", line 935, in execute_sql
cursor.execute(sql, params)
File "/lib/python2.7/django/db/backends/util.py", line 41, in execute
return self.cursor.execute(sql, params)
File "/lib/python2.7/django/db/backends/postgresql_psycopg2/base.py", line 58, in execute
six.reraise(utils.DatabaseError, utils.DatabaseError(*tuple(e.args)), sys.exc_info()[2])
File "/lib/python2.7/django/db/backends/postgresql_psycopg2/base.py", line 54, in execute
return self.cursor.execute(query, args)
DatabaseError: current transaction is aborted, commands ignored until end of transaction block
|
What didn’t make sense was that the exact same sequences of calls would work just fine in ipython, they just wouldn’t work when under Unit test.
Turns out, whenever an IntegrityError happens while using Django and PostgreSQL, for whatever reason you need to close the entire connection to reset the transaction in the exception handler, which looks like:
|
1
2
3
4
|
try:
[...]
except IntegrityError:
connection.close()
|
Django Unit Testing On Shared Hoster
So I’ve been trying to write unit tests for Django, but, as I’m doing a lot of work on a shared hoster, this isn’t working. The problem is that our user account doesn’t have CREATE/DROP DATABASE privileges on the server. This has been documented here, here, here, and here.
Which means I see the following if I try to run manage.py test, I get:
|
1
2
3
4
5
|
Creating test database for alias 'default'...
Got an error creating the test database: permission denied to create database
Type 'yes' if you would like to try deleting the test database 'database_dev', or 'no' to cancel: no
Tests cancelled.
|
Unfortunately, there isn’t a first-class solution from the Django devs, so after reading that django-nose had a handy REUSE_DB parameter, I tried to install django-nose.
|
1 |
pip install --no-deps django-nose
|
(no-deps because I’m working with Django 1.5 beta 1)
Then, in settings.py, I set TEST_RUNNER=’django_nose.NoseTestSuiteRunner’, and set the TEST_NAME key on your ‘default’ DATABASES entry to the same value as NAME.
Unfortunately, you still get errors, if you’re trying to use PostGIS, seems like transactions may not be a happy thing there:
|
1
2
3
4
5
6
|
$ REUSE_DB=1 python2.7 manage.py test
/home/user/lib/python2.7/nose/util.py:14: DeprecationWarning: The compiler package is deprecated and removed in Python 3.x.
from compiler.consts import CO_GENERATOR
nosetests --verbosity 1
AttributeError: 'PostGISCreation' object has no attribute '_rollback_works'
|
So it looks like the only option I now have is to write unit tests that just run against a development database, and do a decent job of cleaning up after themselves in the setUp and tearDown methods. Using the standard nose distribution will do this, but if you just do “pip install nose” and try using “nosetests”, it will error out:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
======================================================================
ERROR: Failure: ImportError (No module named django.test)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/user/lib/python2.7/nose/loader.py", line 390, in loadTestsFromName
addr.filename, addr.module)
File "/home/user/lib/python2.7/nose/importer.py", line 39, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/user/lib/python2.7/nose/importer.py", line 86, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/app/polls/tests.py", line 8, in <module>
from django.test import TestCase
ImportError: No module named django.test
|
The problem is that nose doesn’t know about Django’s settings or modules. You have to make a copy of nosetests into the same directory where “manage.py” is located and change it so the DJANGO_SETTINGS_MODULE is incorporated into the os.environ resource. Like so:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
$ cat nosetests
#!/usr/local/bin/python2.7
# EASY-INSTALL-ENTRY-SCRIPT: 'nose==1.2.1','console_scripts','nosetests'
__requires__ = 'nose==1.2.1'
import os
import sys
from pkg_resources import load_entry_point
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")
sys.exit(
load_entry_point('nose==1.2.1', 'console_scripts', 'nosetests')()
)
|
Then you’ll see:
$ ./nosetests
……
———————————————————————-
Ran 6 tests in 4.955s
OK
$
Damn You, Emacs Autoinsert
Emacs can also be annoying when it autoinserts text without checking to see if it’s already there. Case in point, gettext .po files under Emacs 23.1. For whatever reason, my copy kept inserting this header whenever I opened a translation file:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR Free Software Foundation, Inc.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSIONn"
"PO-Revision-Date: 2012-12-10 05:06+0000n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>n"
"Language-Team: LANGUAGE <LL@li.org>n"
"MIME-Version: 1.0n"
"Content-Type: text/plain; charset=CHARSETn"
"Content-Transfer-Encoding: 8bitn"
|
If this header sneaks into a .po file, when you try to compile the file with Django’s compilemessages command, you get the following error:
|
1
2
3
4
5
6
7
|
$ django-admin.py compilemessages
processing file django.po in /home/user/project/app/locale/de/LC_MESSAGES
/home/user/project/app/locale/de/LC_MESSAGES/django.po: warning: Charset "CHARSET" is not a portable encoding name.
Message conversion to user's charset might not work.
/home/user/project/app/locale/de/LC_MESSAGES/django.po:21: duplicate message definition...
/home/user/project/app/locale/de/LC_MESSAGES/django.po:7: ...this is the location of the first definition
msgfmt: found 1 fatal error
|
What seems to happen is that this header is only inserted when you use C-x C-f to open a file. When opening a file directly from the command line, this corruption does not seem to occur.
If you look at the PO Group configuration options, this text is listed there as the default PO file header. I haven’t validated this as a solution, but you should be able to switch this value to be the comment character “#”, and that should take care of the problem.
Also, why the hell does the PO major mode not allow you to destroy entire msgid’s and their translations? This is really annoying. Yes, it’s nice sometimes when programs limit your options, but in this case, Emacs was messing up by inserting the bad header, then refusing me the option of removing it.
Custom Pystache Template And Loader Classes For Django
Update: I’ve posted the code at github here.
Wanting to keep my server-side and client-side templates identical, I noticed there wasn’t a good solution for using Mustache/Pystache/Handlebars templates alongside the Django templating language.
The existing one that I saw used Pystache internals, and I’m not even sure they’re still valid in the latest Pystache versions.
The other existing one that I saw let you do the equivalent of #include “something.mustache” within the Django templates, which wasn’t what I was looking for either.
So I went ahead and wrote a set of PystacheTemplate and Pystache*Loader classes which do it all within the specifications of the Django infrastructure, and, I set them up so that you can specify the exact file extensions you want them to operate on, i.e. just .mustache, .handlebars, and .hbs files by default. This doesn’t mean you can mix Mustache directly into Django templates, which doesn’t make sense (and is mildly redundant), but it does mean you can render complete Mustache template files directly, which is awesome if you’re wanting to share those templates or partials between client and server.
Copy this file somewhere into your project or app folders:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
|
"""
Copyright (c) 2012 Max Vilimpoc
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the "Software"),
to deal in the Software without restriction, including without limitation
the rights to use, copy, modify, merge, publish, distribute, sublicense,
and/or sell copies of the Software, and to permit persons to whom
the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
OR OTHER DEALINGS IN THE SOFTWARE.
"""
from django.template.loaders import app_directories, filesystem
from django.template.base import TemplateDoesNotExist
import pystache
import os
"""
Based on:
https://docs.djangoproject.com/en/dev/ref/templates/api/#django.template.Template
https://docs.djangoproject.com/en/dev/ref/templates/api/#loading-templates
https://docs.djangoproject.com/en/dev/ref/templates/api/#using-an-alternative-template-language
"""
class PystacheTemplate():
def __init__(self, templateString):
self.parsed = pystache.parse(templateString)
self.renderer = pystache.Renderer()
def render(self, context):
# Flatten the Django Context into a single dictionary.
flatContext = {}
for d in context.dicts:
flatContext.update(d)
return self.renderer.render(self.parsed, flatContext)
"""
Based on:
https://docs.djangoproject.com/en/dev/ref/templates/api/#loader-types
Yeah, it's two identical classes separated by a name.
Metaprogramming comes to mind.
According to django.template.loader.BaseLoader:
"A loader may return an already-compiled template instead of the actual
template source. In that case the path returned should be None, since the
path information is associated with the template during the compilation,
which has already been done."
---
We can make the Loader check the file extension as well,
so it will only parse Mustache-compatible files:
.handlebars, .hbs, .mustache, and so on.
This means you can actually keep using the normal Django loaders
too, for other file extensions!
---
The key piece is in django/template/loader.py
def find_template(name, dirs=None):
[...]
for loader in template_source_loaders:
try:
source, display_name = loader(name, dirs)
return (source, make_origin(display_name, loader, name, dirs))
except TemplateDoesNotExist:
pass
raise TemplateDoesNotExist(name)
[...]
Note:
This could still mess up if the custom loader passes
through a Handlebars template with an .html extension.
Later loaders would then throw a fit.
"""
EXTENSIONS = ['.handlebars', '.hbs', '.mustache']
class PystacheAppDirectoriesLoader(app_directories.Loader):
is_usable = True
def load_template(self, template_name, template_dirs=None):
# Only allow certain template types.
filename, extension = os.path.splitext(template_name)
if extension not in EXTENSIONS:
raise TemplateDoesNotExist
source, origin = self.load_template_source(template_name, template_dirs)
template = PystacheTemplate(source)
return template, None
class PystacheFilesystemLoader(filesystem.Loader):
is_usable = True
def load_template(self, template_name, template_dirs=None):
# Only allow certain template types.
filename, extension = os.path.splitext(template_name)
if extension not in EXTENSIONS:
raise TemplateDoesNotExist
source, origin = self.load_template_source(template_name, template_dirs)
template = PystacheTemplate(source)
return template, None
"""
Now update the settings.py file to use the custom Loaders,
putting them ahead of Django's default Loaders in the
TEMPLATE_LOADERS setting.
TEMPLATE_LOADERS = (
'project.templates.PystacheFilesystemLoader',
'project.templates.PystacheAppDirectoriesLoader',
[...]
)
"""
|
Then you can just update the settings.py file to use the custom Loaders, putting them ahead of Django’s default Loaders in the TEMPLATE_LOADERS setting:
|
1
2
3
4
5
|
TEMPLATE_LOADERS = (
'project.templates.PystacheFilesystemLoader',
'project.templates.PystacheAppDirectoriesLoader',
[...]
)
|
Then if you have, say, a file called index.handlebars somewhere in one of your app directory template/ folders or somewhere else in the TEMPLATE_DIRS list, you can just do this in one of your views:
|
1
2
|
def someView(request):
return render_to_response('index.handlebars', { 'title': 'Hello, Awesome.' })
|
And all will be good.
Please let me know in the comments if you have any problems with this. (Speed might one of them, I haven’t extensively tested this yet. But I plan to cache these using the Django template cache loader.)