[Headdesk] In Django, Querysets are Lazy!

[elfs]

As I’ve been working on a project at Indieflix, I’ve been evaluating other people’s code, including drop-ins, and for the past couple of days a pattern has emerged that, at first, bugged the hell out of me. Django has these lovely things called context processors- they allow you to attach specific elements of code to the request context before the

Comments

[tamino]

Actually, building a QuerySet can be expensive even if it doesn't involve any database I/O. Part of it is the fact that when you do:

my_query = \
models.Foo.objects.filter(field1=val1) \
.exclude(field2=val2, field3=val3) \
.extra(where='func(field4, %s) < 0',
params=[val4])

you're actually creating (and garbage collecting) two QuerySets that you don't care about, before you create the one that you do care about. Django doesn't know that the intermediate ones are useless, so it has no choice but to flesh them out fully enough so that they could be evaluated if necessary. One of the "fleshing out" things it does is to call copy.deepcopy() on the data it inherited from its parent. For instance, in the example above, when it created the second, it would call copy.deepcopy() on val1. When it created the third, it would call copy.deepcopy() on val1, val2, and val3.

We profiled one particularly database-heavy piece of our code and found that it was spending 30% of its time in copy.deepcopy(), because of this!

Creating Q objects and and-ing and or-ing them together to reduce the number of times you have to refine a given QuerySet doesn't seem to help, either.

[elfs]

If you need that kind of performance, you're probably better off writing your own advanced extra(select=) pairings to contain the entirety of the SQL in a single call.

Django 1.2 seems to be much better at avoiding this kind of nonsense.

Also, I've discovered that if your querysets are lazy but even constructing is expensive, you can still avoid the construction costs when they're not needed by crafting a constructor into a functor or closure that doesn't get triggered until invoked by the template handler.

[Edit: Wow, that last paragraph is so buzzword-compliant, if I were hearing it from someone else I couldn't be sure if they were bullshitting me or not.]