I have decidedly mixed feelings on Python decorators, but tend to shy away from them. The parts I like are that they can be a good way to reuse certain functions and they make surface level code much more concise and readable. The trouble comes when you need to delve into a stack of decorators to try to untangle some logic. Bouncing between decorators feels harder to track than a set of functions or classes.
One project I worked on used decorators heavily. Things worked great on the happy path but trying to track a subtle bug happening in a function with nine complex decorators is not a fun experience.
It's absolutely baffling that an article called demystifying decorators does not actually describe decorators at all, preferring to keep them a mystery. Decorators are nothing more than a special syntax for calling a function, the meat of which can easily be explained in a single paragraph including a supporting example.
> Congratulations! You wrote your first decorator. Even if it looks different to what you think a decorator should look like—you usually see them used with the @ notation, which we'll discuss in Part 3—store_arguments() is a decorator.
And:
> A decorator is a function that accepts another function as an argument and returns yet another function. The function it returns is a decorated version of the function you pass as an argument. (We'll return to this definition and refine it later in this decorator journey)
I have no idea why you are claiming something that is plainly false based on the text of the article.
More bafflement: is this refering to the programming decorator pattern or something else? I.e. the one everyone learns about when they learn patterns for the first time??
This is about Python decorators: https://docs.python.org/3/glossary.html#term-decorator. In summary, they are functions that return functions, which allow you to wrap their behaviour (e.g. add a cache, transform a function into an HTTP API handler, etc.)
As far as I can tell, they're not related to the design pattern, but I never had to use that.
I wonder if the people that struggle with decorators have a fundamental struggle with seeing functions as objects that can be redefined or modified.
A decorator basically just does something when a function is defined. They can be used to do something when a function is defined (such as register the function with a list of functions), or modify the function's behavior to do something before and/or after the function is called (like create a poor man's profiling by timing the function call).
You can do all that without decorators, of course, but the code looks a lot cleaner with them.
That's a long read, but it is an awkward thing tbh.
My take on it is that, if you're going to write a decorator, make sure you test it thoroughly and be sure to cover the ways that it will actually be used.
To properly handle a decorator that needed to be called both with and without arguments, I ended up writing a function decorator that returned a wrapped function when there were no parameters provided for the decorator (*args param to the decorator function undefined) and, otherwise, returned an instance of a class that defined __call__ and __init__ methods.
I'm still a bit surprised that I had to do it that way, but that's how it shook out.
Is there any good resource about class decorators specifically? It is more common to see function decorators in the wild, so my knowledge about the class ones is negligible at best
I can't name one off the top of my head, but conceptually they're the same because classes are also first-class in Python (i.e. they can be passed around as function arguments and even modified).
Classes in Python are actually themselves instances (of builtin class 'type'). So to make a decorator for one, you create a function that takes a class (i.e. an instance of 'type' with a lot of duck-typing already applied to it in the class definition) as an argument and returns a class (either the same class or a brand-new one; usually you make a brand new one by creating a new class inside the decorator that subclasses the class passed in, `MyNewClass(cls)`, and you return `MyNewClass`... The fact it's named `MyNewClass` inside the decorator won't matter to anyone because that's a local variable inside the decorator function body).
The syntactic sugar Python does when you go
@decorator
class Foo:
... is basically:
* Create a class (with no name yet)
* Pass that class to function `decorator`
* Bind the return value of `decorator` to the variable `Foo`.
---
Before decorators came along, you'd get their effects on classes with this pattern:
class Foo:
# the definition of Foo
Foo.some_new_method = ... # since Foo is also an instance of a class, you can just modify it in-place.
Foo = decorate_it(Foo) # ... or pass it as an argument to a function and re-bind its name to the result
Thanks for the detailed explanation! I assume that essentially, the same rules as with function decorators apply(in regards to parameterizing them etc.) but the main difference is that we accept the class object and then return the class object vs the function object and that's it?
outside of __init__, I thought it wasn't the pythonic way to directly invoke double underscore functions like closure? I recall reading that you should implement a class that overrides/implements such functions instead? I get why the author might be doing it this way to describe the topic of the post, but would calling __closure__() be acceptable in production code?
First, `__closure__` is a plain data attribute, not a function. It isn't callable, and isn't being "called" here.
Further, in the cases you're talking about, the functions specifically are methods which you can implement in your own class. But you aren't doing this in order to avoid calling them directly; rather, you're doing them to implement functionality for that class (i.e., there's some other bit of syntax already which will call it indirectly for you). And `__init__` isn't much of an exception; normally you only call it explicitly where necessary for subclassing (because the corresponding syntax would create a separate base class instance instead of initializing the current base).
But the point here is to inspect an implementation detail, for pedagogical purposes. There isn't a better way to do it in this case, exactly because you aren't ordinarily supposed to care about that detail. There's no question about whether you'd inspect an object's `__closure__` directly in production code - because not only is there no alternative, but it would be extremely rare to have any reason to do so.
The naming convention is what is throwing me off. "object.closure" I understand as an attribute/property being accessed. In this case, the naming style suggests there should be a wrapper like:
def closure(self):
return self.__closure__
I'm just talking form, not function here. In other words, if it is supposed to be accessed directly by arbitrary external code (non-class functions), it shouldn't use the double underscore syntax? If any property can have that syntax, then the syntax loses its meaning?
>In other words, if it is supposed to be accessed directly by arbitrary external code
There is no such thing as "supposed to be" in this context. It can be accessed, because Python fundamentally doesn't protect against such accesses anywhere. There is no wrapper because there is no ordinary purpose for the access.
>If any property can have that syntax, then the syntax loses its meaning?
There is no special syntax here, so there is nothing that can lose meaning. Leading underscores are a convention. The parser doesn't care, and the compiler only makes minor adjustments (name mangling) in very limited circumstances (to avoid mistakes with subclasses).
One thing I've learned writing documentation (and struggling) is that, when you find yourself writing at length, splitting text into sections, and peppering a copious amount of examples, you need to stop and do 2 things:
- Audit for conciseness
- Add a tl;dr summary at the top
And I do wish this had a real example it was playing with, and it consistently used that same example all the way through as it built the layers of what the problem being solved with decorators is. It's a lot easier to teach a concept when you can contextualize it in a way that's shows it's usefulness.
I'm not sure if this metric really matters, but this is wordier than the PEP that describes decorators.
I'm not sure who this article is for, but I think it doesn't strike at the heart of the topic.
Half of the article is devoted to closures, but closures aren't essential for decorators. And the __closure__ attribute is an implementation detail that is really irrelevant. (For comparison, JavaScript has closures just like Python, but it doesn't expose the closed-over variables explicitly the way Python does.)
Decorators are simply higher order functions that are used to wrap functions. The syntax is a little funky, but all you need to know is that code like:
@foo
@bar
def baz(args):
return quux(args)
Is essentially equivalent to:
baz = foo(bar(lambda args: quux(args))
...i.e. decorators are functions that take a callable argument and return a new callable (which typically does something and then calls the argument function--or not, as the case may be).
Then there are seemingly more complex expressions like:
@foo(42, 'blub')
def bar(..): ...
This looks like a special kind of decorator that takes arguments, but it looks more complex than it really is. Just like `foo` was an expression referencing a function `foo(42, 'blub')` is just a regular Python function call expression. That function call should then itself return a function, which takes a function argument to wrap the function being decorated. Okay, I admit that sounds pretty complex when I write it out like that, but if you implement it, it's again pretty simple:
This is an extra level of indirection but fundamentally still the same principle as without any arguments.
And yes, these examples use closures, which are very convenient when implementing decorators. But they aren't essential. It's perfectly possible to declare a decorator this way:
class Invoker:
def __init__(self, f):
self.f = f
def __call__(self):
print('before call')
self.f()
print('after call')
def decorator(f):
return Invoker(f)
@decorator
def hello():
print('Hello, world!')
hello()
# prints:
# before call
# Hello, world!
# after call
It's the same thing but now there are no closures whatsoever involved.
The key point in all these examples is that functions in Python are first-class objects that can be referenced by value, invoked dynamically, passed as arguments to functions, and returned from functions. Once you understand that, it's pretty clear that a decorator is simply a wrapper that takes a function argument and returns a new function to replace it, usually adding some behavior around the original function.
> decorators are functions that take a callable argument and return a new callable
there's nothing forcing a decorator to return a callable. A decorator _could_ return anything it wants. I don't know why you would want that, but Python won't stop you.
I think you overestimate how many programmers would find this explanation clear. Most programmers are not used to functional programming, an anything that manipulates with functions is not that intuitive to them. That is the reason why some people steer away from decorators
The reference I usually offer people is https://stackoverflow.com/questions/739654/ . Admittedly the encyclopedic length answer there, while historically highly praised, barely addresses the actual question. But there's no real way to ask a suitable question for that answer - it's just way too broadly scoped. It's just one of those artifacts of the old days of Stack Overflow. Just, you know, good luck finding it if you don't already know about it.
the thing that bothered me most reading through it was using decorators to mutate some global state with the `data` list variable.
like… it… just… it felt wrong reading that in the examples. felt very `def func(kw=[])` adjacent. i can see some rare uses for it, but eh. i dunno.
(also didn’t find the closure stuff that insightful, ended up skipping past that, but then i know decorators, so… maybe useful for someone else. i dunno.).
> it was using decorators to mutate some global state
It isn't. The original version was doing that, but the "decorator" one wasn't. The data variable is internal to the closure, so different invocations of the decorator would have different data variables.
> didn’t find the closure stuff that insightful
It's used to attach state to a function. Different invocations of the function would have different state. IME, I'd rather use an explicit class for that, but it's useful with decorators.
My "proudest" use of decorators has been in adding metrics gathering for functions in python, so you'd get automated statsd (and later prometheus) metrics just by having @tracked (or whatever I had its name be - it's been like 7 years) on the function header.
In a sense, that was mutating a global variable by including and tracking the metrics gathering. I imagine this person's early professional exposures to it and need to create their own also came from a similar situation, so "mutating global state" and closures sorta clicked for them.
People learn things by coming to those things from many different entry points and for many different reasons. This is another one of those instances :)
I'm getting old. Something so obvious that I thought everybody knew is getting reiterated in a blogpost by a younger generation encountering it for the first time.
You're certainly getting older :). You're assuming that since the author writes about something obvious, they must belong to the "younger generation" and encountered it for the first time.
Meanwhile, the author finished their PhD in 2004, and wrote 3 books about Python.
Which 3 books are you talking about? I can only find 1 on Amazon (and it has a grand total of 12 reviews, so... you know, not exactly a best seller).
Frankly it's a bit suspicious how defensive you are of this author, and combined with the blatant downvoting of my toplevel comment, makes me think there is some astro-turfing going on in this thread.
I have decidedly mixed feelings on Python decorators, but tend to shy away from them. The parts I like are that they can be a good way to reuse certain functions and they make surface level code much more concise and readable. The trouble comes when you need to delve into a stack of decorators to try to untangle some logic. Bouncing between decorators feels harder to track than a set of functions or classes.
One project I worked on used decorators heavily. Things worked great on the happy path but trying to track a subtle bug happening in a function with nine complex decorators is not a fun experience.
It's absolutely baffling that an article called demystifying decorators does not actually describe decorators at all, preferring to keep them a mystery. Decorators are nothing more than a special syntax for calling a function, the meat of which can easily be explained in a single paragraph including a supporting example.
It most certainly does.
> Congratulations! You wrote your first decorator. Even if it looks different to what you think a decorator should look like—you usually see them used with the @ notation, which we'll discuss in Part 3—store_arguments() is a decorator.
And:
> A decorator is a function that accepts another function as an argument and returns yet another function. The function it returns is a decorated version of the function you pass as an argument. (We'll return to this definition and refine it later in this decorator journey)
I have no idea why you are claiming something that is plainly false based on the text of the article.
More bafflement: is this refering to the programming decorator pattern or something else? I.e. the one everyone learns about when they learn patterns for the first time??
No idea why they would be related to closures
This is about Python decorators: https://docs.python.org/3/glossary.html#term-decorator. In summary, they are functions that return functions, which allow you to wrap their behaviour (e.g. add a cache, transform a function into an HTTP API handler, etc.)
As far as I can tell, they're not related to the design pattern, but I never had to use that.
They implement the design pattern.
I wonder if the people that struggle with decorators have a fundamental struggle with seeing functions as objects that can be redefined or modified.
A decorator basically just does something when a function is defined. They can be used to do something when a function is defined (such as register the function with a list of functions), or modify the function's behavior to do something before and/or after the function is called (like create a poor man's profiling by timing the function call).
You can do all that without decorators, of course, but the code looks a lot cleaner with them.
That's a long read, but it is an awkward thing tbh.
My take on it is that, if you're going to write a decorator, make sure you test it thoroughly and be sure to cover the ways that it will actually be used.
To properly handle a decorator that needed to be called both with and without arguments, I ended up writing a function decorator that returned a wrapped function when there were no parameters provided for the decorator (*args param to the decorator function undefined) and, otherwise, returned an instance of a class that defined __call__ and __init__ methods.
I'm still a bit surprised that I had to do it that way, but that's how it shook out.
I think much of the challenge people have with decorators is not in understanding how they work, but in identifying when they are useful.
Is there any good resource about class decorators specifically? It is more common to see function decorators in the wild, so my knowledge about the class ones is negligible at best
I can't name one off the top of my head, but conceptually they're the same because classes are also first-class in Python (i.e. they can be passed around as function arguments and even modified).
Classes in Python are actually themselves instances (of builtin class 'type'). So to make a decorator for one, you create a function that takes a class (i.e. an instance of 'type' with a lot of duck-typing already applied to it in the class definition) as an argument and returns a class (either the same class or a brand-new one; usually you make a brand new one by creating a new class inside the decorator that subclasses the class passed in, `MyNewClass(cls)`, and you return `MyNewClass`... The fact it's named `MyNewClass` inside the decorator won't matter to anyone because that's a local variable inside the decorator function body).
The syntactic sugar Python does when you go
... is basically:* Create a class (with no name yet)
* Pass that class to function `decorator`
* Bind the return value of `decorator` to the variable `Foo`.
---
Before decorators came along, you'd get their effects on classes with this pattern:
Thanks for the detailed explanation! I assume that essentially, the same rules as with function decorators apply(in regards to parameterizing them etc.) but the main difference is that we accept the class object and then return the class object vs the function object and that's it?
Yes, pretty much.
outside of __init__, I thought it wasn't the pythonic way to directly invoke double underscore functions like closure? I recall reading that you should implement a class that overrides/implements such functions instead? I get why the author might be doing it this way to describe the topic of the post, but would calling __closure__() be acceptable in production code?
First, `__closure__` is a plain data attribute, not a function. It isn't callable, and isn't being "called" here.
Further, in the cases you're talking about, the functions specifically are methods which you can implement in your own class. But you aren't doing this in order to avoid calling them directly; rather, you're doing them to implement functionality for that class (i.e., there's some other bit of syntax already which will call it indirectly for you). And `__init__` isn't much of an exception; normally you only call it explicitly where necessary for subclassing (because the corresponding syntax would create a separate base class instance instead of initializing the current base).
But the point here is to inspect an implementation detail, for pedagogical purposes. There isn't a better way to do it in this case, exactly because you aren't ordinarily supposed to care about that detail. There's no question about whether you'd inspect an object's `__closure__` directly in production code - because not only is there no alternative, but it would be extremely rare to have any reason to do so.
The naming convention is what is throwing me off. "object.closure" I understand as an attribute/property being accessed. In this case, the naming style suggests there should be a wrapper like:
def closure(self): return self.__closure__
I'm just talking form, not function here. In other words, if it is supposed to be accessed directly by arbitrary external code (non-class functions), it shouldn't use the double underscore syntax? If any property can have that syntax, then the syntax loses its meaning?
>In other words, if it is supposed to be accessed directly by arbitrary external code
There is no such thing as "supposed to be" in this context. It can be accessed, because Python fundamentally doesn't protect against such accesses anywhere. There is no wrapper because there is no ordinary purpose for the access.
>If any property can have that syntax, then the syntax loses its meaning?
There is no special syntax here, so there is nothing that can lose meaning. Leading underscores are a convention. The parser doesn't care, and the compiler only makes minor adjustments (name mangling) in very limited circumstances (to avoid mistakes with subclasses).
One thing I've learned writing documentation (and struggling) is that, when you find yourself writing at length, splitting text into sections, and peppering a copious amount of examples, you need to stop and do 2 things:
- Audit for conciseness
- Add a tl;dr summary at the top
And I do wish this had a real example it was playing with, and it consistently used that same example all the way through as it built the layers of what the problem being solved with decorators is. It's a lot easier to teach a concept when you can contextualize it in a way that's shows it's usefulness.
I'm not sure if this metric really matters, but this is wordier than the PEP that describes decorators.
I'm not sure who this article is for, but I think it doesn't strike at the heart of the topic.
Half of the article is devoted to closures, but closures aren't essential for decorators. And the __closure__ attribute is an implementation detail that is really irrelevant. (For comparison, JavaScript has closures just like Python, but it doesn't expose the closed-over variables explicitly the way Python does.)
Decorators are simply higher order functions that are used to wrap functions. The syntax is a little funky, but all you need to know is that code like:
Is essentially equivalent to: ...i.e. decorators are functions that take a callable argument and return a new callable (which typically does something and then calls the argument function--or not, as the case may be).Then there are seemingly more complex expressions like:
This looks like a special kind of decorator that takes arguments, but it looks more complex than it really is. Just like `foo` was an expression referencing a function `foo(42, 'blub')` is just a regular Python function call expression. That function call should then itself return a function, which takes a function argument to wrap the function being decorated. Okay, I admit that sounds pretty complex when I write it out like that, but if you implement it, it's again pretty simple: This is an extra level of indirection but fundamentally still the same principle as without any arguments.And yes, these examples use closures, which are very convenient when implementing decorators. But they aren't essential. It's perfectly possible to declare a decorator this way:
It's the same thing but now there are no closures whatsoever involved.The key point in all these examples is that functions in Python are first-class objects that can be referenced by value, invoked dynamically, passed as arguments to functions, and returned from functions. Once you understand that, it's pretty clear that a decorator is simply a wrapper that takes a function argument and returns a new function to replace it, usually adding some behavior around the original function.
One tiny correction:
> decorators are functions that take a callable argument and return a new callable
there's nothing forcing a decorator to return a callable. A decorator _could_ return anything it wants. I don't know why you would want that, but Python won't stop you.
They also don’t have to act on callables, see @dataclass for instance.
Classes are callable.
You can abuse this fact with type hunting to get a ParamSpec of a class's __init__ arguments.
But yeah type systems get weird at the margins. Your class is an instance of type which has a __call__ method which creates an instance of your class.
> Is essentially equivalent to...
I think you overestimate how many programmers would find this explanation clear. Most programmers are not used to functional programming, an anything that manipulates with functions is not that intuitive to them. That is the reason why some people steer away from decorators
The reference I usually offer people is https://stackoverflow.com/questions/739654/ . Admittedly the encyclopedic length answer there, while historically highly praised, barely addresses the actual question. But there's no real way to ask a suitable question for that answer - it's just way too broadly scoped. It's just one of those artifacts of the old days of Stack Overflow. Just, you know, good luck finding it if you don't already know about it.
the thing that bothered me most reading through it was using decorators to mutate some global state with the `data` list variable.
like… it… just… it felt wrong reading that in the examples. felt very `def func(kw=[])` adjacent. i can see some rare uses for it, but eh. i dunno.
(also didn’t find the closure stuff that insightful, ended up skipping past that, but then i know decorators, so… maybe useful for someone else. i dunno.).
> it was using decorators to mutate some global state
It isn't. The original version was doing that, but the "decorator" one wasn't. The data variable is internal to the closure, so different invocations of the decorator would have different data variables.
> didn’t find the closure stuff that insightful
It's used to attach state to a function. Different invocations of the function would have different state. IME, I'd rather use an explicit class for that, but it's useful with decorators.
My "proudest" use of decorators has been in adding metrics gathering for functions in python, so you'd get automated statsd (and later prometheus) metrics just by having @tracked (or whatever I had its name be - it's been like 7 years) on the function header.
In a sense, that was mutating a global variable by including and tracking the metrics gathering. I imagine this person's early professional exposures to it and need to create their own also came from a similar situation, so "mutating global state" and closures sorta clicked for them.
People learn things by coming to those things from many different entry points and for many different reasons. This is another one of those instances :)
I'm getting old. Something so obvious that I thought everybody knew is getting reiterated in a blogpost by a younger generation encountering it for the first time.
You're certainly getting older :). You're assuming that since the author writes about something obvious, they must belong to the "younger generation" and encountered it for the first time.
Meanwhile, the author finished their PhD in 2004, and wrote 3 books about Python.
Maybe not "by", but "for" a younger generation.
Which 3 books are you talking about? I can only find 1 on Amazon (and it has a grand total of 12 reviews, so... you know, not exactly a best seller).
Frankly it's a bit suspicious how defensive you are of this author, and combined with the blatant downvoting of my toplevel comment, makes me think there is some astro-turfing going on in this thread.
Decorators in Python are not bad at all.
Annotations in Java, on the other hand, which have the same syntax, are a monument to all our sins.
Jdjfm df