title: How Hy backported "yield from" to Python 2
date: 2014-11-20 23:45
author: Christine Lemmer-Webber
tags: hy, python, macros
slug: how-hy-backported-yield-from-to-python2
---
Hello everyone! Time for a bit of a diversion towards one of my favorite
projects, [Hy](http://hy.readthedocs.org/en/latest/)! (I'm an occasional
committer, but the main mastermind behind the project is my good friend
[Paul Tagliamonte](http://pault.ag/).) For those of you who don't know,
Hy is a Lisp that transforms into the Python AST. Even more fun: you can
import .hy files in .py files and .py files in .hy files! Crazy!

Now, when many people hear that, they say, "Huh what, why on earth would
you do such a thing?" The usual response is something like, "Because
it's fun!" But today, dear readers, I am going to show you a real...
dare I say *practical* reason for using Hy. Because a cool feature just
landed in Hy: a backport of "[yield
from](https://docs.python.org/3/whatsnew/3.3.html#pep-380)" to Python 2.

Let's back up a bit. First, you might not know what "yield from" is or
why it's cool. Well, Python has this thing called coroutines which allow
you to do cool things, including suspending and resuming functions,
which it turns out is really great for writing asynchronous code (sure,
just wake me back up when we get the next network stanza, eh buddy?).
Once you provide the ability to nest together coroutines by "delegating
to subgenerators" (what "yield from" does that "yield" does not), this
stuff starts to get *really* powerful. This feature is so useful that
it's the basis of maybe the world's coolest asynchronous programming
environment,
[asyncio](https://docs.python.org/3.4/library/asyncio.html). Only one
problem: "yield from" didn't exist until Python 3.3... which means you
can't use it with Python 2. Bummer!

Or can you? Time for our second bit of context. Ever hear of a Lisp
programmer talk about something called a "macro"? No? Okay, think
harder. Maybe it was in one of those conversations where you were
talking about your favorite new your-pet-language feature, and the Lisp
hacker was like, "Oh that's cute... yeah Lisp had that decades ago." And
then you got really mad and brought up a bunch more features, and the
Lisp hacker kept saying that Lisp had them before you were born, and "Oh
yeah, and whatever features Lisp doesn't have, you can add really fast
because Lisp has macros. You can basically program any feature with
macros." Maybe you asked them, what the heck is a macro, and they said
something inane like "it's a feature where you can program other
features, or write code that writes code", but you barely remember,
because at that point you just wanted to punch them in their smug little
face. (And besides, you wondered, if you have higher order functions,
isn't that enough?)

Well friends, today it is my face that you will want to do the punching
to, because I'm about to show you how cool macros are and why having
them makes Hy so awesome. But, after the face punching thing, you'll
also thank me. (Also, please don't punch me in the face, this blogpost
is not consent for face-punching.)

Enough with the talk. Time for examples! Let's look at some code. Say
you have this Python 3.3+ code:

``` python
from awesomelib import IrcBot, bake_cookie, async

def irc_to_cookies(**connection_stuff):
    our_bot = IrcBot(**connection_stuff)
    yield from our_bot.open_connection()

    while True:
        message = yield from our_bot.get_next_message()

        if message.command == "bake_cookie":
            yield from async(bake_cookie())
```

Groovy. In our example above, we built a cookie baker that can be
plugged into our awesomelib asynchronous network library and cookie
baking pipeline system. (And of course, we wrote it in Hy, because we
love Hy, and you can still run Hy code in vanilla Python.) We're feeling
pretty good about this. We kind of wish we could run it in Python 2.X
still, but Python 3.3+ is the future anyway, and no use worrying about
the past really... right?

Our Hy example looks pretty similar:

``` scheme
(defn irc-to-cookies [&kwargs connection-stuff]
  (setv our-bot (apply IrcBot [] connection-stuff))
  (yield-from (.open-connection our-bot))

  (while True
    (setv message (yield-from (.get-next-message our-bot)))

    ;; If an irc user gives the "bake_cookie" command,
    ;; put a cookie in our ez_bake oven
    (if (= message.command "bake_cookie")
      (yield-from (async (bake-cookie))))))
```

But there's something magical... this code makes use of yield-from,
which in Python 3.3+ Hy just uses the actual real built in "yield from"
(or more accurately,
[ast.YieldFrom](https://docs.python.org/3.4/library/ast.html)). But what
about Python 2? Mere higher ordered function magic can't save us here.
We need a way to implement a new feature.

Except oh right, *this is a lisp*, and we have macros! So why not write
a macro for yield-from?

And it turns out that's [exactly what paultag
did](https://github.com/hylang/hy/pull/686):

``` scheme
(if-python2
  (defmacro/g! yield-from [expr]
    `(do (import types)
         (setv ~g!iter (iter ~expr))
         (setv ~g!return nil)
         (setv ~g!message nil)
         (while true
           (try (if (isinstance ~g!iter types.GeneratorType)
                  (setv ~g!message (yield (.send ~g!iter ~g!message)))
                  (setv ~g!message (yield (next ~g!iter))))
           (catch [~g!e StopIteration]
             (do (setv ~g!return (if (hasattr ~g!e "value")
                                     (. ~g!e value)
                                     nil))
               (break)))))
           ~g!return))
  nil)
```

This simple macro above is an implementation of yield-from which works
in Python 2. The macro is more or less a function that writes new code
to be expanded in place... allowing us to use basic building blocks of
the language to build more complex features. Since in Lisp, code is a
very simple, manipulatable data structure (lists!), we can literally
write out code that writes code without too much trouble. (There's some
magic going on with the \` character above, called backquoting... but
it's best to read a tutorial on macros if it's not clear to you how the
backquote is building the list of code there.) Hey look... we just
brought a feature back to the future... as long as we're writing code in
Hy, we can do subgenerator delegation with yield-from. Cool! That sure
makes coroutines a lot more useful to those of us living in the past.

So wait, does this mean you can now use asyncio with Python 2? Well, not
quite... asyncio is written in normal Python syntax, which means that
it's using "yield from", not our more versatile "yield-from", and the
library itself isn't written to support Python 2.7. So, no. (But, if
asyncio was written in Hy, we could, even though Python 2.7 doesn't have
"yield from"!)

The real goal of this article isn't to convince you to start backporting
features to Python 2.X via Hy, though. Really, Python 3 is the future,
write Python 3 code! But the point here is to get you thinking about how
having macros allows you to implement new features *now*! Why wait for
the features of Python 4.X? In Hy you can have them now! (Or even start
prototyping them today!)

And *that's* worth getting excited about. And once you realize that,
it's a bit easier to understand where lispers are coming from when they
nerd out about how cool macros are. (Even if you still want to punch us
in the face.)

(PS: think this is pretty cool? Hy is a really welcoming community, and
there's a lot of fun stuff to do! Learn about language implementation
and learn about lisp in a paradoxically fun and pythonic environment!
We'd love to have you [join in hacking with
us](http://hy.readthedocs.org/en/latest/hacking.html)!)
