5 An introduction to pyffi🔗

The pyffi library makes it possible to use Python libraries from a Racket program.

A Racket program can start a Python process by requiring pyffi and calling initialize. After the initialization run and run* can be used to evaluate expressions and statements in the Python process.

> (require pyffi)

> (initialize)

> (post-initialize)

> (run "1+2")

3

Here Racket starts an embed Python process. The Python "1+2" is parsed, compiled and evaluated by Python. The resulting Python value 3 is then converted to the Racket value 3.

5.1 Atomic values: numbers, Booleans and None🔗

Atomic Python values (numbers, Booleans and None) are automatically converted to their corresponding Racket values.

> (require pyffi)

> (initialize)

> (post-initialize)

> (run "12")

12

> (run "34.")

34.0

> (run "5+6j")

5.0+6.0i

> (run "False")

#f

> (run "True")

#t

> (list (run "None"))

'(#<void>)

5.2 Compound Values: strings, tuples, lists, and, dictionaries🔗

Compound (non-atomic) Python values such as strings, tuples, lists and dicts are not converted to Racket values. Instead they are wrapped in a struct named obj. Due to a custom printer handler these wrapped values print nicely.

> (run "'Hello World'")

(obj "str" : 'Hello World')

> (run "(1,2,3)")

(obj "tuple" : (1, 2, 3))

> (run "[1,2,3]")

(obj "list" : [1, 2, 3])

> (run "{'a': 1, 'b': 2}")

(obj "dict" : {'a': 1, 'b': 2})

The values display nicely too:

> (displayln (run "'Hello World'"))

Hello World

> (displayln (run "(1,2,3)"))

(1, 2, 3)

> (displayln (run "[1,2,3]"))

[1, 2, 3]

> (displayln (run "{'a': 1, 'b': 2}"))

{'a': 1, 'b': 2}

Printing and displaying a Python object use the __repr__ and __str__ methods of the object respectively.

The idea is that Racket gains four new data types: pystring, pytuple, pylist and pydict.

To convert a compound value use pystring->string, pytuple->vector, pylist->list or pydict->hash.

> (pystring->string (run "'Hello World'"))

"Hello World"

> (pytuple->vector (run "(1,2,3)"))

'#(1 2 3)

> (pylist->list (run "[1,2,3]"))

'(1 2 3)

> (pydict->hash (run "{'a': 1, 'b': 2}"))

'#hash(("a" . 1) ("b" . 2))

Similarly, you can convert Racket values to Python ones.

> (string->pystring "Hello World")

(obj "str" : 'Hello World')

> (vector->pytuple #(1 2 3))

(obj "tuple" : (1, 2, 3))

> (list->pylist '(1 2 3))

(obj "list" : [1, 2, 3])

> (hash->pydict (hash "a" 1 "b" 2))

(obj "dict" : {'b': 2, 'a': 1})

It’s important to note that creating Python values using string->pystring, vector->pytuple, list->pylist and hash->pydict is much more efficient than using run. The overhead of run is due to the parsing and compiling of its input string. In contrast string->pystring and friends use the C API to create the Python values directly.

The data types also have constructors:

> (pystring #\H #\e #\l #\l #\o)

(obj "str" : 'Hello')

> (pytuple 1 2 3)

(obj "tuple" : (1, 2, 3))

> (pylist 1 2 3)

(obj "list" : [1, 2, 3])

> (pydict "a" 1 "b" 2)

(obj "dict" : {'a': 1, 'b': 2})

The new types pystring, pytuple, pylist and pydict can be used with for.

> (for/list ([x (in-pystring (string->pystring "Hello"))]) x)

'(#\H #\e #\l #\l #\o)

> (for/list ([x (in-pytuple (vector->pytuple #(1 2 3)))]) x)

'(1 2 3)

> (for/list ([x (in-pylist (list->pylist '(1 2 3)))]) x)

'(1 2 3)

> (for/list ([(k v) (in-pydict (hash->pydict (hash "a" 1 "b" 2)))]) (list k v))

'(((obj "str" : 'b') 2) ((obj "str" : 'a') 1))

5.3 Builtin functions and modules🔗

> (run* "x = 1+2")

Here the statement x = 1+2 is parsed, compiled and executed. The result of the expression 1+2 is stored in the global variable x.

But due to the overhead of run it is better to make a direct variable reference.

> main.x

3

Here main is the name we have given to the module used for the global namespace of the Python interpreter. The dotted identifier main.x thus references the variable x in the global namespace.

Since Python modules are first class values, we can see their printed representations:

> main

(obj "module" : <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>)

> builtins

(obj "module" : <module 'builtins' (built-in)>)

> (builtins.abs -7)

7

> (builtins.list "Hello")

(obj "list" : ['H', 'e', 'l', 'l', 'o'])

> (builtins.range 2 5)

(obj "range" : range(2, 5))

> (builtins.list (builtins.range 2 5))

(obj "list" : [2, 3, 4])

If you find the name builtins too long, then you can give it a new, shorter name.

> (define b builtins)

> (b.abs -7)

7

If you access the abs functions directly, you get a callable object:

> b.abs

(obj callable "builtin_function_or_method" : <built-in function abs>)

A callable object can be used just like a normal Racket function:

> (map b.abs '(1 -2 3 -4))

'(1 2 3 4)

To use functions from the Python standard library, you need to import it before you can use it. The standard library sys provide a lot of system information. Let’s use it to find the version of the Python interpreter.

> (import sys)

> sys.version_info

(obj "version_info" : sys.version_info(major=3, minor=12, micro=3, releaselevel='final', serial=0))

The list of modules in The Python Standard Library is long, so let’s just try one more.

We want to print a text calendar for the current month.

The expression (calendar.TextCalendar) instantiates a TextCalendar object. The syntax (object .method arg ...) is used to invoke the method formatmonth with the arguments 2022 and 7 (for July).

The documentation for formatmonth shows its signature:

formatmonth(theyear, themonth, w=0, l=0)

The two first arguments theyear and themonth are positional arguments and the two last arguments w and l are keyword arguments both has 0 has as default value.

The keyword argument w specifies the width of the date columns. We can get full names of the week days with a width of 9.

> (displayln ((calendar.TextCalendar) .formatmonth 2022 7 #:w 9))

July 2022   Monday   Tuesday  Wednesday  Thursday   Friday   Saturday   Sunday                                              1         2         3      4         5         6         7         8         9        10     11        12        13        14        15        16        17     18        19        20        21        22        23        24     25        26        27        28        29        30        31

5.4 Objects, Callable objects, Functions, Methods and Properties🔗

All values in Python are represented as objects. This differs from Racket, where most data types (e.g. numbers and strings) aren’t objects.

In the data model used by Python, all objects have an identity, a type and a value. In practice the identity of a Python object is represented by its address in memory [the CPython implementation never moves objects].

To represent a Python object in Racket it is wrapped in an obj struct. The structure contains the type name as a string and a pointer to the object. The wrapper sets the structure property gen:custom-write to display, write and print the wrapped objects nicely.

The result of repr() is used to write an object.
The result of str() is used to display an object.

> (define s (string->pystring "foo"))

> (repr s)

"'foo'"

> (writeln s)

(obj "str" : 'foo')

> (str s)

"foo"

> (displayln s)

foo

Functions and in general callable objects support the well-known syntax f(a,b,c). Such callable objects are wrapped in an callable-obj struct, which has obj as a super type. The callable-obj use the struct property prop:procedure to make the wrapper applicable.

> (run* "def f(x): return x+1")

> (define f main.f)

> f

(obj callable "function" : <function f at 0xf5b8251f0040>)

> (f 41)

42

Function calls with keywords work as expected. A Python keyword is simply prefixed with #: to turn it into a Racket keyword, as this example shows:

> (run* "def hello(name, title='Mr'): return 'Hello ' + title + ' ' + name")

> (displayln (main.hello "Foo"))

Hello Mr Foo

> (displayln (main.hello #:title "Mrs" "Bar"))

Hello Mrs Bar

In order to illustrate methods, let’s look at the Calendar class in the calendar module.

> (import calendar)

> calendar.Calendar

(obj callable "type" : <class 'calendar.Calendar'>)

Calling the class gives us an instance object. We pass 0 to make Monday the first week day.

> (calendar.Calendar #:firstweekday 0)

(obj "Calendar" : <calendar.Calendar object at 0xf5b8251e1b20>)

One of the methods of a calendar object is monthdatescalendar.

> (define cal (calendar.Calendar #:firstweekday 0))

> cal.monthdatescalendar

(obj callable "method" : <bound method Calendar.monthdatescalendar of <calendar.Calendar object at 0xf5b8251b1940>>)

The syntax obj.method gives us a bound method, which we can call. Bound methods are wrapped in method-obj to make them applicable.

However, we can also invoke the monthdatescalendar method directly with the help of the syntax (obj .method argument ...).

> (pyfirst (pyfirst (cal .monthdatescalendar year month)))

(obj "date" : datetime.date(2022, 8, 29))

Method invocations can be chained. That is, if a method call returns an object, we can invoke a method on it. The fist element of a list can be retrieved by the pop method, so we can replace the two calls to pyfirst with two invocations of .pop.

> (cal .monthdatescalendar year month  .pop 0 .pop 0)

(obj "date" : datetime.date(2022, 8, 29))

Besides methods an object can have properties (attributes). The syntax is obj.attribute. Most Python objects carry a little documentation in the oddly named __doc__ attribute.

> (displayln cal.__doc__)

Base calendar class. This class doesn't do any formatting. It simply     provides data to subclasses.

5.5 Exceptions🔗

A Python exception raised during a pyffi-mediated call is converted to a Racket exception that carries the live Python exception object and its class. The conversion is lossless: every Python attribute (args, __cause__, __context__, __traceback__, __notes__, custom attributes the raiser attached, every method) remains reachable from Racket through the usual py-attr machinery, and the original exception can be re-raised back into Python with full identity preserved.

The hierarchy mirrors Python’s split between Exception and BaseException:

• Python Exception subclasses (ValueError, KeyError, RuntimeError, user-defined application errors, …) become exn:fail:pyffi:python — a subtype of exn:fail. The standard idiom (with-handlers ([exn:fail? handler]) ...) catches them.

• Python KeyboardInterrupt becomes exn:break:pyffi:python — a subtype of exn:break. Caught by exn:break? but not by exn:fail?, matching how an ordinary Racket break behaves.

• Python SystemExit is honoured directly with exit, mirroring how Python embeddings handle sys.exit.

• Python StopIteration continues to be returned as the symbol 'StopIteration for iterator drivers, so normal iteration paths do not pay exception-machinery cost.

The two new structs share four observable fields and a predicate/property surface that lets callers handle either flavour uniformly.

> (run "1/0")

run: Python exception occurred;

ZeroDivisionError: division by zero

File "<string>", line 1, in <module>

5.5.1 Catching by Python class🔗

Catch any Python exception:

> (with-handlers ([python-exception?                    (λ (e)                      (list (python-exception-type-name e)                            (exn-message e)))])     (run "int('not a number')"))

'("ValueError" "run: Python exception occurred;\n ValueError: invalid literal for int() with base 10: 'not a number'\n \n  File \"<string>\", line 1, in <module>\n")

The standard Racket idiom also works because exn:fail:pyffi:python is a subtype of exn:fail:

> (with-handlers ([exn:fail? (λ (_) 'caught)])     (run "{}['missing']"))

'caught

Dispatch on a specific Python class without parsing the message:

> (with-handlers ([(λ (e)                      (and (python-exception? e)                           (string=? (python-exception-type-name e)                                     "ValueError")))                    (λ (_) 'value-error)]                   [python-exception? (λ (_) 'other)])     (run "int('abc')"))

'value-error

5.5.2 Reaching the live Python object🔗

python-exception-value returns the live Python exception instance as an obj. Custom attributes the Python raiser attached (such as e.errno or e.filename) are reachable exactly as in Python:

> (run* "\nclass CustomError(Exception):\n    def __init__(self, msg, errno):\n        super().__init__(msg)\n        self.errno = errno\n")

> (with-handlers ([python-exception?                    (λ (e)                      (define v (obj-the-obj                                 (python-exception-value e)))                      (PyLong_AsLong                       (PyObject_GetAttrString v "errno")))])     (run* "raise CustomError('boom', 42)"))

42

The same access pattern reaches __cause__, __context__, the live traceback object via __traceback__ (walkable frame-by-frame), __notes__, and every method.

5.5.3 Round-tripping back into Python🔗

reraise-into-python re-installs a previously caught Python exception in the current thread’s Python error indicator, so the next Python call surfaces it exactly as if Racket had not intercepted. Identity, traceback, and chained __cause__/__context__ are all preserved.

raise-into-python installs a fresh Python exception of an arbitrary class for surfacing Racket-side failures to Python code as a typed Python exception.

5.5.4 Exception API🔗

class is the live Python class object (equivalent to type(e) in Python). value is the normalised live Python exception instance (equivalent to the e bound by except ... as e:). type-name is the Python class name as a string, pre-cached so dispatch on the class name avoids an FFI hop. traceback is the formatted traceback as a list of strings; the live traceback object is reachable via (PyObject_GetAttrString (obj-the-obj value) "__traceback__").

The continuation field inherited from exn:break is an escape continuation captured at the point of detection. Invoking it returns control past the failed Python call — the closest Racket analogue to "ignore the break and continue".

This struct is constructed only by pyffi’s internals; consumers test the predicate but do not call the constructor directly.

The pyffi/python-exception module provides the inverse direction: re-raising into Python with full fidelity.

This is the lossless inverse of the conversion that produced e: a Python except clause downstream of the re-raise sees the same object identity (is-check passes) and can introspect every attribute as if pyffi had never intercepted.

When value is supplied, it is used as the exception instance directly; otherwise the class is constructed with no arguments via class().

When cause is supplied, sets __cause__ on the new exception, mirroring raise New(...) from cause.

5.6 Calling Racket from Python🔗

The bridge from Racket to Python is symmetric. Where run / run* / the obj prop:procedure machinery let Racket call Python, the pyffi/python-callback module wraps a Racket procedure as a Python callable that any Python code can invoke.

This is what makes Python’s higher-order builtins (map, filter, sorted(key=…), reduce, …), library hooks (pandas apply, JSON default=, scikit-learn custom metrics, …) and callback-driven APIs (asyncio, GUI toolkits, signal handlers, plugin architectures) usable from Racket: Python sees an ordinary callable; calling it routes through a single C trampoline that invokes the Racket procedure with converted arguments and translates the result back.

> (define keep-positive? (racket-procedure->python positive?))

> (run* "\ndef _apply(pred, xs):\n    return list(filter(pred, xs))\n")

> ((run "_apply") keep-positive? (run "[1, -2, 3, -4, 5]"))

(obj "list" : [1, 3, 5])

The wrapping is shallow on the way in (Python str/int/float/bool arrive as Racket strings/numbers/booleans; compound types stay as obj) and shallow on the way out (Racket strings, numbers, booleans, lists, vectors, void/None convert directly; an obj returned passes through with refcount preserved). Exceptions raised by the Racket procedure surface to Python: a python-exception? re-enters Python losslessly via reraise-into-python; any other exn:fail? becomes a Python RuntimeError carrying the Racket exception message.

Each racket-procedure->python call interns the Racket procedure in a per-module hash so it remains live for as long as the Python wrapper exists; when Python finalises the wrapper, the hash entry is released and the Racket procedure becomes eligible for garbage collection.

Also exported under the alias py-lambda.