This post is a while overdue, but it’s been a busy summer for everyone!

In June, several of us got together for the “annual” (well, we’ve had it twice now) Scientific Python developer summit in Seattle. Our friends at the eScience Institute were again kind enough to host us. This time around, we made the event a bit shorter to avoid clashing with graduation.

As before, the Developer Summits are for members of the community to come together, in person, so they can work. Of course, we work together already online, but the event allows us to focus our energies on cross-project concerns (that often fall by the wayside) with planning and intent.

This is why, before the summit, we have planning Zoom calls, where we identify topics of interest, which are turned into issues, which are then fleshed out and discussed prior to the event. That way, we hoped to hit the ground running—as we did!

Topics#

You can get a rough idea of what we worked on by browsing the planning issues and the summit worklog.

Broad topics included SPECs, documentation, tools & bots, lectures, scipy.sparse, telemetry, Array API, and type annotation.

Documentation#

Documentation was a much more popular topic than anticipated!

• The new mystmd tooling generated some excitement, and an experimental port of the NumPy tutorials was made by Melissa and Ross.
• Recommendations on consistent use of backticks and monospaced font were submitted to numpydoc and pydata-sphinx-theme, respectively.
• Madicken, Paul, and Dan worked together to extend PyData Sphinx Theme’s testing infrastructure, by combining Sphinx Build Factory (for generating small test sites) with Playwright (for browser automation).
• Eric and Elliott fixed an intersphinx issue in sphinx-gallery.

SPECs#

The Scientific Python Ecosystem Coordination documents (SPECs) aim to improve coordination of technical development across the ecosystem.

Several new SPECs were started:

• SPEC-?: Dispatching (spatch)
• SPEC-8: Securing The Release Process
• SPEC-9: Governance
• SPEC-10: Changelog and release documentation
• SPEC-12: Formatting mathematical expressions
• SPEC-13: Naming conventions

Some existing SPECs were discussed and improved:

• SPEC-7: Seeding pseudo-random number generation (SPRaNG)

Matplotlib endorsed several SPECs.

Tooling#

• We created a new tools team to handle the ever-growing list of tools we maintain.
• Eric added his circleci-artifacts-redirector-action to the suite.
• Matthias brought over his backport bot and set up a maintenance team.

SciPy#

Several of the SciPy developers were present, and we used the opportunity to celebrate Dan Schult joining as a core developer 🎉! Matt and Pamphile did some work on the new distribution infrastructure, Dan worked on sparse (remotely with CJ), and a PR adding newly-supported const statements to Cython code got reviewed and merged. Eric isolated a non-deterministic bug in Sphinx that was impacting parallel builds of SciPy’s documentation. He found a work-around that had been eluding the team for months!

Unplanned collaborations#

As is the nature of these events, some collaborations arise spontaneously. For example:

• Nick and Ariel explored using Awkward Array for neuro-tractography.
• Nick and Mridul explored using scipp for high-energy physics data.
• Guen worked on telemetry.
• Inessa and Sanket discussed best practices for community surveys and project governance.
• Sebastian and Thomas discussed parallelization APIs.
• Inessa, with input from Tim and Thomas, finalized the design of the 2024 scikit-learn user survey.
• Erik and Dan discussed index compression options for CSR-like N-D sparse arrays.

Conclusion#

Numerous other PRs were made, of which a number were probably not even captured in the worklog. But, besides the inherent satisfaction of working together with this great group, the best feature of the summit was that we were able to hang out, bonding over our communal joys and struggles—both technical and personal.

We are grateful to the ecosystem developers who gave up their time to attend the summit (many had to put in leave just to do more work!). The summits are valuable, and translate to a lot of work getting done and decisions being made. We hope that there will be more on the horizon!

One outcome of the 2023 Scientific Python Developer Summit was the Scientific Python Development Guide, a comprehensive guide to modern Python package development, complete with a new project template supporting 10+ build backends and a WebAssembly-powered checker with checks linked to the guide. The guide covers topics like modern, compiled, and classic packaging, style checks, type checking, docs, task runners, CI, tests, and much more! There also are sections of tutorials, principles, and some common patterns.

This guide (along with cookie & repo-review) started in Scikit-HEP in 2020. During the summit, it was merged with the NSLS-II guidelines, which provided the basis for the principles section. I’d like to thank and acknowledge Dan Allan and Gregory Lee for working tirelessly during the summit to rework, rewrite, merge, and fix the guide, including writing most of the tutorials pages and first patterns page, and rewriting the environment page as a tutorial.

The guide#

The core of the project is the guide, which is comprised of four sections:

• Tutorials: How to go from “research” code to a basic package, for beginning readers.
• Topical guides: The core of the guide, for intermediate to advanced readers.
• Principles: Some general principles from the NSLS-II guide.
• Patterns: Recipes for common situations. Three pages are there now; including data, backports, and exports.

From the original Scikit-HEP dev pages, a lot was added:

• Brand new guide page on documentation, along with new sp-repo-review checks to help with readthedocs.
• A compiled projects page! Finally! With scikit-build-core, meson-python, and maturin. The page shows real outputs from the cookiecutter, kept in sync with cog (a huge benefit of using a single repo for all three components!)
• Big update to GHA CI page including a section on Composite Actions given at the Dev summit.
• CLI entry points are now included.
• Python 3.12 support added, Python 3.7 dropped.
• New sp-repo-review badges throughout (more on that later!)
• Updates for Ruff’s move and support for requires-python.
• Lots of additions for GitHub Actions.

The infrastructure was updated too:

• Using latest Jekyll (version 4) and latest Just the Docs theme. More colorful callout boxes. Plugins are used now.
• Live PR preview (provided by probably the world’s first readthedocs Jekyll build!). Developed with the Zarr developers during the summit.
• Better advertising for cookie and sp-repo-review on the index page(s).
• Auto bump and auto-sync CI jobs.

Cookie#

We also did something I’ve wanted to do for a long time: the guide, the cookiecutter template, and the checks are all in a single repo! The repo is scientific-python/cookie, which is the moved scikit-hep/cookie (the old URL for cookiecutter still works!).

Cookie is a new project template supporting multiple backends (including compiled ones), kept in sync with the dev guide. We recommend starting with the dev guide and setting up your first package by hand, so that you understand what each part is for, but once you’ve done that, cookie allows you to get started on a new project in seconds.

A lot of work went into cookie, too!

• Generalized defaults. We still have special integration if someone sets the org to scikit-hep; the same integration can be offered to other orgs.
• All custom hooks removed; standard jinja now used throughout templates. Using cookiecutter computed variables idiom too. Windows still fully supported and tested. Adding new choices is much easier now.
• Added cruft (a cookiecutter updater) testing.
• Dual-supporting copier now too, a cookiecutter replacement with a huge CLI improvement (and also supports updating). Might be the first project to support both at the same time. CI (or nox locally) checks to ensure generation is identical. Much better interface with copier, including validation, descriptive text, arrow selection, etc.
• Improved CLI experience even if using cookiecutter (like no longer requesting current year).
• Reworked docs template.
• Support for cookiecutter 2.2 pretty descriptions (added about four hours after cookiecutter 2.2.0 was released) and cookiecutter 2.2.3 choice descriptions.
• GitLab CI support when not targeting github.com URLs (added by Giordon Stark).
• Support for selecting VCS or classic versioning.

Repo-review#

See the introduction to repo-review for information about this one!

Along with this was probably the biggest change, one requested by several people at the summit: scientific-python/repo-review (was scikit-hep/repo-review) is now a completely general framework for implementing checks in Python 3.10+. The checks have been moved to sp-repo-review, which is now part of scientific-python/cookie. There are too many changes to list here, so just the key ones in 0.6, 0.7, 0.8, 0.9, and 0.10:

• Extensive, beautiful documentation for check authors at (used to help guide the new docs guide page & template update!)
• Support for four output formats, rich (improved), svg, json (new), html (new).
• Support for listing all checks.
• GitHub Actions support with HTML step summary, pre-commit support.
• Generalized topological sorting to fixtures, dynamic fixtures.
• Dynamic check selection (via fixtures! Basically everything is powered by fixtures now.)
• URL support in all output formats (including the terminal and WebApp!)
• Support for package not at root of repo.
• Support for running on a remote repo from the command line.
• Support for select/ignore config in pyproject.toml or command line.
• Pretty printed and controllable sorting for families.
• Supports running from Python, including inside a readme with something like cog.
• Support for dynamic family descriptions (such as to output build system and licence used).
• Support for limiting the output to just errors or errors and skips.
• Support for running on multiple repos at once, with output tailored to multiple repos. Also supports passing pyproject.toml path instead to make running on mixed repos easier.
• Support for linting [tool.repo-review] with validate-pyproject.

The full changelog has more - you can even see the 10 beta releases in-between 0.6.x and 0.7.0 where a lot of this refactoring work was happening. If you have configuration you’d like to write check for, feel free to write a plugin!

validate-pyproject 0.14 has added support for being used as a repo-review plugin, so you can validate pyproject.toml files with repo-review! This lints [project] and [build-system] tables, [tool.setuptools], and other tools via plugins. Scikit-build-core 0.5 can be used as a validate-project plugin to lint [tool.scikit-build]. Repo-review has a plugin for [tool.repo-review].

sp-repo-review#

Finally, sp-repo-review contains the previous repo-review plugins with checks:

• Fully cross-linked with the development guide. Every check has a URL that points to a matching badge inside the development guide where the thing the check is looking for is being discussed!
• Full list of checks (including URLs), produced by cog, in readme.
• Also ships with GitHub Action and pre-commit checks
• Released (in sync with cookie & guide, as they are in the same repo) as CalVer, with release notes.
• Split CI that selects just want to run based on changed files, with green checkmark that respects skips (based on the excellent contrition to pypa/build).

Using the guide#

If you have a guide, we’d like for you to compare it with the Scientific Python Development Guide, and see if we are missing anything - bring it to our attention, and maybe we can add it. And then you can link to the centrally maintained guide instead of manually maintaining a complete custom guide. See scikit-hep/developer for an example; many pages now point at this guide. We can also provide org integrations for cookie, providing some customizations when a user targets your org (targeting scikit-hep will add a badge).

The first Scientific Python Developer Summit (May 22-26, 2023) brought together 34 developers at the eScience Institute at the University of Washington to develop shared infrastructure, documentation, tools, and recommendations for libraries in the Scientific Python ecosystem.

Pre-summit planning#

Prior to the summit we held several hour-long planning meetings:

• General (2023-02-27)
• May 15, Package metrics, DevStats
• May 15, SPECs
• May 18, Community & Documentation
• May 19, Build Systems & CI Infrastructure
• May 19, PyTest plugins & Sphinx extensions

Summit execution#

At the summit, we had a brief check-in and then split into several groups based on each developers time and interests. Raw work progress and log have been collected in a document, we highlight just a few of things we accomplished below:

Sparse arrays#

Almost a quarter of the group worked on sparse arrays for the entire week. This work is part of a larger, multi-year effort to improve and expand SciPy’s sparse array API, which will eventually involve removing the sparse matrix API and eventually np.matrix.

More details can be found in the Developer Summit 1: Sparse blog post.

Scientific Python Ecosystem Coordination documents#

We made significant progress on several SPECs, which had been drafted during previous sprints.

SPEC 0—Minimum Supported Versions , an updated and expanded recommendation similar to the NEP 29, was discussed and endorsed by several core projects.

SPEC 1—Lazy Loading of Submodules and Functions was discussed and endorsed by two core projects.

SPEC 2—API Dispatch was discussed (in a follow-up video meeting just after the summit) and is in the process of being marked as withdrawn or something similar.

SPEC 3—Accessibility was discussed and updated. We hope to see it endorsed by several core projects in the near future.

SPEC 4—Using a creating nightly wheels was rewritten, a helper GitHub action upload-nightly-action was created, and PRs to update the various projects to use the new nightly wheels location were made. The updates are now complete and the SPEC was endorsed by two core projects.

We anticipate several more core projects to endorse the existing SPECs over the coming months and we are now holding regular SPEC steering committee meetings to continue developing and expanding the SPECs.

Community building#

We created a comprehensive community guide to empower projects in fostering their communities. This guide includes essential information on the role of community managers, along with practical strategies for community meetings, outreach, onboarding, and project management.

Development Documentation#

We created a development guide, a new project template, and existing project review.

Serendipitous Collaboration#

One of the fun things that happens at summits like these are the chance encounters of people from different projects. For example, a couple of attendees worked on creating a co-collaboration network across the broader scientific python ecosystem. This gave us the opportunity to look at how contributors collaborate across projects. We could see how the bigger projects were all clustered together as there are multiple contributors who share maintenance duties for multiple projects. We could also, for example, see how the Scikit-HEP cluster was a bit further away from the usual scientific Python cluster. An action item for us :) We need more collaboration!!

Pytest plugins/Sphinx extensions#

Several attendees worked on pytest plugins and Sphinx extensions:

• pytest-regex was created to support selecting tests with regular expressions.

• pytest-doctestplus was moved upstream into the Scientific Python organization. The summit provided new momentum to develop new features (e.g. produce updated docstring), and to use it for the NumPy documentation testing.

• sphinx-scientific-python, a new extension as a home for various features from the ecosystem, e.g., we agreed on bringing existing extensions from MNE tools to this extension.

• pydata-sphinx-theme updates

SciPy release management progress#

The first release candidate of SciPy 1.11.0 was published on PyPI on May 31, 2023, five days after the conclusion of the summit. The summit facilitated high-bandwidth decision making on several proposed SciPy code changes by allowing the current SciPy release manager (Tyler Reddy, Los Alamos National Laboratory) to consult with other SciPy core developers in person. Specific code changes were discussed with the following SciPy maintainers: Stefan van der Walt (scipy.ndimage), CJ Carey (scipy.sparse), Matt Haberland (scipy.stats), and Pamphile Roy (scipy.stats). When SciPy releases are performed out of band from the summit, the release manager will often have to delay incorporation of useful code changes to the next release six months later, due to lack of availability of the pertinent domain experts.

Package metrics#

We factored out a general developer statistics package from our prototype developer statistics website.

(May 22-26, 2023, Seattle WA) – The first Scientific Python Developer Summit provided an opportunity for core developers from the scientific Python ecosystem to come together to:

1. improve joint infrastructure
2. better coordinate core projects
3. work on a shared strategic plan

Related notes/sites:

• Worklog.
• Planning Meeting Notes and Info.

One of the focuses of the summit was Sparse Arrays, and specifically their implementation in SciPy. This post attempts to recap what happened with “sparse” at the summit and a glimpse of plans for our continuing work. The Sparse Array working group holds open follow-up meetings, currently scheduled every two weeks, to continue the momentum and move this project forward.

At the Summit, we focused on improving the newly added Sparse Array API in SciPy, that lets users manipulate sparse data with NumPy semantics (before, SciPy used NumPy’s 2D-only Matrix API, but that is slated for deprecation). Our goal at the summit was to give focused energy to the effort, bring new people on board, and connect downstream users with the development effort. We also worked to create a working group for this project that would last beyond the summit itself.

The specific PRs and Issues involved in scipy.sparse are detailed in the Summit 2023 scipy.sparse Report, with more detailed description appearing in the Summit Worklog. Some big picture take-aways are:

• Reorganized how to check for matrix/array/format info. This involved adding a format attribute describing which format of sparse storage is used, changing functions issparse/isspmatrix as well as shifting the class hierarchy to allow easy isinstance checking. The interface going forward includes:
  • issparse(A): True when a sparse array or matrix.
  • isinstance(A, sparray): True when a sparse array.
  • isspmatrix(A): True when a sparse matrix. To check the format of a sparse array or matrix use A.format == "csr" or similar.
• Made decisions about how to approach the “creation functions” for sparse arrays. The big-picture approach is to introduce new functions with an _array suffix which construct sparse arrays. The old names will continue to create sparse matrix until post-deprecation removal. Some specific changes made include:
  • Add the creation function diags_array(A) (and planned for eye_array, random_array and others).
  • Create a sparse.linalg.matrix_power function for positive integer matrix power of a sparse array
• Made progress toward 1D sparse arrays. The data structures for 1d may be quite different from 2d. A prototype coo_array allowed exploration of possible n-d arrays, though that is not a short-term goal.
• Explored feasibility and usefulness of defining __array_ufunc__ and other __array_*__ protocols for sparse arrays
• Made clearer distinction between private and public methods for sparse arrays
• Improved documentation for sparse arrays

Our goal is to have a working set of sparse array construction functions and a 1d sparse array class (focusing on coo_array first) in plenty of time for intensive testing before SciPy v1.12. This will then allow us to focus on creating migration documents and tools as well as helping downstream libraries make the shift to sparse arrays. We hope to enable the removal of deprecated sparse matrix interfaces in favor of the array interface. For this to happen we will need most downstream users to shift to the sparse array API. We intend to help them do that.

Our work continues with a community call every two weeks on Fridays. Near term work is to:

• Continue improving sparse creation functions: diags, eye, random and others.
• Deprecate some matrix-specific functionality
• General performance improvements
• Adapting scikit-learn to support sparse arrays (to be discussed with scikit-learn’s maintainers)