When high-performance engineers want to make imaging tools more efficient and more portable across devices and operating systems, they use Halide. Halide is an open source embedded programming language that was first developed through a collaboration between Adobe Research and researchers at MIT. The language is now used by engineers and researchers in industry and academia around the world.
In 2012, the team first published the details of their work in a groundbreaking paper at SIGGRAPH. Just over a decade later, the paper was selected by SIGGRAPH as one of the seminal works in graphics research. Today, Halide code helps power the tools billions of people use, from digital cameras and video platforms to many of Adobe’s products. And recent innovations are making Halide more easily accessible to more users than ever before.
To learn more, we talked to five people who are on the front lines of building, refining, and programming in the language. Jonathan Ragan-Kelley, one of the original developers of Halide, is an Associate Professor of Computer Science at MIT, Adobe Research consultant, and former Adobe Research intern. Andrew Adams, Adobe Research Principal Scientist, leads engineering for the open-source Halide community and is one of the people helping apply Halide to Adobe tools. Alex Reinking, Adobe Research Scientist and Engineer, helps maintain Halide, dealing with bug fixes and ecosystem integration. Shoaib Kamil, Principal Research Scientist, uses the language regularly and was part of the original group deploying Halide for Adobe products. Jacob Correia, Senior Engineering Manager, Photoshop Engineering, has collaborated with the team on migrating Photoshop code to Halide, along with other Photoshop team members.
What can people do with Halide? And how is it different from traditional programming languages?
Andrew Adams: Halide is all about improving a high-performance engineer’s development cycle. The process usually goes like this: an engineer writes an algorithm—maybe an image processing algorithm—and debugs it to make sure it’s correct. Then they enter a loop where they make changes to the code and check to see where the computer is spending its time. Then they make more changes and see if they can make the algorithm more efficient. It’s very experimental.
In a traditional language, doing all of this is quite slow. There are lots of optimization techniques you can apply and all of them take time—and you can make mistakes along the way. The aim of Halide is to make that development cycle as fast as possible so that you can be scientific and try lots of different hypotheses quickly and safely.
Halide does this by turning the techniques of high-performance engineering into commands in the language. Instead of going through the slow manual process of tiling a loop, for example, you just tell the language, ‘This loop should be tiled by this much.’ The language will do it for you with two promises: you’ll get the same output as your original code and the changes won’t go out of bounds for memory—which means it won’t crash.
Jonathan Ragan-Kelley: Outside of Halide, the traditional world of performance engineering is split. You can write programs at a high level of abstraction and then use an automatic tool to, hopefully, make something fast. That works decently well lots of the time. But for the most performance-critical things, there’s almost always a human performance engineering expert who writes low-level code by hand and takes control—because they can usually do better than an automatic optimizer. They’re rewriting code by hand that worked, hoping to optimize it without introducing bugs. With Halide, as Andrew explained, you don’t have to rewrite, so you save time and you can’t accidentally introduce bugs as you go.
How are people using Halide, and what’s the impact?
Alex Reinking: People from all sorts of places in the open-source world use Halide to make their imaging applications go faster. For example, if you put a camera on a robot and then process those images, Halide’s a good choice. Adobe and many of the other tech organizations use Halide for things like processing videos, making cameras work, and working with graphics.
Adams: As one example, a couple of years ago I reviewed Photoshop algorithms and rewrote them with Halide. Now, many of the low-level things that Photoshop does are written in Halide, and that makes them about two-and-a-half-times faster. I couldn’t have been nearly that productive with any other tool.
Ragan-Kelley: On top of efficiency, another big advantage of Halide is that the code you write isn’t specific to a particular processor, so it has allowed tools like Photoshop to adapt quickly to changes in technology. In terms of Halide’s reach, the language is designed for the very small group of people who want explicit control over the performance of imaging tools. But the code they write can be extremely high impact. In the case of Adobe, the small number of people writing in Halide ship critical code that runs every second you’re running Adobe programs. And Halide has a high impact at several other companies, too. So there are dozens of people who use Halide for their jobs—but they ship code to billions of people.
Jacob Correia: On the Photoshop team, we’ve seen this product impact first-hand. Photoshop’s compositing engine relies extensively on Halide for pixel blending operations. Migrating our legacy code to Halide enhanced our ability to leverage modern hardware more effectively across desktop, mobile, and web. And the Photoshop team continues to benefit from close collaboration with our research team partners as they improve Halide. This results in performance improvements that benefit our users and support Photoshop’s architecture modernization initiatives with lower development overhead.
Why did the team want to make Halide open source? And how has that shaped the way people use it?
Adams: This technology first came out of our work at MIT, so our default approach as academics was to make it open source. We wanted the world to be able to use it so that we could benefit from what other people are doing and other people could benefit from what we’re doing.
Shoaib Kamil: Making compilers open source helps build a community of users that can influence, optimize, and fix any issues that arise in their specific use cases. With Halide, we’ve seen major contributions from Adobe and other large companies, as well as a number of developers who use it to build software. In addition, open sourcing has helped academic researchers use Halide as a testbed for their ideas, further pushing innovation into the compiler.
Adams: And now a new generation of students is coming up with new languages and new systems that take the lessons of Halide to solve different problems in better ways. It’s really cool to see the ideas keep growing.
Where do you hope Halide goes next?
Kamil: With some of the incredibly exciting recent developments, including better vectorization and instruction selection and enhancements to GPU execution, I hope Halide will continue to be a powerful tool for academic research as well as product development. At Adobe specifically, I’m excited by new use cases, including new features built on top of Halide-powered pipelines for cross-platform high performance.
Adams: We also want Halide to be able to express and optimize more kinds of code, and we want it to be able to generate code for more kinds of hardware. I’d also love to see the tool used to help teach high-performance engineering.
Reinking: We’re making Halide accessible to more users, too. Last year, we took what used to be a multi-hour process to get Halide up and running and reduced it to just “pip install halide.” Now, as long as you have a working Python installation, you’ll be able to use it. So we’re excited to see what happens as even more people tap into Halide.
Wondering what else is happening inside Adobe Research? Check out our latest news here.
