Controlled Serendipity
Building an AI-assisted workflow for early-stage architectural design.
4 min read
Halfway through a month-long project pursuit, my two-person team became a solo mission. I doubled down on an AI workflow we’d been experimenting with, allowing me to explore at least five more conceptual directions than a conventional process would have supported while still delivering the study on time. The preferred option for the final client presentation came out of a generative surprise, made viable by the geometry-controlled method.
Problem
SHoP was asked to generate facade concepts to refresh a large existing structure in New York City, with the understanding that if a concept was well received, SHoP would be awarded paid work from the client. As is often the case in architecture, this study was speculative and unpaid, so any resources allocated were overhead absorbed by the office. The pursuit was scheduled for one month, culminating in a presentation to the client. Two full-time staff members, including me, were assigned to the pursuit. Midway through the month, my teammate was pulled to work on another project. I needed to deliver the remainder of the study solo.
Research
Before my teammate left the project, we had experimented a bit with AI to generate concept images. At the time, AI-generated imagery in architectural practice was still uncommon, but SHoP leadership sanctioned our experiments so long as the resulting work met the office’s quality standards. Without my teammate, I decided to double down on the AI workflow.
The major issue with AI-generated imagery in architectural design, particularly at the time, was that the diffusion models would produce a building that looked compelling and plausible to the untrained eye but lacked the precision and control demanded by professional design work. Because this was a renovation to an existing structure whose interior programming needed to remain nearly untouched, geometric control was essential. ControlNets allowed the most important geometry to remain stable while I used the diffusion models to test concepts quickly.
Insights
Creating a realistic, client-facing rendering without generative imagery requires a combination of well-mapped textures and detailed modeling. The modeling required for a high quality rendering can take days, but with the ControlNet and generative imagery combination, I only needed to model the critical geometry. Days of modeling became mere hours. The important part was knowing which aspects of the geometry needed to stay fixed and weighting the inputs properly so the output wouldn’t drift unacceptably. Both of the final schemes presented to the client were only viable because the geometry was controlled.
Solution
The workflow boiled down to four steps. First, I created a lightweight 3D model of the existing building and proposed additional elements. Second, I fed the desired view of this lightweight model into the diffusion model as a strong ControlNet along with conceptual imagery and a text prompt. Third, I iterated by refining the prompts and weights based on the resulting outputs. Sometimes, I fed in promising earlier generative outputs. In one particular run, the model combined the ControlNet, a concept image, and a previous output into an elegant interpretation of the prompt. The result was unexpected and better than what I’d been aiming for. Because of the geometric control, the scheme was also viable, and it became the basis of one of the two concepts that were selected for presentation to the client. Finally, I manually refined both selected concepts in Photoshop.
The process wasn’t always smooth. Over time, the outputs’ geometry tended to stray from the source geometry. I saw this happen with both the building itself and the surrounding context buildings. I noticed this happened more often when I fed in previously generated outputs and weighted them more heavily. I later learned this was a known failure mode in AI-image generation. In one case, a neighboring skyscraper in the background was so warped in the generated image that one of the supervising partners asked, “What the f*ck happened to the tower?” I said, “Yeah, that’s one of the drawbacks of this AI workflow, but I can fix it in Photoshop if we like how the foreground is working.” Ultimately, the workflow allowed me to test at least five additional conceptual directions, including one that became the main option presented to the client.
Outcome
Despite the mid-month staff reduction, SHoP was able to present two options to the client on time, and the work was well received. SHoP was later invited to collaborate on a related, paid competition where the AI-generated scheme was further refined. The geometry-controlled AI workflow spread across the office, and while advancing technology and others’ refinements have changed the exact technique, the general process is still used on many early-stage projects.
Reflection
While I was one of the first at SHoP to successfully employ this process on a live project, others in the office were really the ones who documented and helped spread the method. In retrospect, documenting and distributing the workflow myself may have helped the method become more widely adopted within the office earlier.
Giving up control over exactly how the rendered output would look was uncomfortable. Fixing one bad element in an otherwise promising image meant going back to Photoshop, which was frustrating at times. All that said, being able to harness the aspects that mattered while remaining open to chance elsewhere led to a better design than I would’ve been able to produce with a conventional workflow.