WWDC 22 Digital Design Lounge — Prototyping

Process

Is SwiftUI used by the team for prototyping?

We’ve used SwiftUI a little bit, yes. We still use Keynote a lot though!

What types of products do you prototype, what's the process of making a design, and how many rough drafts do you usually have before finalizing anything?

Anything that Apple has shipped and might ship is the sort of thing we prototype! 🙂 I personally have worked a lot on cameras (iPhone, iPad) and Pencil over the years, as well as Watch and Health related features more recently. The process of making a design is generally making something, showing it to people, and learning from their feedback, and doing it over and over again. We've given a few WWDC talks on that topic over the years! We don't really count how many "drafts" we make, but everything we work on undergoes many, many iterations.

What are some of the tools you use for prototyping?

As far as digital tools I personally use Sketch and Keynote a lot for static/2D interfaces, After Effects for animation, and Swift/UIKit/Core Animation/Metal when I need to make things a bit more real/custom or interface with hardware. But everyone on our team uses different tools and has workflows that work for them.

Do you have a shrine to Bill Buxton? More seriously, do you start with paper prototypes and, if so, do you use any tooling to go from them to something digital?

I love sketching on my iPad. Note taking and drawing in meetings is important to me.

I've also found that Looom makes animation so fast I am able to make simple hand drawn animations describing interactions / motion in or immediately after meetings too.

(Looom won an apple design award a few years back. Highly recommend! Makes putting thoughts to motion so much faster! No more pushing pixels key frame by key frame out through molasses.)


We love Looom!!

How does your design process look like? Do you take part in problem define, before you start work on some ideas?

We try not to be too rigid. Often we are starting with a specific problem to solve, yes. But sometimes we make things just because they seem interesting, and then figure out why and what they can help solve. It’s about giving ourselves space to figure out what feels great.

We can fit both types of process into the cycle we describe in our WWDC sessions: Make things, Show them to people, Learn from their feedback.

Are there differences between the final product hardware and software, or are the prototypes meant to test a specific feature and include some hardware and software features, but not all of it? aka. are you trying to test something with the mindset of a customer, or as a engineer?

The mindset of a customer, or a person using the product or feature. We collaborate closely with engineering teams that figure out feasibility. Our prototypes help them understand which use cases need to be supported.

When testing with people, how often do you change the prototype in between sessions, or not at all? i.e picking up issues from the first session and adjusting it before the next session. Is this recommended or should we stick to one prototype throughout?

We try to keep more than one direction open at a time. It might mean having multiple very different prototypes, or one prototype than has sliders / prefs and can be adjusted. If someone gives us good feedback, we incorporate it or try it out. If it’s in conflict with the previous direction, we keep both around to let people compare.

It makes sense to let people interact with different prototypes and observe, if I may ask, how flexible do you make prototypes? Are people allowed to tap on anything interactive on the screen, or is it limited to one flow?

It’s usually very limited. We make it look like everything’s there, but only certain parts will be functional. It’s important that the rest look real because it’s often in the whole context of a product that we learn the most.

When you're prototyping new ideas that are likely to meet some resistance (touch bar or dropping headphone jack), what process, if any other than just gut, do you have to decide whether to ship it or kill it?

Generally our team doesn't get to decide whether to ship or kill things. Our job is to explore (by making prototypes) + document all the possible directions that seem interesting or plausible for a specific topic, share with relevant teams, and see what the reactions are. It happens that something we're very excited about is not something that makes it out the door for some reason, in those cases we tend to keep bringing it up in future work that touches on that concept, and work with teams inside the company who are excited about it too to try to build momentum together.

How long does it usually take from the first in hand prototype to the final polished version?

Because we’re often working with new hardware features: Years! The shortest turnaround we’ve had on a project has been about two years. Some other projects we’ve worked on have been on the order of seven years.

I saw Julian mention the process of 'Make things, Show them to people, Learn from their feedback' and I was wondering if you've ever had a product that's had little to no changes after feedback? A product thats a hole in one? And how much does the product usually change after feedback?

Never.

If we’re not getting feedback on something, we’re just not showing it to the right people. We will eventually show it to someone who will have feedback about it (improvements or reasons why it won’t work!)

Things change a lot between our first conception of it and what makes it out the door. That’s the fun part about working with a whole lot of people who are very talented at what they do.

Is there ever a time where you have to stop and refocus the vision or design? Like, if too many new ideas have been added and the original focus has been lost?

Yes, but usually not because of too many new ideas! If there are too many new ideas it means we have a lot we can try out.


Yes absolutely. When that happens we typically try to focus on what it was that people loved the most in that set of work. If you have dozens of things competing for your attention, focusing on the 2-3 that really seem to be winning hearts over is often a good way to move forward without getting bogged down. But sometimes you also have to accept that while you may have a bunch of kind of cool things, there's nothing in there that's a true winner. And that's okay! There's always way for those things you liked but which weren't clear winners to make their way into other work in the future.


Yeah the latter part of Guillaume’s answer was the second part of what I was going to say. If we pause a project it’s because we feel we’ve thoroughly explored a space and we just haven’t hit on what feels great.

How has your workflow & design reviews changed in a WFH world? What have you found works well for collaborating remotely?

I've done much more green screening with remote work. (green screening with props and suits with OBS running the effects and compositing in realtime to a virtual camera I use in webex).

I'm not in a green suit day to day of course 😅, but I was very happy the extensive setup came in use for a project in 2020-2021. I had thought to myself "never thought I'd be grateful for webex"


We have a lot of collaborative Keynote decks too. Even for in-hand prototypes we build we’re taking a lot more video of them that we then place in those Keynote decks.


I like how it gives more opportunity to make real life situation prototypes, because your setup is at home, and not in a studio where things are impersonal.

It forces you to stay even more connected with your teammates, to make sure you share regularly about the work.

I often find that because there are so many considerations to make for a design to be truly great, that can hamper my creativity and almost block me from making progress. Do you deal with similar things, and if so, how do you manage to stay creative and keep going?

Absolutely! I think any creative in any field suffers from the "blank page" problem... for me I find that spending time doing things not tech/design related (playing music, spending time outdoors, reading books about random obscure topics,...) will often spark unexpected connections & inspiration that somehow find their way into my work. Something a little more specific I like to do too is to dive in the history of a topic I'm working on - for example the history of human mark making & handwriting through various cultures was very inspirational when I worked on Pencil.


"too many things to consider" is totally relatable, and absolutely something present in the day to day. I have personally really enjoyed organizing and grouping ideas along a "design space", by trying to figure out what common axis the ideas might have. It helps me feel less overwhelmed, and also allows me to identify any areas I haven't considered.

Prototyping

How accurately does the original design idea translate to the prototype? Are there many changes? I saw Jony Ive say that the best ideas come from conversation, does that continue through the prototype stage?

The purpose of prototypes is to help advance the conversation. Instead of going in circles verbally about what might be, we start to talk about what’s in front of us. Or what could be in front of us that we haven’t made yet!

Which is to say, we often find the things we think are interesting change dramatically once we have a few prototypes in front of us. And we have conversations about that!

Do you differentiate between designing user interfaces and prototyping? Are they related or separate focuses?

Yes they are definitely 2 different things, although for us they overlap a lot! I think prototyping in general refers to building artefacts quickly to answer questions that will then inform how to build something "for real". So Apple also has silicon prototypers working on the M series chips, for instance! For us, we prototype interactions and user interfaces, which means we try to explore what new modes of input or new capabilities (think for example something like FaceID/Depth Camera, or Apple Pencil) would bring to our products, which then informs whether the company should invest the resources in actually building that hardware/product.


It depends on what the prototype is trying to answer and at what stage of the project we are. But generally a polished UI would help better communicate with other teams, make it look real.


Yes to add to Mylène’s answer, we often have to design user interfaces as a part of prototyping something!

Though sometimes we can get away with re-using large parts of the existing designs.

I’m not sure we’re necessarily doing more — the final user interface designs require a lot of work to be designed for all the different devices, screen sizes, edge cases, accessibility cases, error cases… we’re fortunate to get to pass on that until it becomes a specific issue for our concept. We build on what’s already there for those parts.

Have you ever had to build a new tool in order to design a new experience? Do you ever use custom or internal prototyping tools that apple makes?

Yes! And yes! The “tools” are a lot rougher than you might have in mind, but we’re often cobbling together our own sets of code to be able to try things. We don’t build it out into a generic reusable tool as much as we build the frameworks / scaffolding necessary to try out the very-specific thing we want to get into our hands.

We build what’s necessary to try the specific things we want to try. We try really hard to stay away from prematurely abstracting. It’s too easy to accidentally become a framework engineer or tools engineer when we really were just trying to try something out.

Do you have any tips on prototyping? (best practices, stages of prototyping, and anything else you would consider important)

Always remember what are you building a prototype for, what are you trying to answer. We can get caught up sometimes in trying to have a very polish prototype but it should always be about quickly and efficiently testing a panel of different ideas. Sometimes it helps to just use low tech tools, quickly sketch things on paper, get away from screen.

Our team like to repeat the process of:
make things -> show to people -> get some feedback and repeat until it the experience feels good!

How do you prototype Siri experiences?

Keynote animations with audio, in short! It’s like anything else in that we’re just faking the audio response portion.

Do you use human VO in place of Siri in prototypes? Or do you generate mock Siri audio?

For our purposes it is better if it sounds like Siri. Mac’s “Speak selection” is very useful for this.

I’m curious, when first designing a prototype of a new product, what would you say the ratio is of looks to functionality? Are looks secondary or are they the focus from the get go?

I think the main thing is to think about what questions you are trying to answer by building one (or several) prototypes. Are they questions of technical feasibility, product usefulness,…? etc. Looks for the sake of looks are rarely worth spending lots of early prototyping time on, but sometimes different aesthetic directions or visual metaphors are definitely things you want to prototype!


I’d add that the key balance to strike in prototyping is to make the least amount you need to make to still learn something. Sometimes that means things have to look very real so that you don’t distract from the part you’re trying to learn.

What are some philosophies that the team adheres to when designing and prototyping things?

Make things, show them to people, learn from their feedback!


That should be a tattoo at this point!


Fake it until you make it.


Considering an interaction "Before, During, and After" (i.e. in the example of a button before people press a button, how do they know they can do it, and what it will do, during the press how do they know the action is registered and whether they need to continue pressing ( any visuals that encourage continued action would be 'feed-forward'). And afterwards, what is the feedback that ensures we know how the action went?) Julian;s 2018 WWDC talk covers this in more detail.


Always bring positive feedback when sharing the work. It should never be about personal judgement but how to make your app experience better.
For example:
"I don't like this color" vs "I think blue instead of read would better communicate what the experience is about".


"Always design a thing by considering it in its next larger context – a chair in a room, a room in a house, a house in an environment, an environment in a city plan." is a quote that I think also nicely captures the spirit of always being able to step back and not design something in isolation.

Delight

How do you go about adding \(magic || whimsey || delight) to a prototype? I feel a lot of my designs lack that extra ✨ to make a great experience.

Give yourself time to not worry so much about solving the problem. “What other ideas does this give us?” can mean completely unrelated things. But if they seem interesting they’re worth trying. Those weird-but-interesting ideas can inspire us to connect the weird / whimsical inspiration to something that actually does solve the problem.


well said 🥲

Interactions

How does your team go about quickly prototyping advanced interactions? I have lots of ideas for interactions I want to test but I feel my momentum is burnt out by the time I get everything set up in, say, a SwiftUI project.

Finding a way to fake it! We talk about this in most of our old WWDC sessions, but Prototyping for AR (2018) has some good examples of finding clever ways to prototype that don’t involve code at all.

There are ways to fake things with paper printouts, or with clever video capture. But often doing some simple Keynote animations can teach a lot too.

Collaboration

Do your prototypes sometimes lead to product ideas that Apple pursues, or is the context of your prototyping often within the confines of product ideas that already exist?

It’s often the opposite direction — we’re typically prototyping new products or features that in turn teach us things we could make better about existing products!

And when prototyping new products, is that something your team decides to explore, or do you get “asked” to explore a certain thing?

Both. We pursue things that we think might be interesting, but we’re also keeping tabs on the possibilities that interest partner teams that are asking for our help.

What is the collaboration dynamic between the prototyping team and Design / Eng partners?

We work closely with them! With engineering teams to understand the technical capabilities and limitations that we are working with, and with design teams to understand the roadmap they have for their products, and what problems they're trying to solve right now. The time horizon we work on (3-5 years) is a bit longer than the one most teams operate on, so communication is key for us to be aligned.

How much individual work vs collaboration do you do? Do you work together within a document, or work independently and then present that work for feedback?

Different people on the team have different preferences. But we all attend a weekly design review with the whole Prototyping team in which all our work is shared.

It helps us build a joint understanding of what a project is, even if it’s mostly one individual “doing the work”, we’re all helping with feedback, new ideas, and critique. The team’s brain makes the work better.

Some folks on the team prefer being in collaborative documents 100% of the time. Others will just bring their individual work to design review.

You mentioned not being too rigid on your team and often times working on things in the span of several years. Do you have any rituals as a team to stay connected and get into a "let’s make some awesome things!" mode?

Yes, as Julian said we have weekly team meetings and design reviews where we share very openly what we're working on. Those have happened every week as long as the team's been around, over 15 years now. I think being a small tight knit team is essential to our success.


And in those design reviews we’re very open (and excited) to see random things people made that aren’t directly related to current/ongoing projects.

How do you scale a weekly meeting as the team / scope grows?

We'll have smaller, more focused meetings for individual projects. The goal for the team meetings is for the whole team to see what's going on that week, but they're not exhaustive. Our team has never been bigger than ~12 people, which I think is a feature.

How do you come up with an idea of a product? For example, do you do brainstorming sessions? And how do you transform an idea into a prototype?

I think it's important to know what the biggest questions around an idea you have, because that's what you'll want to prototype first. Is it about testing the usefulness of a feature? About exploring how to best display the results from an algorithm or machine learning model? Or is it about seeing if a new gesture interaction you came up with is easy for people to understand and perform? The goal of prototypes is to answer questions before investing a lot of time into making things real (hence why it's important to keep your prototyping process light & nimble). As far as coming up with ideas... ask people questions, and listen to what they have to say!

Part of my day to day work is taking prototypes and being able to communicate findings to groups and teams that may not have an interest in them (yet..). If you're able to share high level insight - I'm curious how the findings from your prototypes are communicated (format, findings, risks) to external teams or even execs. To some extent the prototype itself carries this weight, but in cases where not and in person communication is infrequent - I'm curious how your team tackles this :)

It’s an interesting challenge, because different people respond to different communication styles. We do have to adapt how we communicate work depending on what people are looking for.

The nice thing about a prototype that appears high-fidelity is that it lessens the amount of assumptions people are making. If it looks like a real app, but we fake one part of the interaction, we are helping to direct feedback toward the parts that seem new.

Tools

The devs behind one of the ADA winners this year, Not Boring Habits, wrote a really interesting blog post (https://www.andy.works/words/the-future-of-design-tools) about the future of prototyping/design tools, arguing for a shift from flat tools like Figma to rich, 3D-first tools like Unity and SceneKit. Do you see those tools becoming first-class citizens in a prototypers toolkit in the future?

Making in the medium the design is meant to be in is important! For a prototype seeking to answer an interaction design question, interactive prototypes are essential. For the design of a graphic poster, print out poster iterations.

If the end artifact is meant to be spatial, then making in 3D is essential too.

I'd say there's an underlying principle that the medium that one makes/prototypes in is also a very intentional choice, and should be chosen to best align with the desired end artifact.

What tool would you suggest for rapid prototyping?

Whatever you're most comfortable with will be what lets you try things rapidly! For some people that's code, others animations in a tool like After Effects... to get started it's really about what you're able to quickly be productive with.

MagicMove in Keynote lets you prototype a lot of simple animations - do you have preferred animation prototyping tools?

One of my gripes about most "modern" tools is they stop short of helping with animation handover or code generation, if they have any animation at all.

AfterEffects has been my go to (light scripting, integration with C4D, able to combine with footage, I end up doing a lot of green screening).

Recently got excited about a library that makes Unity's interface more like a traditional-timeline-animation-style ui.

I understand and commiserate with your gripes about modern animation tools though. I have to pull up Grapher and just fiddle with my equations (when transitioning to code).

(so happy you mentioned magic move in keynote though! Early decks where we try to cover as many ideas as possible, going for breadth, there's plenty of keynote animations for our light sketches)

I'm a UX/UI designer who mostly use Sketch and Figma to prototype apps mockup, but recently I have discovered SwiftUI as an alternative to make high fidelity prototypes, do you think it's worth learning to program just enough to prototype better? Which tools your team use?

Haha, the question of "should designers learn to code" is an eternal one... long answer short, I think code should be seen as one more tool in your toolbox. If you feel like it might let you make things you couldn't make otherwise, and you're drawn to it, then go for it! SwiftUI is the easiest to learn it's ever been 🙂 And Meng's website is great!

I would love to get better at design User Interfaces. What are the benefits of Sketch/Figma/etc. and would you recommend their usage?

Sketch and Figma are two popular tools for UI design and prototyping, but ultimately they're just tools - they won't give you magical results on their own. If you'd like to get better at designing UIs and you're starting from scratch I'd encourage you to focus more on things like layout, type, color, interaction patterns (the Apple HIG is a great place to get started on those topics as they specifically pertain to Apple platforms! https://developer.apple.com/design/human-interface-guidelines/) than focusing on a specific tool.

Recommendations

Are there any good WWDC sessions on prototyping past or present you’d recommend?

Absolutely! You can search for the "prototyping" keyword in WWDC sessions, but here are a couple we did:
Life of a Button: https://developer.apple.com/videos/play/wwdc2018/804/
Prototyping for AR: https://developer.apple.com/videos/play/wwdc2018/808

Oh and this one too! Not specifically about prototyping but discoverability, something we care a lot about.
Discoverable Design https://developer.apple.com/videos/play/wwdc2021/10126/


And some say the Fake It Til You Make It talk may still be on youtube if you search 😉 And gasp! Here it is again: https://developer.apple.com/wwdc14/223.

What are some non-digital sources of inspiration for each of ya? Books, music, art, something that you aspire to perhaps.

Ah. So many things, how does one dimension reduce to answer this? haha, everything from me spending too much time on a saturday morning brunch watching caustics, of light going through different glassware, to thursday night Exploratorium trips, adoring Tokuujin Yoshioka's installation work, Olafur Eliasson.

Looking at the special effects work for movies ("making of The Mandalorian" a year or two ago with their realtime rendering was 🔥) (does this count as non-digital, it is half physical ahaha)


See some fine art in museum is always inspiring! But yeah so many things. For me, just taking long walk randomly in San Francisco can help my thoughts wander in unexpected places.

To answer the last part, "something you aspire to perhaps" I aspire to make things as beautiful as poetry 🥺. I like Amanda Gorman, Jack Gilbert, some good ol' TS Eliot, but ahah, thus far haven't personally been able to draw direct application for poetry.

Favorite books?

So many, I love books! We reference Tufte a lot in our studio. Some recent design books I read and enjoyed were Extra Bold https://papress.com/pages/extra-bold, and Politics of Design https://www.counter-print.co.uk/products/the-politics-of-design.


I always like The Humane Interface and The Design of Everyday Things. Been re-reading Interface Culture recently.

Career

Working on the prototyping team seems like such an incredible thing to be a part of. Can you tell us about your journeys to becoming prototypers at Apple? And what do you look for in potential candidates for your team?

A lot of us some mix of backgrounds that don’t fit neatly into the typical design / engineering teams.

I taught myself programming outside of school and dabbled in a number of open source projects and the like. Realized I enjoyed psychology / cognitive science to learn more about how people think and approach technology. Studied Cognitive Science, HCI, and Communication Design in school. Had some design jobs but eventually accidentally found this team where I got to use my programming skills too!


A teammate was at an HCI conference (TEI '17) in Japan that I happened to have work in. He passed on my name to the recruiter. I studied Design and HCI in school, and then I interviewed!


I studied computer science, spent some time in HCI academia, and worked for a few startups before joining Apple. For me making delightful interactive things has always been a single pursuit, and the boundaries between things like “design” and “engineering” never made too much sense to me. My teammates will have different answers but when I look at candidate portfolios, I like to see inspiration from fields beyond design & tech, a general sense of curiosity and playfulness… and just weird, unexpected, unique things! (even if they’re not the most polished).


I was a fresh graduated from ECAL in Lausanne Switzerland, where I studied interaction design. A few months after that, Jeff our current Manager saw my bachelor graduation project on a blog and then reached out. It was such a suprise! I was super excited and curious to know more about the team and I end up joining in 2018. It was great that even though I had not much professional experience, the team was interested in my portfolio. When we are looking at candidate, I think we are interested about ideas and how the candidate communicates them, even if you have short resume like I did back then.

Show Comments