So we have pasta boiling, the sauce is bubbling, oh, and I need the pepper grinder… But for how long has the pasta been cooking? When did I start? What was I saying? What now? Ask these questions no more with VRCooking Chef – our Apple Vision Pro cooking software.
As the digital landscape evolves, so does the way we interact with our devices. With the introduction of Apple’s Vision Pro, the frontier for spatial interfaces has opened up, introducing a new direction in which technology may evolve.
As an enthusiast for cutting-edge technology, I was eager to dive into this new world from the get-go. To learn, explore and perhaps gain a head start in developing commercial native visionOS apps.
Cooking Chef – the cooking companion app designed for the Apple Vision Pro is a demonstration of the platform’s capabilities and my journey through the entire lifecycle of app development—from conception to distribution.
Inspiration and Concept
The genesis of this app wasn’t born from a desire to revolutionize cooking but rather from a curiosity about how a traditionally two-dimensional experience could transition into the spatial realm. I realized a cooking assistant may be the perfect domain for such a scenario.
It offered a contained user experience where I could implement step-by-step instructions and separate windows for timers that users can position over their cooking stations. This concept allowed for a direct and practical application of visionOS’s spatial capabilities, showcasing how everyday tasks could be enhanced through augmented reality.
For the demo version of the app, the content was limited to three recipes with detailed and illustrated step-by-step guidance. The focus was put on ensuring a seamless demonstration of the app’s capabilities. You may wonder whose grandma’s cookbook I decided to borrow the recipes from. And the exciting part is that both the recipes and their associated images were generated using OpenAI’s ChatGPT and DALL-E. This way, I obtained unique, detailed, and high-quality content.
Development
The process of bringing the cooking companion app to life unfolded entirely within the realm of the visionOS simulator. Given the current accessibility and availability of the Apple Vision Pro, the simulator served as a crucial tool for development and testing. While there’s no substitute for experiencing an app firsthand on the actual device, the simulator provided a valuable and practical starting point for this project.
Developing an app for visionOS without a physical headset presented an intriguing challenge. Traditionally, even with basic iOS apps, the transition from the simulator to running the app on an actual iPhone can reveal experiential differences. This discrepancy is magnified when it comes to developing an app for a device such as the Apple Vision Pro. The simulator allows developers to set presets for their surroundings and preview the app, but it cannot replicate the full user experience of wearing and interacting with the device in a three-dimensional space.
Acknowledging this limitation, the development of our cooking assistant demo was approached with a focus on learning and showcasing what could be achieved within these constraints. Despite not owning an Apple Vision Pro, the simulator proved to be sufficiently effective for creating a small-scale demo. It enabled the exploration of visionOS’s capabilities and the implementation of features tailored to an augmented reality environment.
However, the insights gained during this process underscore the importance of testing on actual hardware. For developers serious about creating applications for the Vision Pro, accessing the physical device is imperative to fully understand the user experience, gauge the app’s spatial dynamics, and truly immerse oneself in the augmented reality they’ve created. Experiencing the app in the 3D world, it’s designed to offer invaluable feedback for refining and enhancing its functionality and user interface.
App Flow
The outcome of my research and development in spatial computing is the Cooking Chef app, now available for download on the App Store. Developed as a window app using SwiftUI, the application has been precisely crafted in compliance with Apple’s Human Interface Guidelines. I aimed to ensure an easy-to-navigate and well-suited spatial experience for Apple Vision Pro users.
Upon launching the app, a user is presented with a menu featuring generated dish options. The user can select a choice by directing their gaze toward a list element and tapping two fingers together. This action is preceded by a highlighting hover effect, visually indicating the focused subject.
Once a menu position has been selected, the app presents a recipe overview. The user can explore details such as ingredients, required hardware, and a dish description (including obligatory personal heartwarming stories, such as “When I was five years old, my grandma prepared this dish every time I…”). The overview is displayed as a modal view that is strategically leveraged in the user’s direction, further enhancing the spatial effect.
After selecting the “Start Cooking” option on the recipe overview modal, a new window is presented. This window features an illustrated, step-by-step guide explaining how to prepare the chosen dish. The user has the flexibility to position the window wherever is most convenient, allowing them to check on the next steps without obstructing their view while cooking.
To navigate between recipe steps, the user can utilize controls placed in the top and bottom ornaments—view additions introduced by visionOS. These additions do not block the view but offer optional interaction options. Clicking the top-left button brings out a window with the menu in the foreground. If the user wishes to dismiss the recipe guide, they can use the native controls located at the bottom of the window.
When a recipe step requires the user to wait for a specific amount of time, for instance, cooking pasta for 10 minutes, the user can initiate a timer by clicking the timer option in the guide’s top-right corner. This action opens a new window with a timer that comes prefilled with the designated time. The user can freely place this timer window in any convenient location, such as over the pot containing the boiling pasta (and the timer will stay there regardless of your head movements!)
These simple functionalities have allowed me to delve into the process of designing and developing a spatial app for Apple Vision Pro. I’ve gained insights into the basics of creating a window-based app, navigating through the provided documentation, and utilizing available resources to ensure a seamless user experience. Even with a straightforward app flow design, I encountered a few flaws and identified aspects that require more in-depth documentation of the newly introduced operating system.
Challenges and Solutions
The design process for a spatial interface introduced a new set of challenges. The concept of having an “unlimited space” for user interaction departs significantly from the constraints of mobile phones and traditional aspect ratios. This freedom offers endless possibilities but also lacks established patterns for optimal user experience. In navigating this uncharted territory, I found that Apple’s Swift API, with its support for window group creation, offered a semblance of familiarity, drawing parallels to aspects of iPadOS and macOS.
However, the journey was not without its trials. We encountered limitations within the current API and had to rethink some of our initial concepts due to these practical limitations and our own inexperience with spatial interfaces. Through iteration and validation, we focused on simplifying the app to ensure intuitiveness. Stripping down to essential features allowed us to create a user-friendly experience that leveraged the unique capabilities of visionOS without overwhelming potential users.
Navigation, modal views, and all of the windows floating around you
The development of SwiftUI apps for visionOS includes a fusion of iOS, iPadOS, and macOS approaches. One of the significant challenges I encountered during the development process was coming up with optimal solutions for app navigation, presenting modal views, and managing windows. To address this, I had to integrate and leverage approaches introduced for various operating systems.
For example, in the Cooking Chef app, different types of windows are combined, including a singular main menu, step-by-step recipe guides, and multiple timers. Ensuring a seamless experience requires careful consideration of various scenarios, such as when a user dismisses all windows except for a specific timer. In such cases, it was crucial to design a proper navigation flow, allowing the user to call out the main menu window and avoid getting stuck on the timer window.
Some of the modifiers and methods facilitating window management were originally introduced for macOS development, leveraging the familiar concept of apps with multiple windows from operating systems such as macOS and (obviously) Windows. However, I’ve found that the visionOS documentation was not detailed enough, and I had to resort to macOS resources that weren’t entirely tailored for the visionOS platform. More comprehensive visionOS documentation, including additional use cases and expanded UX guidelines for such scenarios, would be immensely helpful in simplifying the development process.
Through the app development process, I’ve learned that it’s reasonable not to overly rely on programmatic window dismissal, like implementing a button with an “x” mark to close a window. This approach has some limitations, such as the dismiss operation not being executed if the window is the last one presented. Instead, leveraging the built-in options provided by the controls below the window proved to be more convenient. This approach ensures a natural app flow and offers the best user experience.
Implementing visually appealing modal views, such as the recipe overview, also presented its own set of challenges. In designing a spatial app, it’s crucial to keep in mind that modal views cannot exceed the parent’s size. Additionally, all attached ornaments are positioned by default based on the parent’s size and depth parameters. These considerations are important to ensure the seamless integration and proper rendering of modal views within the spatial environment.
Distribution
Releasing a visionOS app follows a process similar to iOS app submissions, with a few additional considerations. The App Store now mandates visionOS-specific 4k screenshots to showcase the app’s features. Additionally, new standards for app icons have been introduced to ensure their reflection in the device’s spatial home menu. The distribution process for our app went smoothly, as it was accepted in the first review. Consequently, releasing an app developed exclusively with the simulator was a relatively straightforward process. Now all that remains is to await user feedback and reviews.
Lessons Learned
Familiarity with SwiftUI and iOS Development
Leveraging existing knowledge of SwiftUI and iOS development was instrumental in navigating the initial stages of app creation. However, the transition to spatial computing required a paradigm shift in design and interaction patterns.
Importance of Documentation
The documentation for visionOS, while in its nascent stages, served as a critical resource. Its concise nature underscored the importance of thoroughly understanding the available guidelines and API references to utilize the platform’s capabilities effectively.
Evolving User Experience Patterns
Developing apps for visionOS revealed the necessity of exploring new user experience patterns. The spatial interface introduces a realm of possibilities that deviates from traditional screen-based interactions, requiring innovative approaches to design and functionality.
API Limitations
The current limitations of the visionOS API present both challenges and opportunities for creativity. As the platform evolves, it is anticipated that Apple will introduce expanded APIs that accommodate a broader range of interactions and complexities.
Future Enhancements
The Cooking Chef app, as introduced here, represents just the beginning of what’s possible with visionOS and spatial computing. While the current demo is limited in scope, the potential for expansion and innovation remains vast. Future development could significantly enhance the app’s capabilities and user experience should further exploration be undertaken. Such as:
Dynamic Recipe Integration
Implementing an API to fetch new recipes or integrating a local AI model for generating recipes based on user input would significantly enhance the app’s utility and personalization.
Exploring Volumes for 3D Models
VisionOS’s support for volumes opens up exciting possibilities for incorporating 3D models into the cooking experience. Utilizing Apple’s VisionOS RealityKit could enable the presentation of food items or cooking utensils in 3D within the user’s physical space, offering an immersive guide through recipe preparation and kitchen setup.
Social Sharing Features
Integrating social sharing capabilities, including the ability to share screenshots and cooking achievements, could foster a community of users who share a passion for cooking and technology.
Read also: A quick guide to Apple app clips, a new feature for iOS 14
