How Apple could bring augmented reality to the masses with ARKit and iOS 11

apples arkit to bring augmented reality the masses apple car
Some of the biggest tech companies — Apple, Google, and Microsoft to name a few – seem sure that augmented reality (AR) is going to take smartphones to the next level. So far, it’s all been talk, but that could change before the end of this year. Apple is preparing to unleash ARKit, a brand-new framework for creating this kind of content.

AR has been a priority for Apple for some time. The company has spent a great deal of time and effort hiring staff and acquiring startups to ensure that it has all the talent it needs once the technology is mature enough for the masses. Now, we’re on the verge of Apple’s AR coming-out party: the release of iOS 11.

To understand the implications of ARKit, we spoke to a developer who’s worked on augmented reality tech for the better part of a decade, and has already spent some time putting Apples developer kit through its paces.

The ground floor of AR

Jan-Hein Pullens and his team produce AR content for clients in the home furnishing and real estate industries – and demand for their work may skyrocket as Apple attempts to bring the technology to the masses.

Today, there’s sufficient hardware and infrastructure to allow users to run high-quality AR content.

However, when Pullens and Pieter Aarts founded RoOomy back in 2009, the technological landscape was very different. It would still be three years before the Oculus Rift Kickstarter campaign even brought virtual reality into the public consciousness. Google Glass, one of the first high-profile AR projects to get underway, wouldn’t be successfully prototyped until mid-2011.

Pullens and Aarts were initially excited by the prospect of giving people a way to see how large pieces of furniture might look in their own home using AR. However, the hardware to run that kind of software simply wasn’t available to the public.

“Eight years ago, for example, there wasn’t an iPad,” said Pullens when he spoke to Digital Trends on the phone last month. “People had desktops, and phones.”

Desktop PCs aren’t ideal for AR content because you can’t move them around to see different angles. And back then, smartphones weren’t much better off. They simply didn’t have the horsepower (or the sensors) needed to present AR software.

Whether you’re trying to sell someone a luxury sofa, or a luxury apartment, it’s crucial that your virtual visualization plays to the strengths of the product. “It needs to be very realistic, otherwise it looks gimmicky and like a game,” he explained.

Today, there’s sufficient hardware and infrastructure for some phone owners to run high-quality AR content. Google Tango is the most well-established platform on the scene right now. Unfortunately, it’s only compatible with two smartphones – the Lenovo Phab 2 Pro and the Asus Zenfone AR. Do you know anyone who owns those phones? Neither do we. But with the iPhone entering the fray, suddenly a huge chunk of smartphone users will be AR capable.

Apple opens the gates

ARKit will be supported by iOS devices that use the Apple A9 or A10 processors – the 2017 iPad, the iPhone 6S, and onward. Admittedly, that does leave the millions of users with older hardware unable to access AR content built using the platform, but it absolutely dwarfs the userbase for Google Tango, Microsoft HoloLens, and every other AR platform.

Sophisticated AR functionality requires specialized sensors, like a depth-sensing camera.

There is another complication. Some of the most sophisticated AR functionality requires specialized sensors, like a depth-sensing camera. It’s true that the iPhone 7 Plus has some depth-sensing capabilities, utilizing two lenses working in sync to measure relative distance. However, in the grander scheme of AR tech, it’s a relatively primitive solution.

In February 2017, there were rumblings that the next iPhone would implement an infrared sensor similar to the one used in Microsoft’s Kinect accessory for the Xbox 360, as reported by The Verge. This kind of sensor would provide much more detailed information on an object’s relative position to the device than the current dual-lens set-up. It’s also rumored that Apple will introduce some kind of component that serves this purpose as part of its 2017 iPhone refresh (read the latest iPhone 8 rumors), but there’s nothing official yet.

These new devices will be considered the baseline for AR developers moving forward, particularly because of the advantages associated with depth-sensing cameras. However, the combination of ARKit and current hardware is already bearing fruit. Pullens and his team spent some time with an early version of the development kit, and they like what they’ve seen.

“The first findings that we have with Apple ARKit are promising, they’re actually very promising,” said Pullens. He praised the way the platform copes with occlusion, and its capacity to prevent virtual objects from interfering with one another.

roOmy Bedroom AR furniture
AR bedroom demonstration by roOmy

For Pullens, the most impressive aspect of ARKit is its stability. Virtual objects can often ‘drift’ when they’re not properly aligned with their real-world surroundings, which can be a big problem for the type of visualizations that he and his team at RoOomy produce.

“What I mean by drifting, is for example, a chair in an AR view,” he said. “You would like to see that chair be very stable – you wouldn’t want it to drift or tremble. So, the first findings that we have with Apple are very promising, because it’s quite stable.”

A virtual leather chair isn’t much help if it insists on floating towards the ceiling, or wobbles like there’s a cat under the cushion.

While Pullen had plenty of praise, he also raised some areas where Apple might make improvements. He noted the way ARKit renders light and shadow maps is alright, but added that he expects it to be even better once the platform is ready for release. He also suggested he can see its surface detection capabilities being refined significantly with an improved depth-sensing camera – so, it’d be ideal if the rumors of an infrared camera on the iPhone 8 prove true.

ARKit makes everything easier

AR developers are excited about ARKit because it should open the technology to a much wider audience. Apple seems heavily invested in AR, so we can expect this kind of content to be a priority for the iPhone and iPad. This is an appealing proposition for the people creating AR experiences.

We can expect this kind of content to be a priority for the iPhone and iPad.

Yet a bigger audience isn’t the only benefit of Apple’s development kit. ARKit also aims to remove a lot of the busywork from creating software, allowing developers to focus on how they can use the functionality to provide new and engaging experiences.

“It helps developers like us to provide new features and make good use of AR technology,” said Pullens. “Otherwise, one has to build everything themselves.”

For example, every AR apps needs surface detection that allows a virtual object to sit on a table or the floor. Previously, developers might’ve spent months creating their own surface detection algorithms, or make do with so-so middleware provided by another company. With ARKit, they have access to a highly sophisticated solution that’s already tailored to iOS.

“You get a lot of features already for free in this kit,” added Pullens, referring to functionality like occlusion and light and shadow maps. “This will give a big push to the development community, for new AR solutions to be out there.”

Early ARKit creations are already impressive

Developers who are interested in a sneak preview of ARKit can get it by downloading the beta version of Xcode 9, which includes the iOS 11 SDK. It is already leading to new ideas. AR and VR feed MixedRealityDesign has set up a website dubbed Made with ARKit, which curates a selection of the very best projects around.

The slightly creepy ‘A robot dancing in my living room’ demonstrates the superior stability that Pullens spoke about. An android performs some fluid dance moves in front of a sofa, and despite the camera moving around, the scene looks incredibly natural. The shadow that the robot casts on the floor is particularly impressive.

‘Inter-dimensional Portal’ places a window to another world in the middle of a city street. The graphics used to render this virtual space aren’t very refined, but the overall effect is arresting, particularly once the user walks through the portal. It’s easy to see how this kind of idea might be used in a location-based game along the lines of Pokemon Go.

While these two examples are fun, ‘ARKit will change how we order food’ is much more practical. Rather than looking at flat images on a paper menu, an app produces 3D visualizations of choices right on the table. Another practical implementation is the ‘AR Measure App Demo,’ which offers up a virtual tape measure.

These projects have rough edges, ranging from awkward user interfaces to ugly assets. It’s important to remember that these are early concepts created using tools that have been available for a matter of weeks. Still, ARKit is providing a solid foundation that allows some intriguing ideas to come to fruition.

Apple, in usual fashion, has taken time to get AR right, instead of being first. The first wave of results suggests it’s a step beyond Google Tango and other peers. The next step will be putting advanced AR-friendly hardware into the hands of a broad range of users, and that looks set to happen when the next iPhone is announced.

Product Review

Yuneec’s Mantis Q will make you wish you bought a DJI drone

Yuneec’s high-end drones are arguably the ones to beat in terms of flight control, design, and their photographic capabilities. But the company has struggled to make a low-end drone that’s worth buying, and the Mantis Q is proof of that…

New rumors suggest the iPhone 2019 will continue to use Lightning over USB-C

While it's not been long since the last iPhones launched, rumors for the next iPhone are already surfacing. Apple's 2019 flagship could include a variety of upgrades ranging from a new design to enhanced features.
Smart Home

Have Siri control your home with the best devices that work with Apple HomeKit

Though they've been slow to trickle out, the list of HomeKit smart home products continues to grow and will only continue to do so now that the HomePod is available. Here's a list of the best Apple HomeKit-compatible devices.
Virtual Reality

Think virtual reality is just for games? These awesome apps will change your mind

Virtual reality isn't all about gaming. Swim with turtles, paint in 3D, and immerse yourself in some unique experiences the platform has to offer with our curated list of the best VR apps.

Apple loses battle to use Intel modems in Germany in latest clash with Qualcomm

Apple is following the Federal Trade Commission's lead and has sued Qualcomm for a massive $1 billion in the U.S., $145 million in China, and also in the U.K., claiming the company charged onerous royalties for its patented tech.

Happy Valentine’s Day! Coffee Meets Bagel dating app data may have been breached

Are you planning on using Coffee Meets Bagel to find love on Valentine's Day? If you've been using the app for a while, you'll probably want to change your password -- the company said a data breach may have taken place before May 2018.

Exclusive: Take a look at what a next-generation 5G phone will look like

With 5G phones debuting at MWC in mere days, there is discussion about whether they will be clunky bricks that die after a few hours? A reference design from Qualcomm offerrs a glimpse of the future: This is what 5G phones will look like.

Worried about extra data charges? Here's how to check your usage on an iPhone

It's common to get a little nervous about nearing data limits. Keep your peace of mind by checking how much data your iPhone is using. Our guide on how to check data usage on an iPhone helps you stay in control.

North Focals smartglasses discount cuts the price by a massive $400

Canadian startup North is hoping smartglasses will be the next big wearable. After announcing its new Focals smartglasses in late 2018, the company opened product showrooms in Brooklyn and Toronto and has made its first shipment.

New Apple patent hints clamshell-style foldable phone may be in the works

Apple has filed a patent for a foldable phone that suggests the company could be following in the footsteps of the likes of Samsung and Huawei. The patent describes a clamshell-style foldable phone with two separate sections.

Xiaomi Mi 9 will be one of the first phones with monster Snapdragon 855 chip

Xiaomi's next major smartphone release will be the Mi 9, and the company hasn't held back in giving us a good look at the phone, revealing the design, the camera, and a stunning color.

Galaxy Watch Active isn't official yet, but you can see it in Samsung's own app

Samsung may be about to resurrect its Sport line of smartwatches under a new name: The Galaxy Watch Sport Active. Leaks and rumors are building our picture of the device at the moment.

Stop buying old tablets, says Samsung, buy the new Galaxy Tab S5e instead

Samsung has launched the Galaxy Tab S5e -- the E is for Essential -- a reasonably priced tablet that includes many of the features we like from the Tab A 10.5, and the Tab S4. Here's what you need to know.

Bag yourself a bargain with the best budget tablets under $200

The battle for your budget tablet affections is really ramping up. Which tablet, costing less than $200, should be commanding your attention? We take a look at some different options for the budget-conscious.