This sensor turns your iPad into a 3D scanner, but get ready to practice

Printing 3D objects has always felt like a daunting and expensive affair, but we are beginning to see 3D printers come down in price substantially. The problem is that it’s still difficult to create something to print on a 3D printer.

Enter the Structure Sensor ($380), a gadget that turns your iPad into a 3D mapping and imaging device. You can use it to measure the depth of a room or create a 3D model of an object or person. The device puts affordable 3D creation into the hands of both consumers and developers. But will this device put more expensive 3D scanners out of business? Not right away, but it has potential.

Occipital Structure Sensor first look video

Kickstarted

We’ve followed the Structure Sensor since its conception as a Kickstarter product, and we even saw an early demo last year.

The Structure Sensor will let you create 3D models of your favorite objects or people.

The Structure Sensor was created by Occipital, a company based in Boulder, Colo. and San Francisco. While Occipital may not be a household name, you may have heard of RedLaser, its barcode scanning app that was acquired by eBay. The company also made 360 Panorama, one of the first mobile apps to capture real-time panoramic images.

According to Occipital, the company decided to concentrate computer vision in mobile devices after experimenting with Microsoft’s Kinect. By connecting it to a PC, the company’s engineers were able to map indoor environments with the Kinect. The barrier in using the Kinect’s technology was that it was tethered to a high-end desktop, so the team sought to make a portable unit that works with lower-power mobile devices. So they created the Structure Sensor, which raised more than a $1 million, becoming one of the most successful Kickstarter campaigns ever.

What is the Structure Sensor, exactly?

This is the world’s first 3D sensor for a mobile device. Despite the finalized hardware that’s now on sale, the Structure Sensor is still very much a work-in-progress. New software is being developed that utilizes the device’s 3D sensing capabilities.

As simple as the hardware looks, the technology is complicated. Here’s how Occipital describes it:

“The Structure Sensor uses structured light to capture depth data. Structured light uses a laser projector to cast a precise pattern of thousands of invisible infrared dots onto objects and spaces. It then uses a frequency-matched infrared camera to record how the pattern changes, thereby understanding the geometry of those objects and spaces. As a result, the Structure Sensor can generate a VGA depth stream at 30 frames per second, with each pixel representing an exact distance to a real-world point.”

Occipital Structure Sensor Interior Map

It’s a bit technical, but for consumers, just imagine how Microsoft’s Kinect works – how it tracks your movement – and you get the concept. How it’s different is that it’s mobile (with the aide of an iPad) and it has its own battery.

Occipital is reluctant to call it a 3D scanner, but it’s a 3D sensor that can be used for 3D scanning purposes. The first major app to utilize the Structure Sensor is ItSeez3D, which combines the data from the sensor and the color information from the iPad’s camera to create realistic 3D models. Occipital also lists interior space mapping (gathering measurements of a room, for example), augmented reality and virtual reality games, and body scanning (for fitness or as a virtual dressing room) as some of the applications that can utilize the sensor technology.

For developers, this affordable and accessible device offers a new way to create apps and services (imagine a video game company using it to render characters and objects quicker and easier for upcoming title), but for consumers, it’s still too novel for mass usage. If you’re willing to shell out a few hundred bucks for early-adopter tech, the Structure Sensor will let you create 3D models of your favorite objects or people, and let you experience the beginnings of consumer 3D scanning that’s on the way.

Setting up the sensor

Depending on the iPad you have, the Structure Sensor comes with a matching bracket that clips onto the side of the tablet; a latch secures it in-place well, but it’s easy for it to come undone if you don’t properly store it in a bag, for example. The proximity of the sensor to the iPad’s camera allows apps to grab color information to be used with the 3D depth data, as we mentioned with the ItSeez3D app. There are no buttons to mess with; once the bracket is on, you simply connect the cable to the iPad’s Lightning port and any associating app will recognize the sensor.

For consumers, the Structure Sensor will only work with the fourth-gen iPad, iPad Air, and iPad Mini with Retina. Regardless of the bracket you get (full-size or mini), you’ll need to do some simple DIY involving screwing the sensor onto the bracket using tiny screws. If you’ve ever had to work with small screws, then you know how maddening it can if you lose one — take care when you assemble.

If you’re a developer, Occipital will sell you an optional USB Hacker Cable that lets you connect the sensor to any supported USB device that runs Android, Windows, Linux, and OS X. Developers, such as app makers, can build support for Android tablets, for example, using available open source drivers and specs that allow them to 3D print their own brackets. While the Structure Sensor natively supports select iPads and Xcode (the coding language used by iOS apps), those with the know-how can use the open software development kit to make the sensor work with non-iOS devices.

A lack of apps

As we mentioned, there are very few apps available that takes advantage of the sensor, at least right now. Because it’s so new, app developers are probably still busy creating software. When you buy the Structure Sensor, you get the Structure app, a fairly basic iPad app that lets you measure distance, check the sensor’s battery, and stream a 3D view to a computer over Wi-Fi.

Occipital also has a few sample apps that show you what’s possible with the sensor:

  • Scanner demonstrates how an object could be scanned
  • Fetch is an augmented reality game and Ball Physics is an augmented reality simulator
  • Viewer is a “raw data visualizer”

The apps are all free, but they are very limited in their usefulness other than showing what the sensor can do.

ItSeez3D is our favorite app, but it’s involved

The most developed app available now is ItSeez3D from ItSeez, a 3D computer vision company. Using the sensor’s depth data and color information from the iPad’s camera, ItSeez3D can create 3D models of objects and people that you can upload, publish, and embed to websites via online 3D viewer Sketchfab, or email the files (PLY and OBJ) for use in 3D rendering programs.

A promising tool for developers and hardcore 3D enthusiasts.

We first played with the ItSeez3D app without the sensor, and were amazed by how realistic and detailed the sample scans looked, so we were stoked to try our hand at 3D scanning of our own. Unfortunately, at first, we weren’t able to create models that looked as good as the samples. Scanning an object requires you move around it; as you do, the sensor measures the object to create a 3D mesh, while, with each pause, the iPad’s camera takes a high-res photograph for color info and to overlay on top of the 3D model. As you move 360 degrees around an object or person, an indicator at the bottom left shows you the progress of the scans; it also shows you areas where data is lacking, which you can move back to that position to rescan. The more rotations and scans you do, the more information is captured, and when it reaches 100 percent, you can stop and begin the rendering process.

The problem is that the process sounds easier that it actually is. Oftentimes the sensor would lose “plane” data, or isn’t able to accurately determine what is part of the object, and what is not. In some of our 3D scans, we noticed portions were missing or that it captured something that’s not part of it. A 3D scan of an iMac came out garbled because there was so much uniformity and lack of contrast. The object-circling process also takes a while, and we never know when it’ll actually take a photo when we pause. Plus, there were times when the app would lose contact with the sensor; you just need to reconnect the cable. We had better success scanning people’s heads: The sensor and app had no issues rendering a scan of a coworker, although even that wasn’t perfect.

To figure out why we weren’t achieving desirable results, we looked at some videos from ItSeez and checked the tips in the Help menu that shows you what you need to do, such as lighting and putting newspaper beneath the object to help the sensor pick it out from the background and surface. We realized it’s all about lighting, and in another attempt, we had better results. We made sure an object was well-lit from all sides, and we set up a small round table that allowed us to better maneuver around it. This time, our scans came out more complete and detailed, although a few still looked “off” in a few places. We scanned some stuffed animals from childhood, and seeing your favorite toys immortalized as a 3D object can be a bit sentimental and awesome. A scan of a coin bank came out the best, despite being dark and single-colored; the lighting, we think, greatly helped. We realized that the Structure Sensor alone won’t give perfect scans every time, and you’ll need a good setup involving proper lighting. The 3D scanning process isn’t a one-shot deal – it will require several attempts.

Once a scan is complete, the information is uploaded to ItSeez’s cloud servers. After about 5 minutes, a 3D model is sent back to you. You could email the scans for further editing in 3D-capable software like Maya, AutoCAD, or Blender, or upload to Sketchfab, like our cat bank above. You can also upload the OBJ file to Shapeways, an online 3D marketplace that will help you turn your 3D scans into an actual 3D model. In the near future, however, ItSeez may build into the app the ability to have your objects 3D printed or other application. While it’s fairly limited for consumers, seeing an object or person transformed into a third-dimensional object (even if it’s a bad one) is cool.

Conclusion

There’s no doubt that 3D scanning (and printing) is becoming more affordable and accessible, and devices like the Structure Sensor are going to help usher that in for consumers and developers. But the Structure Sensor also shows that this type of 3D scanning is still in its infancy. It’s not that the device is bad or the apps don’t work, but right now there’s limited use for consumers and there are still things that can be improved. However, we still had fun using it, and it’s awesome to see everyday objects get turned into 3D models simply by using an iPad accessory.

As Occipital continues to improve the product and more apps are introduced down the line (not to mention other competing products and technologies that are in the works), there could be some neat applications. Until then, the Structure Sensor is a promising tool for developers and hardcore 3D enthusiasts, but it’s still early-adopter territory for most people.

Product Review

The iPad Pro is the best tablet ever. But don't sell your laptop just yet

Apple has unveiled a big redesign for the iPad Pro, slimming down the bezels, adding Face ID, and the ability to attach and charge the Apple Pencil. All of this comes at a high cost however, as the iPad Pro starts at $799.
Mobile

Apple iPad Pro 11 vs. Samsung Galaxy Tab S4: Which top-tier tablet reigns supreme?

If you're in the market for a new tablet and you want something that can double up as a laptop in a pinch, then you owe it to yourself to check out the 11-inch Apple iPad Pro and the Samsung Galaxy Tab S4. We compare the two here.
Mobile

New Galaxy S10 leaks showcase display sizes, confirm headphone jack return

While we still may be months away from an announcement, there's no doubt about it: Samsung is working hard on its successor to the Galaxy S9. Here's everything we know about the upcoming Samsung Galaxy S10.
Emerging Tech

Nvidia’s new A.I. creates entire virtual cities by watching dash cam videos

Nvidia's groundbreaking new machine learning technology can generate a convincing virtual city simply by showing it car dashcam videos. Here's how it works and why that's so exciting.
Emerging Tech

Keep your holiday gift list high-tech and low-budget with these gadgets

Modern technology doesn't always come cheap, but there plenty of premium devices that don't carry a premium price. Whether you're looking for a streaming device or a means of capturing photos from above, our list of the best tech under $50…
Social Media

Hotel chain offers an Instagram ‘sitter’ who will post photos for you

If the pressure to post stunning Instagram photos is ruining your vacations, then how about hiring a local Instagram influencer to do the job for you while you go off and enjoy yourself? Well, such a service now exists.
Social Media

Ride the rails and share your stories with Amtrak’s new social media residency

Amtrak is looking for travel fans with a knack for telling stories on social media. The new Amtrak social media residency program wants amateur travelers to share photos, video, and written content from aboard long-distance trips.
Photography

Full-frame mirrorless cameras just made their Hollywood debut with this thriller

The Possession of Hannah Grace isn't just a thriller -- it's also the first Hollywood feature film to be shot completely with a full-frame mirrorless camera. The film was shot with several Sony a7S II bodies and anamorphic lenses.
Photography

Lens Rentals zooms in on the most popular cameras of the year

As 2018 comes to a close, Lens Rentals is taking a look at most popular cameras of the year, based on rental data. While Sony and Panasonic saw more rentals than the previous year, Canon is still the most-rented brand on the platform.
Photography

Luminar’s new libraries don’t even need you to manually import images

Luminar 3's new libraries feature doesn't require importing -- images are automatically added after clicking on a folder. The long-promised libraries feature gives editors a Lightroom alternative with organization tools as well as syncing…
Emerging Tech

The best drone photos from around the world

Most of today's drones come equipped with high-end cameras, which are quickly revolutionizing the world of aerial photography as we know it. Here are some of the best drone photos from around the world.
Photography

The best mirrorless cameras pack all the power of a DSLR, minus the bulk

Mirrorless cameras offer a lot of photography firepower, inside a compact body. Explore the best mirrorless cameras, from the pro-level to the beginner-friendly shooters, in this guide.
Photography

Photography News: Startup redesigns tripod heads ‘inside out’ for more flexibility

Well, this doesn't look like the ball heads that we've seen before. Instead of designing a tripod ball head with a small cutout, the Colorado Tripod Company created one with most of the ball exposed, allowing for more possible angles.
Photography

MIT science photographer isn’t an artist, but her work could fill galleries

Felice Frankel is an award-winning photographer, but she doesn't consider herself an artist. As a science photographer, she has been helping researchers better communicate their ideas for nearly three decades with eye-catching imagery.