Occipital Structure Sensor: 3D Scanning for the Mainstream

Printing 3D objects has always felt like a daunting and expensive affair, but we are beginning to see 3D printers come down in price substantially. The problem is that it’s still difficult to create something to print on a 3D printer.

Enter the Structure Sensor ($380), a gadget that turns your iPad into a 3D mapping and imaging device. You can use it to measure the depth of a room or create a 3D model of an object or person. The device puts affordable 3D creation into the hands of both consumers and developers. But will this device put more expensive 3D scanners out of business? Not right away, but it has potential.

Occipital Structure Sensor first look video

Kickstarted

We’ve followed the Structure Sensor since its conception as a Kickstarter product, and we even saw an early demo last year.

The Structure Sensor will let you create 3D models of your favorite objects or people.

The Structure Sensor was created by Occipital, a company based in Boulder, Colo. and San Francisco. While Occipital may not be a household name, you may have heard of RedLaser, its barcode scanning app that was acquired by eBay. The company also made 360 Panorama, one of the first mobile apps to capture real-time panoramic images.

According to Occipital, the company decided to concentrate computer vision in mobile devices after experimenting with Microsoft’s Kinect. By connecting it to a PC, the company’s engineers were able to map indoor environments with the Kinect. The barrier in using the Kinect’s technology was that it was tethered to a high-end desktop, so the team sought to make a portable unit that works with lower-power mobile devices. So they created the Structure Sensor, which raised more than a $1 million, becoming one of the most successful Kickstarter campaigns ever.

What is the Structure Sensor, exactly?

This is the world’s first 3D sensor for a mobile device. Despite the finalized hardware that’s now on sale, the Structure Sensor is still very much a work-in-progress. New software is being developed that utilizes the device’s 3D sensing capabilities.

As simple as the hardware looks, the technology is complicated. Here’s how Occipital describes it:

“The Structure Sensor uses structured light to capture depth data. Structured light uses a laser projector to cast a precise pattern of thousands of invisible infrared dots onto objects and spaces. It then uses a frequency-matched infrared camera to record how the pattern changes, thereby understanding the geometry of those objects and spaces. As a result, the Structure Sensor can generate a VGA depth stream at 30 frames per second, with each pixel representing an exact distance to a real-world point.”

Occipital Structure Sensor Interior Map — Image used with permission by copyright holder

It’s a bit technical, but for consumers, just imagine how Microsoft’s Kinect works – how it tracks your movement – and you get the concept. How it’s different is that it’s mobile (with the aide of an iPad) and it has its own battery.

Occipital is reluctant to call it a 3D scanner, but it’s a 3D sensor that can be used for 3D scanning purposes. The first major app to utilize the Structure Sensor is ItSeez3D, which combines the data from the sensor and the color information from the iPad’s camera to create realistic 3D models. Occipital also lists interior space mapping (gathering measurements of a room, for example), augmented reality and virtual reality games, and body scanning (for fitness or as a virtual dressing room) as some of the applications that can utilize the sensor technology.

For developers, this affordable and accessible device offers a new way to create apps and services (imagine a video game company using it to render characters and objects quicker and easier for upcoming title), but for consumers, it’s still too novel for mass usage. If you’re willing to shell out a few hundred bucks for early-adopter tech, the Structure Sensor will let you create 3D models of your favorite objects or people, and let you experience the beginnings of consumer 3D scanning that’s on the way.

Setting up the sensor

Depending on the iPad you have, the Structure Sensor comes with a matching bracket that clips onto the side of the tablet; a latch secures it in-place well, but it’s easy for it to come undone if you don’t properly store it in a bag, for example. The proximity of the sensor to the iPad’s camera allows apps to grab color information to be used with the 3D depth data, as we mentioned with the ItSeez3D app. There are no buttons to mess with; once the bracket is on, you simply connect the cable to the iPad’s Lightning port and any associating app will recognize the sensor.

For consumers, the Structure Sensor will only work with the fourth-gen iPad, iPad Air, and iPad Mini with Retina. Regardless of the bracket you get (full-size or mini), you’ll need to do some simple DIY involving screwing the sensor onto the bracket using tiny screws. If you’ve ever had to work with small screws, then you know how maddening it can if you lose one — take care when you assemble.

Occipital Structure Sensor back top corner

If you’re a developer, Occipital will sell you an optional USB Hacker Cable that lets you connect the sensor to any supported USB device that runs Android, Windows, Linux, and OS X. Developers, such as app makers, can build support for Android tablets, for example, using available open source drivers and specs that allow them to 3D print their own brackets. While the Structure Sensor natively supports select iPads and Xcode (the coding language used by iOS apps), those with the know-how can use the open software development kit to make the sensor work with non-iOS devices.

A lack of apps

As we mentioned, there are very few apps available that takes advantage of the sensor, at least right now. Because it’s so new, app developers are probably still busy creating software. When you buy the Structure Sensor, you get the Structure app, a fairly basic iPad app that lets you measure distance, check the sensor’s battery, and stream a 3D view to a computer over Wi-Fi.

Occipital also has a few sample apps that show you what’s possible with the sensor:

Scanner demonstrates how an object could be scanned
Fetch is an augmented reality game and Ball Physics is an augmented reality simulator
Viewer is a “raw data visualizer”

The apps are all free, but they are very limited in their usefulness other than showing what the sensor can do.

ItSeez3D is our favorite app, but it’s involved

The most developed app available now is ItSeez3D from ItSeez, a 3D computer vision company. Using the sensor’s depth data and color information from the iPad’s camera, ItSeez3D can create 3D models of objects and people that you can upload, publish, and embed to websites via online 3D viewer Sketchfab, or email the files (PLY and OBJ) for use in 3D rendering programs.

A promising tool for developers and hardcore 3D enthusiasts.

We first played with the ItSeez3D app without the sensor, and were amazed by how realistic and detailed the sample scans looked, so we were stoked to try our hand at 3D scanning of our own. Unfortunately, at first, we weren’t able to create models that looked as good as the samples. Scanning an object requires you move around it; as you do, the sensor measures the object to create a 3D mesh, while, with each pause, the iPad’s camera takes a high-res photograph for color info and to overlay on top of the 3D model. As you move 360 degrees around an object or person, an indicator at the bottom left shows you the progress of the scans; it also shows you areas where data is lacking, which you can move back to that position to rescan. The more rotations and scans you do, the more information is captured, and when it reaches 100 percent, you can stop and begin the rendering process.

The problem is that the process sounds easier that it actually is. Oftentimes the sensor would lose “plane” data, or isn’t able to accurately determine what is part of the object, and what is not. In some of our 3D scans, we noticed portions were missing or that it captured something that’s not part of it. A 3D scan of an iMac came out garbled because there was so much uniformity and lack of contrast. The object-circling process also takes a while, and we never know when it’ll actually take a photo when we pause. Plus, there were times when the app would lose contact with the sensor; you just need to reconnect the cable. We had better success scanning people’s heads: The sensor and app had no issues rendering a scan of a coworker, although even that wasn’t perfect.

Occipital Structure Sensor scanning lego

Occipital Structure Sensor scanning model

To figure out why we weren’t achieving desirable results, we looked at some videos from ItSeez and checked the tips in the Help menu that shows you what you need to do, such as lighting and putting newspaper beneath the object to help the sensor pick it out from the background and surface. We realized it’s all about lighting, and in another attempt, we had better results. We made sure an object was well-lit from all sides, and we set up a small round table that allowed us to better maneuver around it. This time, our scans came out more complete and detailed, although a few still looked “off” in a few places. We scanned some stuffed animals from childhood, and seeing your favorite toys immortalized as a 3D object can be a bit sentimental and awesome. A scan of a coin bank came out the best, despite being dark and single-colored; the lighting, we think, greatly helped. We realized that the Structure Sensor alone won’t give perfect scans every time, and you’ll need a good setup involving proper lighting. The 3D scanning process isn’t a one-shot deal – it will require several attempts.

Once a scan is complete, the information is uploaded to ItSeez’s cloud servers. After about 5 minutes, a 3D model is sent back to you. You could email the scans for further editing in 3D-capable software like Maya, AutoCAD, or Blender, or upload to Sketchfab, like our cat bank above. You can also upload the OBJ file to Shapeways, an online 3D marketplace that will help you turn your 3D scans into an actual 3D model. In the near future, however, ItSeez may build into the app the ability to have your objects 3D printed or other application. While it’s fairly limited for consumers, seeing an object or person transformed into a third-dimensional object (even if it’s a bad one) is cool.

Conclusion

There’s no doubt that 3D scanning (and printing) is becoming more affordable and accessible, and devices like the Structure Sensor are going to help usher that in for consumers and developers. But the Structure Sensor also shows that this type of 3D scanning is still in its infancy. It’s not that the device is bad or the apps don’t work, but right now there’s limited use for consumers and there are still things that can be improved. However, we still had fun using it, and it’s awesome to see everyday objects get turned into 3D models simply by using an iPad accessory.

As Occipital continues to improve the product and more apps are introduced down the line (not to mention other competing products and technologies that are in the works), there could be some neat applications. Until then, the Structure Sensor is a promising tool for developers and hardcore 3D enthusiasts, but it’s still early-adopter territory for most people.

This sensor turns your iPad into a 3D scanner, but get ready to practice