Adobe's Project Sky Replace could be Photoshop's next big feature

Of the many nascent technologies demonstrated at Adobe MAX this year, Project Sky Replace was one of the most intriguing. As the name implies, the tool can automatically replace a boring or overexposed sky in a photo with a more interesting one from another image. It doesn’t sound terribly complex, but there is much more going on behind the scenes to make the final image look as realistic as possible. Digital Trends spoke with Adobe’s Xiaohui Shen, Senior Research Scientist and chief engineer on the Sky Replace project, about how the technology works.

Sky Replace is smart enough to know which areas of an image need to be preserved and which need to be replaced. It can handle relatively complex horizon lines and will automatically create a matte for any shapes, such as buildings and trees, that stick out. But it goes much further than that.

One of Sky Replace’s most impressive aspects is that it doesn’t just copy and paste one sky over another; it actually adjusts the image foreground to accurately mimic the lighting and colors of the new sky. Several examples of this are given in the above video.

We asked Shen how this was accomplished, and the answer was more complex than we anticipated. It goes far beyond global exposure and white balance adjustments, and uses information from the foreground of the reference image to enhance the foreground of the original.

“We segment the foreground to different semantic regions such as grass, buildings, and people, and computer transfer functions between semantically matched regions,” Shen said. In short, this means grassy areas of the reference image affect only the grassy areas of the original, skin tones affect skin tones, and architecture affects architecture. In this way, the system ensures there are “no very weird colors” in the final result, according to Shen.

For all its power, Sky Replace is still far from being complete, and Shen was open about its current limitations. Water scenes, for example, can pose a challenge. “When there are no sky reflections in the water, our method generally works well,” Shen said. “At this point, we are not handling the cases with mirror reflections of skies, as the reflections need to be segmented and replaced as well to make the photo look realistic.”

Another challenge comes in the form of original images with highly directional light, including backlit scenes or direct sunlight with strong shadows. The issue here isn’t so much a technical limitation of the algorithm, but rather with the selection of a replacement sky. “The sky to be used for replacement should have very similar lighting directions,” Shen explained.

The software can automatically generate replacement sky suggestions based on the content of the original image. However, Adobe simply doesn’t have a large enough collection of skies to match all the possible variations in lightning. Shen said the team planned to address this issue in the future with an improved image search function.

Sky Replace has obvious value to both casual and professional photographers, but one thing not talked about in the Adobe MAX presentation was the idea of applying it to video. While it’s currently limited to still images, Shen mentioned that video presents an intriguing use for the technology, one the team plans to investigate. In theory, Sky Replace could be used in place of shooting on a green screen or going through an involved masking process in post. It could be particularly beneficial to budget science-fiction productions, which could easily replace a boring old Earth sky with an artist’s rendering of a Mars sky, for example.

As with other Adobe sneak peeks, there is no guarantee when, or even if, Project Sky Replace will become a commercial product or make it into a future version of Photoshop. The tool has been in the works for over a year, and Shen confirmed development will continue. This gives us reason to hope that Sky Replace will one day be more than a behind-the-scenes side project.

Gaming

New ‘Battlefield V’ patch gives Nvidia’s ray tracing support a chance to shine

‘Battlefield V’ is the first game to use Nvidia’s ray tracing support, now available with the RTX 2080 and 2080 Ti graphics cards. The feature can, in an ideal scenario, make the game look better, but the performance hit may not be…
Product Review

The 2019 Porsche Macan S is a luxurious and quick SUV, but it's no road tripper

The roster of models challenging the Porsche Macan grows annually. The German firm updated its smallest, most affordable SUV with a new engine, more tech features, and subtle design tweaks to keep it looking fresh.
Movies & TV

'Stranger Things' season 3 teaser reveals the new episodes' titles

With a sophomore season as strong as its first, Stranger Things is now moving on to season 3. Here's everything we've learned so far about the Netflix series' upcoming third season.
Movies & TV

Ice meets fire in HBO's latest 'Game of Thrones' season 8 teaser

With the eighth and final season looming, Game of Thrones fever has officially become a pandemic. Our list of all the relevant news and rumors will help make the wait more bearable, if you don't mind spoilers.
Photography

The best mirrorless cameras pack all the power of a DSLR, minus the bulk

Mirrorless cameras offer a lot of photography firepower, inside a compact body. Explore the best mirrorless cameras, from the pro-level to the beginner-friendly shooters, in this guide.
Photography

Photography News: Startup redesigns tripod heads ‘inside out’ for more flexibility

Well, this doesn't look like the ball heads that we've seen before. Instead of designing a tripod ball head with a small cutout, the Colorado Tripod Company created one with most of the ball exposed, allowing for more possible angles.
Photography

MIT science photographer isn’t an artist, but her work could fill galleries

Felice Frankel is an award-winning photographer, but she doesn't consider herself an artist. As a science photographer, she has been helping researchers better communicate their ideas for nearly three decades with eye-catching imagery.
Emerging Tech

Awesome Tech You Can’t Buy Yet: Folding canoes and ultra-fast water filters

Check out our roundup of the best new crowdfunding projects and product announcements that hit the web this week. You may not be able to buy this stuff yet, but it sure is fun to gawk!
Photography

Leica targets street photographers with a pricey camera bundle

Described by the camera company as "your perfect companion in the city," Leica's Street Kit comprises a Leica CL camera body, a 23mm (35mm full-frame equivalent) F2 lens, batteries, a handgrip, and a black leather carrying strap.
Product Review

Meet Z6, the breakout star in Nikon's new mirrorless lineup

The Nikon Z6 is the sibling to the new mirrorless Z7 -- but for some photographers, the cheaper Z6 may be the better option. Read where the $2,000 camera beats the $3,400 one (and where it doesn’t) in our Nikon Z6 review.
Social Media

Snapchat facial recognition could soon power a new portrait mode, code suggests

Digging into Snapchat's code suggests a handful of upcoming camera features, including a portrait mode. The feature appears to use facial recognition A.I. to blur the background. The code also suggests an updated camera interface.
Social Media

#ThrowbackThursday is only the start: Instagram hashtags for every day of the week

Not getting your hashtag fill with #ThrowbackThursday or #ManCrushMonday? Here's a list of some of the more popular Instagram hashtags, so you can outfit your next post with the proper tag, regardless of what day it is.
Photography

From DSLRs to mirrorless, these are the best cameras you can buy right now

From entry-level models to full-frame flagships, many cameras take great photos and video. The best digital cameras, however, push the industry forward with innovative sensors and improved usability, among other things. Here are our…
Photography

Photographers can now customize the layout of Lightroom Classic controls

Tired of scrolling past Lightroom tools that you don't use? Adobe Lightroom Classic now allows users to reorganize the Develop panel. The update comes along with new sharing options in Lightroom CC, and updates to the mobile Lightroom app.