Microsoft continues AI push with expanded Bot Framework, new Cognitive Services

microsoft build 2017 cognitive bot framework harry shum
During the Build 2017 keynote on Wednesday, Harry Shum, executive vice president of  Microsoft’s Artificial Intelligence and Research Group, detailed the company’s plans to bring artificial intelligence into the hands of every developer. This initiative includes making enhancements to the Microsoft Bot Framework, adding new AI-based “cognitive” services and a new “lab” for testing experimental cognitive services, incorporating artificial intelligence into additional products and services, and more.

For starters, Microsoft’s Cognitive Services suite brings machine-based intelligence spanning vision, speech, language, knowledge, and search to applications. They’re the backbone of features like facial recognition, speech recognition, text translation, image search, and so on. Microsoft now provides 29 distinct Cognitive Services, with Bing Custom Search, Custom Vision Service, Custom Decision Service, and Video Indexer added to the portfolio Wednesday.

microsoft build 2017 cognitive bot framework custom vision service

Cornelia Carapcea, Microsoft senior product manager, came on stage to demonstrate Custom Vision Service. This enables developers to create their own custom artificial intelligence models across speech, language, text, and image understanding. To create a model, developers merely bring their training data (a dozen photographs, for example), click a button to train the model in minutes, and then create the final model to be used in applications.

In the onstage demo, Carapcea used a smartphone app to recognize a just-plucked plant in a photograph taken by the device. She pointed out that the recognition aspect feature didn’t require deep learning, thousands of training images, and a wait time of “hours” as the model learned that specific plant.

Prior to the demo, she uploaded around 24 images of that specific plant to a photo album stored on the Custom Vision Service platform. She also had loads of other plants uploaded, and she eventually clicked on a big green “train” button to create what she called an “end point” that would be used by the mobile app to recognize a variety of plants.

She said that once a model is created, “active learning” kicks in to analyze all additional images taken by users of the app. This feature will hold on to all images that best define the plant visually, and then present those selected images to the developer along with a “confidence” percentage. The developer clicks on each questionable image to save it as a stored reference and then retrains the model, thus making it “smarter.”

To see this feature in action, head over to Microsoft’s dedicated Custom Vision site.

microsoft build 2017 cognitive bot framework starship commander

In another demo demonstrating the power of Microsoft’s Cognitive Services platform, creative director Alexander Mejia of Human Interact talked about a pre-alpha build of Starship Commander, a choose-your-own virtual reality adventure narrative. The game uses natural language processing enabling players to have “real” verbal conversations with the in-game characters. What’s cool about this aspect is that the studio created custom words based on the game’s science-fiction theme, which isn’t possible using off-the-shelf voice recognition software.

One of the services used in the Starship Commander backend is Language Understanding Intelligent Service (LUIS), which enabled Human Interact to create the custom language model. The game also relies on Custom Speech Service so that it understands all the sci-fi lingo spoken by the players.

“With LUIS, you basically give it a couple of statements,” Mejia said. “You crunch it, you start typing in your own statements, and it’s shockingly scary how good it picks up, even stuff you didn’t put in there.”

Microsoft also launched the Cognitive Services Labs platform, enabling developers to play with experimental cognitive services. One of the first experiments on the plate is a gesture API that enables end users to control and interact with apps using simple hand motions. Shum indicated more experiments are coming.

“The improvements we are making in understanding speech and language are driving a paradigm shift — moving away from a world where we’ve had to understand computers to one where computers understand humans. We call this conversational AI,” Shum said in press release.

In addition to new Cognitive Services, the company announced expansions to the Microsoft Bot Framework, a platform for creating intelligent “robots” within apps and services. Since the platform’s launch in 2016, more than 130,000 developers have signed on, and now Microsoft is enhancing the platform with “Adaptive Cards” that can be used across multiple apps and platforms.

“Developers also can now publish to new channels including Skype for Business, Bing and Cortana, and implement Microsoft’s payment request API for fast and easy checkout with their bots,” Shum added.

Microsoft also injected its Azure cloud platform with a service called Batch AI Training, which enables developers to train their own deep neural networks using any framework they choose, such as Microsoft Cognitive Toolkit, TensorFlow, and Caffe. Developers can create their environment and run these models against multiple processors, multiple graphics chips, and “eventually” field-programmable gate arrays (FPGAs), which are integrated circuits that can be configured by developers.

microsoft build 2017 cognitive bot framework translator powerpoint

Finally, Microsoft announced a new Office feature called Presentation Translator. Available in PowerPoint, it uses the company’s AI capabilities to translate PowerPoint presentations between multiple languages in real time. Attendees of a meeting can use a link generated by Presentation Generator to see the translation on their device. It joins PowerPoint Designer and Office Researcher, two previous features that used the company’s AI knowledge.

According to Yina Arenas, Microsoft principal program manager, the Presentation Translator add-in relies on the Custom Speech Service and Translator Speech API services offered on Microsoft’s Cognitive Services platform. To demonstrate the new add-in’s use, she went into PowerPoint and clicked on the “Start Subtitles” button. A pop-up menu appeared that allowed her to choose the language she would verbally use in the presentation (Spanish), and the language she wanted to use in the subtitles (English). She also chose where the subtitles would reside in the presentation.

Now here’s the kicker. Shum downloaded the Microsoft Translator app to an iPhone, scanned a barcode on the PC’s screen (users can also enter a unique conversation code), and then chose his native language — Mandarin Chinese. After that, Arenas moved on to demonstrate the real-time translation across multiple devices.

“Artificial intelligence can eliminate the linguistic barriers between the presents,” Arenas said aloud in Spanish. Meanwhile on the PC monitor, her words appeared in English while they also appeared in Chinese on Shum’s iPhone. After that, she unmuted the presentation, allowing Shum to verbally make a statement in Chinese through the iPhone’s built-in microphone. The statement appeared on the PC monitor in English.

“The PowerPoint Translator add-in custom-trains the model based on my voice and my slides using the power of the Custom Speech Service,” Arenas said.

Overall, these changes don’t have much immediate impact on consumers. However, Microsoft hopes that its Cognitive Services and Bot Framework will allow the design of next-generation applications that use AI to interact more intuitively with users. Only time can reveal the wisdom in this approach, but the company’s continued commitment in these areas, which were first detailed in-depth during last year’s Build conference, shows the company expects big things from these technically complex initiatives.

Updated 5-10-2017 by Kevin Parrish to reflect three onstage demos.

Emerging Tech

A.I.-generated text is supercharging fake news. This is how we fight back

A new A.I. tool is reportedly able to spot passages of text written by algorithm. Here's why similar systems might prove essential in a world of fake news created by smart machines.

Renault-Nissan-Mitsubishi group uses Microsoft cloud platform for connected cars

The Renault-Nissan-Mitsubishi Alliance is launching a new cloud platform for its cars. Based on Microsoft Azure, the Alliance Intelligent Cloud will enable features like connected services and over-the-air updates.
Movies & TV

Netflix confirms it won’t be a part of Apple’s new video-streaming service

Netflix has confirmed that subscribers to Apple's new video streaming service won't have the option to view Netflix content on it. Apple is set to unveil its new TV service next week.

Nvidia’s smart paintbrush tool turns simple doodles into works of art

Nvidia has announced a new deep-learning-enhanced drawing app that can turn your doodles and sketches into highly realistic images. With Nvidia's GauGAN, simple sketches can be automatically embellished with more vivid details.

Delete tracking cookies from your system by following these quick steps

Cookies are useful when it comes to saving your login credentials and other data, but they can also be used by advertisers to track your browsing habits across multiple sites. Here's how to clear cookies in the major browsers.

Get the Surface Pro 6, with keyboard included, for $1,000 at Microsoft

Thinking of buying a Surface Pro 6? Microsoft is currently running a deal on its latest Windows 2-in-1, letting you bring one home for $1,000 with the keyboard included in the price.
Product Review

Acer Predator Triton 500 review

Nvidia’s new RTX 2080 Max-Q is the fastest GPU you’ll find in any laptop, but it usually comes at a steep price. Acer’s Predator Triton 500, starting at $2,500, makes it a little more affordable. But what must you sacrifice in the…

T-Mobile goes after big cable companies, pilots wireless home internet service

In a shot at big cable companies, T-Mobile is launching a new pilot program to bring an unlimited wireless LTE home internet service to up to 50,000 homes across the United States by the end of 2019.

Type away on the best iPad keyboard cases, from the Mini to the Pro

Whether you're looking to replace your laptop with a tablet or merely want to increase your typing speed, a physical iPad keyboard is the perfect companion to the iPad. Check out our top picks for every available iPad model.

Tablet or notebook? Our favorite 2-in-1 PCs give you the best of both worlds

If you can’t decide if you need a tablet or a notebook, then don’t bother. The best 2-in-1 laptops are both, and they can provide all the power you need. Check out our list for the best 2-in-1s for any user.

How the Google Stadia could lead to a new era of multi-GPU gaming

Google's Stadia could use more than one graphics card to deliver the high-performance visuals it's promised. If that leads to better developer support for multi-GPUs, could that mean gaming with two or more graphics cards could finally be…

Intel gives a peek at what its Arctic Sound GPU could look like

A new set of concept images shown at GDC 2019 is providing a peek at what Intel's upcoming modern discrete GPU, code-named. Arctic Sound, could end up looking like when released in 2020.

Here are the best Chromebook deals available in March 2019

Whether you want a compact laptop to enjoy some entertainment on the go, or you need a no-nonsense machine for school or work, we've smoked out the best cheap Chromebook deals -- from full-sized laptops to 2-in-1 convertibles -- that won't…

From Chromebooks to MacBooks, here are the best laptop deals for March 2019

Whether you need a new laptop for school or work or you're just doing some post-holiday shopping, we've got you covered: These are the best laptop deals going right now, from discounted MacBooks to on-the-go gaming PCs.