Skip to main content

Anthropic’s Claude can now control computers like people do

the claude computer control logo
Anthropic

Anthropic’s already impressive Claude 3.5 Sonnet gains a significant performance boost on Tuesday as the generative AI startup rolls out an enhanced and updated version of the model alongside the new, lightweight Claude 3.5 Haiku. The Sonnet update includes a public beta feature that gives the AI basic control over the computer it’s running on.

Claude 3.5 Sonnet was already a performance leader when it comes to coding tasks, but the new version shows significant across-the-board improvements over its predecessor and steadily outperforms both Gemini 1.5 and GPT-4o on a variety of industry benchmarks. Gemini 1.5 Pro was the only model to best the new 3.5 Sonnet on any test, and did so on the MATH benchmark.

Recommended Videos

The new 3.5 Haiku is no slouch, either, despite its small size. Scheduled to be released later this month, 3.5 Haiku outperforms Claude 3.0 Opus, the company’s largest last generation model. Like its larger version, the new Haiku is exceedingly proficient at coding tasks, scoring 40.6% on the SWE-bench Verified — higher than both GPT-40 and the original 3.5 Sonnet.

new Claude 3.5 sonnet performance chart
Anthropic

Even more impressive, the new Claude 3.5 Sonnet can now interact with desktop apps via the “Computer Use” API. The AI can generate the necessary keystrokes, mouse clicks, and movements needed to emulate the human user. The company is quick to point out that the system is currently quite experimental and prone to errors. The underlying purpose of the public beta release is to elicit feedback from developers to rapidly improve the API’s performance.

“We trained Claude to see what’s happening on a screen and then use the software tools available to carry out tasks,” Anthropic wrote in a blog post. “When a developer tasks Claude with using a piece of computer software and gives it the necessary access, Claude looks at screenshots of what’s visible to the user, then counts how many pixels vertically or horizontally it needs to move a cursor in order to click in the correct place.”

Claude | Computer use for automating operations

It’s an AI agent, essentially. That is, its an AI that can automate other software processes, whether that’s generating and qualifying marketing leads, uncovering patterns and trends in medical data, or simply navigating to a specific website and filling out a form you need. Think of them as a more advanced version of existing Robotic Process Automation systems.

The company cites Asana, Canva, Cognition, DoorDash, Replit, and The Browser Company as early adopters of the new feature. Replit, for example, is using Computer Control to “develop a key feature that evaluates apps as they’re being built for their Replit Agent product,” per the announcement.

There’s no need to worry about the AI going all Skynet on us (yet), as Anthropic explains. “Humans remain in control by providing specific prompts that direct Claude’s actions, like ‘use data from my computer and online to fill out this form,’” an Anthropic spokesperson told TechCrunch. “People enable access and limit access as needed. Claude breaks down the user’s prompts into computer commands (e.g., moving the cursor, clicking, typing) to accomplish that specific task.”

Anthropic also concedes that Computer Control could be misused to generate spam, spread misinformation, or commit fraud. In response, the company has developed new classifiers that identify when the API is being used and whether that use is “causing harm.”

Andrew Tarantola
Andrew Tarantola is a journalist with more than a decade reporting on emerging technologies ranging from robotics and machine…
Intel CEO says that Lunar Lake was ‘a one-off’
Intel CEO Pat Gelsinger presents Intel's roadmap including Arrow Lake, Lunar Lake, and Panther Lake.

Intel's CEO Pat Gelsinger talked about the future of its top processors in the company's latest earnings call. Apart from reporting a huge $16.6 billion loss, the earnings call revealed a bit about next-gen products like Panther Lake and Nova Lake. According to Gelsinger, those two generations of laptop CPUs will not follow in Lunar Lake's footsteps. In fact, Gelsinger referred to Lunar Lake as "a one-off."

Lunar Lake introduced a first for Intel -- at least in terms of consumer processors. It came with on-package LPDDR5X memory, which brought Intel closer to some of the highly successful M chips manufactured by Apple. On-package memory can improve data transfer speeds and boost efficiency, and Lunar Lake was also proven to have solid battery life. Despite these benefits, Intel isn't going to give Lunar Lake a direct successor.

Read more
Lenovo’s latest laptop had an edge on the MacBook Air until this week
Lenovo Yoga Slim 7i Aura Edition front view showing display and keyboard.

There aren't many options in the 15-inch laptop category, which has given way to slightly smaller 14-inch laptops and larger 16-inch machines. But the MacBook Air 15 stands out as one of the best options you can buy today, as long as you don't need Windows.

The Lenovo Yoga Slim 7i Aura Edition is a new alternative that runs Intel's latest Lunar Lake chipsets that are aimed primarily at efficiency -- aimed most directly at Apple's efficient M3 chipset. Can the Yoga Slim 7i compete? Perhaps before this week. But now that the MacBook Air 15-inch starts with 16GB of RAM, the Yoga Slim 7i's advantages are lessened.
Specs and configurations

Read more
Apple just acquired this beloved Mac app
Editing a photo in Photometer.

Popular photo-editing app Pixelmator published a blog post today announcing its plans to join Apple. The post was spotted by 9to5Mac.  The acquisition is still pending approval, and the Pixelmator team has confirmed that it won't be making any changes to its apps "at this time."

The company offers two apps -- a professional image editing tool called Pixelmator Pro and a free photo-editing app called Photomator. Pixelmator Pro is available for a one-time fee of $50, making it much more affordable and easier to buy than competitors like Photoshop.

Read more