ChatGPT Desktop App: New Features for Mac & Windows, Including App Integration & Voice Mode

ChatGPT's new desktop app automates work by integrating with other apps.

### Enhancements in ChatGPT Desktop Apps: Automating Tasks and Integrating with IDEs
* 👔 Kevin Wheel introduces Day 11 of the OpenAI showcase, emphasizing the advancements in ChatGPT's desktop applications for Mac and Windows that **enhance productivity and automation**.
* 🖥 The Mac desktop app, launched six months ago, is **lightweight** and allows users to interact with ChatGPT **without interrupting their workflow**, utilizing a **quick keyboard shortcut** for easy access.
* 🤖 New features enable ChatGPT to interact with other applications on the user's computer, allowing for **seamless context sharing and automation of tasks**, enhancing user experience **significantly**.
* 📊 Demonstrating the app's capabilities, John Nasto shows how ChatGPT can **generate commands for Git repositories** and create visual data representations like **bar graphs** using Advanced Data Analysis.
* 💻 Justin Rushing highlights the integration of ChatGPT with IDEs like Xcode, showcasing how it can assist in coding by **generating code snippets** and providing **real-time feedback**.
* 📝 The introduction of support for applications like **Apple Notes, Notion, and Quip** opens new possibilities for users to **enhance their writing and research capabilities** within their documents.
* 🎤 The advanced voice mode feature allows users to **interact with ChatGPT using voice commands**, making it easier to get suggestions and feedback on creative projects, like **holiday music playlists**.
* 🚀 All these features are now available for Mac desktop apps and will soon be released for Windows, significantly **improving the way users engage with ChatGPT in their daily tasks**.

[Music] Hey everybody! Welcome to Day 11. I'm Kevin Wheel, I lead product at OpenAI, and I am definitely outclassed by the two gentlemen on my right, who I'm told did not just get these suits 48 hours ago on Amazon—definitely own this already!

100%! Yeah. All right, so you might have noticed we've been putting a lot of effort into our desktop apps. Uh, so we launched our Mac desktop app about six months ago; we launched our Windows desktop app just a couple of months ago. And as our models get increasingly

powerful, ChatGPT will more and more become agentic. Uh, and that means we'll go beyond just questions and answers. ChatGPT will begin doing things for you. We're seeing that already with products like Canvas, where you're collaborating with ChatGPT to help

improve your writing and your code. And that shift will continue, and we'll—ChatGPT will do more and more on your behalf. The desktop apps are a big part of that, too, because being a desktop app, you can do so much more than you can

just in a browser tab. That includes things like, with your permission of course, being able to see what's on your screen and being able to automate a lot of the work that you're doing on your desktop. So we'll have a lot more to say on that as we go into 2025, but we've

also got some exciting stuff that we're launching today, so let's dive in. All right. Hi, I'm John Nasto, and I work on the ChatGPT desktop team. My name is Justin Rushing, and I also work on the ChatGPT desktop team. Uh, we've got a lot

To show you today, so I'm just going to go ahead and get started here. Um, so first things first, this right here is the fully native ChatGPT desktop app for Mac. Uh, it does all the things that you know we've come to expect from our clients, um, but what I

really love about it is that, being native, it's really lightweight, doesn't use a lot of resources; it uh, it lives in its own window and um, it I'm able to use it without having to context switch away from what I'm already doing. Right? So we've got this keyboard shortcut option

space that makes it really fast to show and hide ChatGPT, so it's it's always there when you need it. This button right here is our entry point for working with apps on your computer, and the way that I like to think about this feature is that we all

copy and paste things into ChatGPT right? All the time! All the time! All the time! Um, this feature makes that way smoother by, when we are working with an app on your computer, we'll automatically pull that context in for you, so you just

focus on asking your question and we we handle the rest. So you might notice that I've also got this Warp uh, console window open as well. It's currently navigated to a repository that that I'm getting up to speed on, um, and it might

seem kind of silly, but I want to figure out how many commits per day are happening in this repo. You know, we talk about velocity a lot here, so I want to see that for myself. I have no idea how to do that, though, so I'm going to

Use ChatGPT. So, when I click on this button, I'm going to see all of the apps that are currently running on my computer. ChatGPT can work with... Important note: until you select one of these, we will never look at the contents of another

app. So you are always fully in control over what you're sharing with ChatGPT. To get started, I'm going to click on Warp. At this time, a huge shout-out to the Warp team for all of their help in getting this going.

When we first announced working with apps, we did not have support for Warp, and it was, I think, literally the first request: adding support. Um, so huge shout-out to the team; they worked really hard to help us get it ready for today. So thank you so

I'm going to get started by saying, 'Write a command to get the number of commits per day over the past two months.' And now I don't need to tell ChatGPT that I use Git because it can tell from Warp that I do, um, and it's just going to

give me the command that I need. So I'll push this button to copy and paste it into Warp. And I... I think this looks right. Yeah, it looks like the right information, but it's also kind of hard to tell what we're looking at. Right. Yeah.

I'm... I'm a visual learner myself, so, um, normally what I would do is I'd figure out how to get this into a spreadsheet, make a chart there, and then find that spreadsheet again in three years. But, um, instead, I'm just going to... I'm just

Going to ask for one, so make a bar graph with all of the results. Why not make it holiday-themed? Great idea! And uh, awesome! So this is going to show off what I think is the coolest part about working with apps, which is that it works with all of the other features and

all of the other models in ChatGPT. So in this case, uh, we decided to use Advanced Data Analysis to crunch some numbers and give me back a bar graph. And what that means, if you really think about it, is that when we build features like Advanced Data Analysis and bring

them to ChatGPT, it's kind of like we're bringing them to every app that ChatGPT works with. Yeah, that's great. So while it thinks about this, um, do you want to talk a little bit about what the model is actually seeing? Is it just what we see on

the screen, or is it something more? Great question. So an easy way to do this would be to just grab a screenshot and let Vision do the rest. Um, but we actually can reach into the application to grab the screen content as well. And so, uh, these

results will contain everything here, not just what you see on screen. Yeah, I was thinking hard about this; it might be the holiday-themed part. Okay, perfect. All right, I mean, this is—that looks pretty holiday-themed to me. What do you think, John? Uh, it's

holiday-themed. I don't know if it's as holiday-themed as we are in these suits, but it's not bad. Literally nothing is as holiday-themed as you are. Yeah, perfect, perfect. But I'd say it's good enough, so I'm just going to download this and now

I can share it with a teammate, but with that, I'm going to hand it back to John to talk a bit more about programming. Great! So I think that the use case that Justin showed is really important and useful: to be able to interact with a

terminal. But I want to show what it's like to interact with code in an IDE. So I have Xcode open here, which is my IDE of choice, and it's running a sample app that is actually a little bit of a peek behind the scenes into how this

with apps feature works. The sample app uses the macOS accessibility APIs to look at Xcode and tell us some things about what's on the screen. So it's telling us that there's a text field with these dimensions; it tells us that it has 37 lines, and we can go down and

check that. 37 lines—yep! And it shows us the content of the text field. And we actually use this to make the feature.

Right, that's right. Yeah, this is a useful sample app for us for sure. So this is nice, but it doesn't do any live updating.

So I'm going to use ChatGPT to help add that feature. I'm going to bring up the chat bar with a very similar shortcut to what Justin showed earlier, but with a slight change. I'm going to use Option-Shift-1, and what that does is it

brings up the chat bar with Xcode automatically paired to it. Xcode, being the topmost app that is open that we support with this feature, it makes it super quick to start working with an app. Yeah, it's great, and you get this immediate feedback that it sees Xcode.

Here, so these accessibility APIs are a little bit inscrutable, definitely hard to remember how to use, and pretty complicated, actually. Um, so I'm going to use the model selector here and I'm going to switch this to 001. 001 is one of

our newer models here at OpenAI, and it does a great job thinking about these difficult coding problems. Um, and I should mention as well that this feature is also available with 001 Pro, if you really want to throw it deep-end coding problems. That model, yeah, all

right. So let's give it a prompt here. I'm going to say, "Add an observer; so if selection changes, `loadTextAreas` is called, and we'll kick off this request to the model." So, 001 is one of our Chain of Thought models, and you can see that

it's thinking about this issue; it's going to tell us some of the steps that it goes through as it considers...and wow, that was a pretty fast response from it. Didn't have to think too hard on that one, I guess. I guess not. You've got to give

it a harder problem next time. Yeah, wow. All right, so it's generating some code, and you know, I have a fair amount of trust in this code here. So, as soon as it's finished generating, I'm just going to copy this into Xcode, and we're going

to run it and see what happens. I don't...I don't see anything that could go wrong with that. No demos...poss...yeah, live demos work 100% of the time. It was one of the rules of the universe. All right, so I've copied that code and I'm pasting it

directly into Xcode. I'm going to take a quick scroll through it to see if it's finding any issues. Right now, it's looking pretty promising. All right, so let's run this and see what happens. You know, it would be really cool if you

didn't have to copy and paste that back into Xcode, though. That would be cool. And you know, people have been suggesting that. Should I build that? You should definitely build that. All right, PM approved. Great! All right, so uh, let's—it's running again—uh, let's take a look

if I select content. Oh no, it didn't—it didn't work like we thought. Okay, should we—should we give it one more shot? Yeah, why don't we—why don't we try? Yeah. All right, so I think I'm actually just going to go back to our previous state since I

don't have a specific error here. Uh, let's try to discard the changes. All right, let's give it one more shot. Yeah, one more shot. Yeah. And while it's working, we can uh, talk about some more of the features here. Okay, add an observer so if selection

changes, load text areas is called again. All right, maybe it didn't think hard enough on that. We'll try it again. Got overconfident. Yeah. Well, it thinks about this. I should mention that I'm using Xcode—uh, like I said, this is my IDE of

choice when working with Swift, but we do support a whole bunch of other IDEs. Um, that means VS Code, uh, the JetBrains ecosystem, which includes Android Studio and PyCharm, RubyMine, things like that. Some really uh, standby Mac apps like TextMate and BBEdit. So we've got a a

A whole lot of different support here. Yeah, I'm I'm actually unreasonably excited for MATLAB support. I would have totally used that in college. Yeah, MATLAB is is another exciting one. I think some students are really going to find that useful. Okay, it's uh it is still generating some

code. There it goes. It's done. I'm going to use this copy button again and again with full trust that everything is going to work. I'm going to... Now we know what could go wrong though, right? Yeah, sure. All right, so let's run this again and uh

see if we have slightly better luck. Okay, it's running. Hey, look at that! If I... things it changes. Wow! It's a holiday miracle! You got the incantation to the demo gods right the second time. Yeah, exactly. Awesome. So we've been talking a lot

about coding today, right? Uh, but another thing that I love to use ChatGPT for is helping me with my writing, and I know I'm not alone here. Um, and so that's why today we're announcing support for three new applications: Apple Notes, Notion, and

Quip. Uh, we think this is going to open up a brand new set of use cases for working with apps, and so we can't wait to see what you all do with it. Uh, with that, uh, John Kevin, you already know this, but for the rest of you, uh, I give

walking history tours in San Francisco outside of work. Um, big history buff. San Francisco's got a great story to tell, and I'm actually working on a brand new walking tour, and so why don't we try out this feature and help me out with that? Let's do it. Great. So

Here, I have a Notion document open on my computer. I always write my tours in Notion, and this is actually a real walking tour that I'm working on, so I hope you all find it interesting. Um, but I'm actually working on a new stop

for my favorite character in San Francisco history: Emperor Norton. Um, I know some high-level talking points: He was the self-proclaimed Emperor of the United States and Protector of Mexico, who lived in San Francisco in the 1800s, and uh, he even made his own

currency that was actually valid in the city for a while. That's something you can just do—yeah, apparently you can! And uh, I think it's going to make for a great tour stop, but I'm a little bit fuzzy on the details, and so I'm going to use ChatGPT to help me out.

Uh, one option would be to copy and paste these bullets over, and I think ChatGPT would do a pretty good job at that, but it would be helpful if it had the context of the entire document. Right? And so, instead, I'm going to have Chat

GPT work directly with Notion. So I'm going to hit Option-Space to bring up ChatGPT, have it work with Notion, and I'm actually going to go ahead and just um, highlight this stop here so that the model knows what to pay attention to, and

so now we can see we're working with Notion in the walking tour document, focused on the selected lines. And I'm just going to go ahead and say, "Fill out these talking points."  I don't need to be any more specific than that. Um, but one

The thing that is really important is that this is a walking tour, right? This is a history tour; things need to be factually correct. And so, to help with that, I'm going to push this button to turn on search. Now, to answer my question, ChatGPT is going to search the web, and

everything it tells me is going to be grounded in citation, right? And so I want to find out more about... I can go ahead and click the links, and you really start to see this awesome interaction loop pop up where ChatGPT is helping me with my research in the

context of the document I'm writing. So awesome! This looks like all of the stuff that I'm hoping to cover. And so it doesn't really sound like me, though. This sounds like, you know, official results. And so I'm going to turn off search and just say, "Uh, make it

match the style of the rest of the stops; keep it short, two paragraphs." And now ChatGPT is going to go out and read the rest of my document, learn how I talk and how I've written the rest of these, and do its best to imitate that. And so, awesome!

This looks great! Let me introduce you to one of San Francisco's most beloved characters. You'll have to come to the tour to find out the rest. Sounds like me, so I'm just going to go ahead and highlight this to copy and paste it back

into Notion. And, of course, you know, I want to iterate from here; I'd want to refine this. Um, but that's just a quick example of using ChatGPT to work with Notion. That's awesome! I think that it's really compelling to work with your

Documents like this, um, not just code like I showed before, but your written prose—um, this is excellent, but it's just one way to work with the model, this sort of text-in, text-out method. And what I'd like to introduce today is support to

use advanced voice mode. With this feature, we're really excited about this. We think that this paradigm is a really interesting way to talk with ChatGPT and get insight into your documents or your code, for that matter. All right, so we're going to show one of mine in a

second here. Um, when I'm not at OpenAI, I spend a lot of time as a professional saxophone player, and I've got this holiday party set list that I'd like some input on. I think ChatGPT would be great at this, but again, I want to

use the voice to do this. Is this why your holiday parties are all so good, by the way?

Yeah, the suits help for sure. Yeah. All right, so, uh, yeah, I want someone special to really help out with

this. Um, so let's bring up ChatGPT again with the shortcut that I mentioned earlier, which is Option-Shift-1, and that brings up our chat bar. It's automatically paired with notes, but this time, instead of typing to the model, I'm going to use this new advanced voice

mode support. So I'll use this icon in the bottom right to bring in our special guest.

Santa!  It is so great to talk to you. Listen, I've got this holiday set...

list that I'm working on, and I've got a couple of songs that I'm just not sure about. This one, I've got my love to keep me warm... I honestly don't really know that song very well. Can you recommend something else? How about replacing "I've got my love to"

keep me warm" with "Frosty the Snowman"? Oh, that's... that's a good idea. Let's see... the Snowman... Did I get that right? Not quite. Oh, keep going. Sure thing. So, is there anything else you'd like to know about your holiday set list? I... I think Santa was

bored with the copy editing, but we can move on. Uh, let's talk about the last song of the second set: "Al Lang Z." I mean, that's a little bit dreary. I'm hoping for something more energetic to really get you and the elves dancing. What do

you think? What should I replace it with? "Deck the Halls" would be a perfect replacement. It's jolly, festive, and sure to get everyone dancing. Santa, I think that's a great idea. I think that song's really going to slay.

Voice model like this is such a great way to talk to your documents and get live feedback on what you're doing. Totally. And, uh, good news—right before we stepped onto this live stream, actually, we pushed the button to ship all of the features we just showed off,

so um, they should be available for you on your computer. Just make sure that you are updated to the latest Chat GPT app for macOS. Awesome. So that's Day 11. It's about using the desktop apps to see, understand, and automate the work that

you're doing with ChatGPT. Uh, this is coming, this is available today, as of, what, 20 minutes ago, on our Mac desktop apps. Coming soon to Windows. I love the desktop apps that they're... they're in the background when you're doing work and then it's a

keyboard shortcut and they're right there and able to help you do whatever you're working on. So that's day 11. We have one day left. We'll be coming to you tomorrow morning, day 12. We've got something super exciting, so don't miss it. Yeah, we

can't wait to get these features out to you. We're really excited, but in the meantime, I've got to start practicing this. Yeah, that Santa recommended. All right, let's see [Music] w

Work with Apps—12 Days of OpenAI: Day 11

OpenAI’s mission is to ensure that artificial general intelligence benefits all of humanity.



OpenAI

Introduction of ChatGPT Desktop Apps and Future Directions

Focus on desktop app development for Mac and Windows.

ChatGPT becoming more agentic, moving beyond Q&A to performing tasks.

Desktop apps enable greater functionality than browser versions.

Future plans include screen access and desktop automation (with user permission).

Showcase of ChatGPT Desktop App Features on Mac

Lightweight, native app with improved performance and context switching.

Keyboard shortcuts for quick access and hiding.

Integration with other apps, automatically pulling context for smoother interaction.

Example: determining commits per day in a Git repository using Warp.

Generating bar graphs from data, leveraging Advanced Data Analysis.

Access to application content beyond what's visually displayed on the screen.

ChatGPT Integration with IDEs (Xcode Example)

Interacting with code within Xcode, using accessibility APIs.

Using keyboard shortcuts to bring up the chat bar with Xcode automatically paired.

Utilizing the 001 model for complex coding problems.

Generating code to add live updating functionality to the Xcode app.

Support for various IDEs: VS Code, JetBrains suite, TextMate, BBEdit, and MATLAB.

Announcing support for Apple Notes, Notion, and Quip.

Example: using ChatGPT to enhance a Notion document for a San Francisco walking tour.

Working directly with Notion, using context from the entire document.

Utilizing web search to ensure factual accuracy with citations.

Matching writing style to the rest of the document.

Using voice interaction for feedback on documents or code.

Example: using voice mode to improve a holiday party set list.

Using a keyboard shortcut to bring up the chat bar and using voice mode.

Interacting with a simulated Santa Claus for song suggestions.

Features are now available on macOS and coming soon to Windows.

Highlighting the convenience of background operation and keyboard shortcuts.