I'm here today with Daniel Heumann, who's the CEO of Intelligent Editing, a company that makes products for editors.
And a quick anti -disclaimer, Daniel advertised on the Grammar Girl podcast, I don't know, 10, 12 years ago, something like that.
This is not a paid placement.
This is not an advertisement.
I'm just having Daniel on today because I think his position about AI is really interesting as the CEO of a company for editors who is putting out a new AI product, but he has also said pretty negative things about AI in the past. So I'm really excited to hear how he's thinking about this.
Daniel, welcome to the podcast. Thank you so much for having me on here.
If you put your disclaimer there, I want to put mine, which is that you have such amazing guests on here.
And compared to the language knowledge that they have. This is really exciting, but I don't think I can quite compare.
I'll do my best. Well, I love that I get to talk to interesting people every week and people have different areas of interest and yours are just as interesting.
I mean, artificial intelligence is really important for editors and writers and teachers and most of the people in my audience.
So I think to start out, when you think of a product that is a tool to help editors, marketers i think a lot of people might think oh kind of like grammarly and you know you're not like grammarly so could you sort of explain what what your product does and and how it's different one thing about that is i you know for a long time i did try and start with explanations if people ask what is perfected and you know make sure i did not mention grammarly in my answer and a few years ago i just gave up you know the simplest thing to say when you ask about our first product, which is Perfecta is it's like
Grammarly, but for professional editors.
And at that point, when I say that 90 % of people don't want to know anything more.
So that's fine. It's actually a really good way of seeing who's truly interested in it.
But when people dig a little bit deeper, the thing that makes it so different to Grammarly is because the audience, we're doing software for editors, for medical writers, for proposal writers, for lawyers, for technical writers.
Grammarly is a product for everyone.
And for people like that, focus on consistency, on house style, on style manuals, on acronyms in the right order.
Are they each one defined?
Lots of much more technical things that really matter to that audience where a product that's designed for everyone might slow them down.
Yeah. So if you have a 200 -page document, it will tell you if you capitalize the same word every time that you're supposed to capitalize throughout the whole document for example is a really simple example of what it might do exactly and that's really useful for professionals because they will understand that sometimes that's intentional and sometimes it's not so you put that software in the wrong person's hands and you can make the document so much worse with perfect it there's you know all sorts of reasons the software is not artificial intelligence it doesn't know oh this is the compound adjective
so you hyphenate it it just knows hang on you've got this in hyphens in one place and not in another take a closer look one of these might be wrong and in the hands of a professional that's like oh yeah click click click those are the wrong ones these are the right ones and then and you hit that highest level of document excellence much faster but for average person in the street they would wouldn't want that wouldn't appreciate it and could probably make a document a lot worse by just by by not thinking through what it's suggesting.
Yeah. But I know a lot of professional editors who use Perfect It and really like it.
So about a year ago now, you launched a product called DraftSmith that actually does have AI in it.
And yet I have heard you say the gem of DraftSmith lies in how much I hate AI writing.
And about a year ago, you said AI is a terrible editor.
So what is going on with With you putting AI in a product called DraftSmith when you have these negative feelings about AI.
Let's be clear, though, the very origin of DraftSmith is fear, right?
I think everyone had the same reaction when they first saw ChatGPT, which is something like, oh, well, that's it.
I'm done here. This has been fun while it lasted.
I really enjoyed this writing and editing stuff, but no more.
And then you go a little deeper and you go, oh, this makes a lot of mistakes.
mistakes and it came through from from looking there and seeing what was good and what wasn't but now i will i will stand by every one of those comments even as we do have an ai product so what was i saying i said that i find it awful for writing and i do i spent after after that initial fear i spent a lot of time experimenting with ai writing and what would do for me and of course you do ai writing and you get to a draft really fast you get to a structure really fast and and And I could get to finished articles.
And what I realized is they weren't as good and they were taking a lot longer than it was when I did the writing for myself.
And one of our advisors, Ivy Gray, she's written on the WordRate blog and she wrote about writing is thinking.
And that's the key on those tasks that why is writing so important?
It's not because you're actually typing.
It's because you're going through a cognitive exercise as you write, which is, is this thing I'm saying correct?
Is this thing not? What are people going to, how are people going to respond?
bond? What's the answer to this?
If I phrase it this way, will that help or hinder the thesis?
And that's what makes something worth reading, right?
That's where the real value is in that thought process.
So if you skip that thought process and just give me a draft, which then I'm editing, there are chunks missing.
And even on a structure, which seems so good, like you get the AI to do a structure, it seems like it's got all the arguments in the right place because you haven't thought through it.
Or at least for me, because I hadn't thought through it, I just found, Yes, I can get there, but it's taking me a lot longer and I like doing it a lot less.
So I really didn't like AI writing.
And as for AI editing, I think what I said was it makes terrible suggestions.
Well, of course it does.
I mean, an AI is going to go wrong.
It's going to do the wrong thing.
It's going to be a percentage of time.
It's going to be right, maybe 80 % or 90 % of the time.
But that's still a lot wrong, right?
If you are correct 80 % of the time, one in five times you're seeing these suggestions and they're wrong.
Even if you get it up to 95%, that's a great number for a tool to hit.
One in 20 is really slowing a professional down.
So yeah, I don't like AI writing.
AI does produce terrible suggestions, and yet there are uses for it where I do think it's helpful.
for specifically where i found ai to be useful was in individual sentence rewrites that was that was my moment of okay the ai is neither so completely knocking us all dead and it's not awful it can do this thing really well i gave it like a sentence and said put this in plain english and that's like oh wow you did it you did a really good job of that and you did that in an instant can you do that again oh my god you do that almost every time and that's where it's like But OK, yeah, maybe one in 10 and one in 20, it's not doing well, but it's getting a very high level of accuracy.
So when you're wanting it to rephrase one sentence, it can do that well.
But then from an editing standpoint, obviously that is not an efficient way to work.
So I think that's sort of the idea behind DraftSmith is making it do one sentence at a time more efficiently.
Is that correct? Exactly right.
You cannot possibly take one sentence or even a paragraph really, but one sentence, chuck it into a browser, tell it to rewrite it the way you want, chuck it back, overwrite, that's no good.
Drow Smith was how can we make that process something that's useful to an editor?
And first thing that's got to go is prompts.
I know people talk a lot about prompts.
I know people love their prompts and share their prompts, but there's a wonderful quote.
Someone said, it's a medical writer and said, medical writers are not prompt engineers.
And that's just really a basic, obvious point, but they shouldn't be.
The way in which we all learn to work with AI should not be everyone goes out and figures out prompt engineering.
The AI should work our way.
It should do the things that we want it to do.
So get rid of prompts, replace them with buttons that would be useful for editors that are underneath it, lie prompts, but you don't need to learn what makes the best one.
So here's a button that instructs it to do a sentence one way or a sentence another way.
And then to make that useful to an editor, bring that into Word. Make it work with track changes so you can see every single thing that the AI is proposing.
Present it in track changes so you can see what it's proposing.
And when you apply it, you see it in the document and it works with track changes.
So So showing markup and then applying markup.
And then thinking about what makes a document a document for professionals.
When you're working with ChatGPT or any of the other AIs, what you see is text.
And when you write a document as an academic, as a technical writer, you don't just use text.
You use things like formats and links and citations and footnotes.
And if you paste in text from any AI app in a browser, it just overwrites all of those.
And all of the work that you've put into formatting, links, footnotes, just immediately disappears.
And you've got no trace of it if it's not working with track changes properly.
So it's really frustrating to use.
And DraftSmith set out to solve those problems. And some of them are tricky.
I'm not going to say we solve them all perfectly, but we put thought into every element of that.
So it works in Word. It shows markup.
And then for each one of those things, like formats and links, we've got a different approach to make sure it preserves what was there.
And at that point, you've got something that is efficient.
You're not copying, pasting anywhere.
You're not thinking about prompts.
You're not thinking about changes.
You see what the AI wants, you think about it, and you can accept or reject or not.
Yeah, and it works in Word. I don't use Word, so I haven't tested it or looked at it really in more than a year.
I looked at it originally when it came out.
You were kind enough to give me like a web trial or something like that.
But, you know, I'm just not a Word user.
So does it work on PC and Mac in Word?
So what Microsoft has done is pretty magic these days.
Now, if you design something like this, it works on the browser version.
It works PC version, Windows version, even works on the iPad version.
So it works completely across everything.
And I mean, hopefully it goes really well.
And so DraftSmith 1 was 15, 16 months ago.
DraftSmith 2 is tomorrow.
and and when when people are listening to this it will already it'll already be out so uh hopefully that goes well and hopefully it's not you know we could design this in other in other applications too but we'll certainly start with word because that's where most editors and and writers live yeah interesting i'm so not a word user i didn't even know there was an ipad version so yeah so so okay so it's been 15 or 16 months since the first version came out i can't believe it's been that long and so now you have this new version and i'm really curious Because the pace of AI seems to be changing
so fast, where it is today versus where it was a year that long ago is pretty significantly different.
As you've been building your product, how has that affected your product?
What have you seen in terms of AI and how it handles sentences like that?
is in your you know the the ai titans will say it's you know vastly better but like what do you think is it is it vastly better or is it just like sort of kind of okay maybe fine it's a little better i would be the first to admit i can't keep up right while we've been talking they probably released what's been 10 minutes they probably released five more models like the pace is completely ridiculous and we can't be testing every single model as soon as they come out it's It's not remotely realistic.
What did we observe there?
We observed something very strange that I will say what it was and I can't speculate on why, but people can try, which is that we started off on GPT 3 .5 and we found its suggestions, when we kept the model the same for all that time, got slowly a little worse.
Not going to try and explain that.
That's just the way it is.
When we released the product right at the beginning, the suggestions were a little bit better than what it was delivering at the end of its time on on gpt 3 .5 in terms of what's possible now we did a lot of experiments you know we can't we can't do every single one but we did a lot of experiments we have an analytical linguist and as you'd hope from a linguist he's obsessed with language in the most wonderful way so when i did the first round i would look at you know 20 or 30 tests of is this prompt right is this good enough and he'll look at a thousand and he'll split that out between between different
types of document and so he will will analyze as best he can which model is best trying to take away there's always gonna be a little bit subjective in terms of is this rewrite good or not but you know in putting himself into the the mind of an editor would you make that change or not or is it doing something wrong and what did we look at we looked at a few of the GPTs.
And we found, interestingly, that for us, GPT -4 -0 Mini was better than GPT -4 -0.
And that obviously wouldn't be the case if we were doing something that's more complex. But a sentence rewrite is relatively simple for an AI.
We have a framework that we use that might be helpful to think about this for everyone, which is designed by a chief engineer, and he calls it the the person in the box framework and obvious nod to alan turing but it's not it's not the imitation game he his question of will an ai be good at this is think of a task and then imagine a person in a box doing it and a person just it's a person who hasn't had any specific training just you know general general task and if you give it two tasks one would be edit this document and put it into plain english and the other is edit this sentence and put it
into plain english almost identical sounding tasks but really really different for an ai and the task that we're giving it the sentence is such a bounded small task that when you picture that person in the box you go oh yeah they probably could do that because it's small it's straightforward it's you kind of know what you're saying and the whole document actually when you break that down across a document that could mean a ton of different things the room for going wrong is so high and so i don't think i I think it's for that reason.
We find it does well without going to the largest, latest models because the task is relatively straightforward and the value in the software is from making that useful.
As we said, you don't want to be copying and pasting each one.
It's from giving that in a way that makes an editor's work efficient.
Did you find that the new models were dramatically better than 3 .5?
Yeah. Massive improvement between, I think the ones we tested in the most depth were GPT -4 .0, GPT -4 mini and GPT -3 .5 and 4 mini won by a really good margin.
Whether you're lounging poolside, hitting the beach or relaxing at home, Rosetta Stone makes it easy to fit in a few minutes of language learning.
Rosetta Stone is the trusted leader in language learning with 30 years of experience, millions of users and 25 languages to choose from, including Spanish, French, German and Japanese.
Rosetta Stone immerses you in your new language naturally, helping you think and communicate with confidence.
And their built -in TrueAccent speech recognition technology provides real -time feedback, helping you sound more natural.
I really like that part.
And Rosetta Stone adapts to your lifestyle with flexible, on -the -go learning.
Learn anytime, anywhere, on desktop or mobile, whether you have five minutes or an hour.
So don't wait. Unlock your language learning potential now.
Grammar Girl listeners can grab Rosetta Stone's lifetime membership for 50 % off.
That's unlimited access to 25 language courses for life.
Visit rosettastone .com slash grammar to get started and claim your 50 % off today.
Don't miss out. Go to rosettastone .com slash grammar and start learning today.
With summer in full swing, you start to feel that familiar urge to refresh your closet.
But why waste money on pieces you'll only wear once or for just one season?
That's where Quince comes in.
Their clothes are timeless, feel luxurious, look elevated, and the quality is way beyond what you'd expect for the price.
Think 100 % European linen tops starting at $30, washable silk dresses and skirts, and soft cotton sweaters, versatile warm weather pieces you'll reach for again and again I still love my poplin a line maxi skirt in blue it has pockets and I wear it all the time multiple times a week it's super comfortable and made of really high quality fabric so find something you love and give your summer closet an upgrade with quince go to quince .com slash grammar for free shipping on your order and 365 day returns.
That's Q -U -I -N -C -E dot com slash grammar to get free shipping on your order and 365 day returns.
Quince dot com slash grammar. Is there a way that you can explain what makes it better?
I know it's a very subjective thing, like this change to this sentence, is it better than another change to a sentence?
But is there sort of, are there any big picture thoughts that you have about in what way the new changes are better than what they were?
I mean, for us, it becomes easy when you do what our linguist is doing, which is check it a thousand times, right?
Because it's how many times is it bad, right?
It goes back to AI is going to give you some terrible suggestions and it's not going to be completely consistent.
So if you check something a thousand times each one starting a new window not not follow -on queries or anything like that new sentence new new query and you give it a thousand goes well sometimes it's going to produce bad things it's going to miss something if you tell it expect like things like the reduced word count function sometimes it's going to miss meaning well that's actually not uh not subjective at all it's objective have you captured the meaning of the original in this in this shorter version now you could you could also do another objective measure might be how much shorter but just
by looking for bad suggestions that's that's where people get really annoyed our audience does not like bad suggestions they are fast they are efficient and if the AI is wrong it's not necessarily we're going to hear about it but you know people are going to stop using it they don't want it so it's really important that we find the one that is giving good suggestions the most. And once you eliminate the bad ones, the ones that have missed something, the ones that are flat out wrong, that's not such a subjective exercise.
That's doable. So I see.
So it's not that the new model is a more nuanced, brilliant writer.
It's just that it makes fewer mistakes.
I would say, yeah, that's right.
It is correct more of the time on average.
Interesting. And so on the back end, are you actually submitting every sentence individually to the model?
Yes. So that's the big difference in what we did 15 months ago and what we've done now.
And I like to joke about this because it's not what we're doing is so complicated.
What we do is we produce something and our audience are mostly editors and editors are great at feedback.
And what we do is we listen to them and we build what they say and what they said based on the first one.
We're feeding it one sentence at a time.
And essentially what they said is that that's giving very good AI is giving good results.
i see why you like that but we think in paragraphs and so for that reason what we did is we in version two is we built we are feeding the ai one sentence at a time but we're presenting it back one paragraph at a time and so an editor can look through and see okay well here's here's my paragraph the thought is still the same as it was i like that improvement i don't like that improvement i like that one that one and i'll go with those things so it becomes a very efficient way of doing things like technical edits or readability or trimming word counts.
Wow. So at this point, I absolutely have to ask you about energy use.
So imagining, you know, even a 50 page document submitting a query, a prompt for every sentence in that document, that is a lot of prompts per document.
And, you know, I've been concerned about energy and water use on AI for a long time.
And I've read about it.
I've read reports, opinions, everything.
I read everything I can get my hands on about it.
And I have still not been able to wrap my head around how bad it is.
Like I have seen credible reports that say, you know, 300 chat GPT prompts are no worse than having a hamburger in terms of water use.
I've I've seen reports that say, you know, the average chat GPT use for a single person for a year is way less CO2 emissions than taking one transatlantic flight.
Right. But then, you know, I've heard other things about how terrible it is.
And recently, you know, some of the AI CEOs were before Congress.
And, you know, Eric Schmidt, for example, from his own mouth, I wrote it down because it was sounded so bad.
bad. They were all begging Congress for more energy.
And Eric Schmidt said, we need energy.
The numbers are profound.
And I'm like, profound?
That sounds bad. So I just, I cannot figure it out.
And it occurred to me, even before we talked, it's like, you, you, you are the CEO of an AI company, which means you have to pay for your use, which may be some sort of indicator of how how much energy it's actually using.
So what are your thoughts here?
What can you tell me?
I mean, you and me both, it's really not clear.
There are a few sort of rules of thumb.
The one I like is, the one I really like is thinking about what the word server means anyway and data center.
And the description is, a server is just someone else's computer.
And once you hear that, you can't unhear it.
It's like, oh, okay, I kind of get what that usage is.
And obviously, it's better than my computer, because it's running these fancy models that I couldn't run on mine.
But okay, there's this computer running somewhere, let's stop calling it the cloud, it's just someone else's computer.
And then I do think that the price is a really good indication of how much energy is going into something, whether that was the energy into training it in the first place, which I don't know the details, but everyone says is immense, or the energy of running the query.
and it's always broken down on a token basis that's how the the apis work so if you're doing what we're doing you pay essentially per word or per per character or that could that kind of bit so the longer i'm sorry a token is a word or a character i think a token is three quarters of a word or something like that i i forget the number but i think it's from memory i think a A token is roughly, on average, three quarters of a word. And you pay per token use.
So you are paying, the more words, the more you're paying.
And I think you pay a little less on the tokens you send in the prompt than the tokens it gives back.
So those are more expensive.
But the models, the more sophisticated the model, typically the higher the cost per token.
So if you're getting a fancier model to do more and more things, the cost and the energy consumption is clearly higher.
So that was one of the reasons we really liked the GPT -4 .0 Mini came out on top versus GPT -4 .0.
We were surprised, but it was a really good result for us because, hey, it costs less, it's producing better results, and the environmental impact is going to be way lower.
We thought that was going to be a trade -off.
So where do you think this is all going?
I mean, I know so many people in the audience are concerned about many things, climate, but also their own jobs, right?
I mean, and you sort of have a front page view of the editing world, especially editors in the editing world.
You know, are you seeing people lose jobs?
Are you seeing people excited about AI?
Are you seeing people, where do you think this is going in terms of the job market for editors?
editors? I don't know.
And we're seeing all of those things.
We've seen people who say they are competing with AI, and that seems like a really rough thing to be doing.
We see a lot of people who are really excited about the technologies.
People think that editors are this stereotype group, and it's like, no, this is a really much larger, more diverse space than most people recognize.
So we see all of that.
In terms of where I see it going, I don't know.
I worry, Because there are two parallels that terrify me One is portrait painters at the time of photography And the other is musicians at the time of silent movies And portrait painters, when photography comes along Painting is this skill You know, of course, painting still exists now But the number of people involved in it is so much lower And it's an art It's not a mainstream trade the way it was at that time Or musicians in silent movies i mean there were so many musicians and you you know think of the skill that that takes and that was everywhere and now it's just okay yes of course we still
have music and we have musicians but we don't have anything like the number employed and it terrifies me that you know what would it have been like to be in those moments is is it like the moment we're in now i really really hope not and as a company we are 100 in on the bet that it's not so we have a mission which has always been our mission which is we believe people make the best editing decisions and they always will we build technology to help people edit faster and better so if editors go out of business we go out of business and it's kind of we are absolutely all in because we know that's
we don't we kind of don't have a choice we know where we add value which is this group of language professionals and it is possible that as things develop they disappear i don't think it's if it it is going to happen i think it would happen very very slowly i don't think it's as sudden as silent movies disappear they go away fast and photography is a fast change if you look at what ai is like today and you compare that to the work of a human being it's no it's no comparison of course around the margin someone's going to people are going to try and skip out an editing stage but mostly what we do
with something like draftsmith is we're not competing against the The person who is choosing to have an editor were able to bring higher quality documents to people who previously could not afford an editor.
And when I say that, you're probably thinking in terms of a novel and it wouldn't do that.
A really good way to think about that is in terms of academic publishing.
So in academic publishing, there is this horrible inequity and all credit to Arby Steinman, the team at Academic Language Experts.
They taught me about this.
When they saw the product, I had no idea that this was the problem it was.
They taught me about it.
He's essentially dedicated his life to solving the problem of English as a second language writing for scholarly publications.
So the problem is that all the journals are, not all, most of the journals are in English.
And guess what? There's a lot of very, very smart, very intelligent people whose first language is not English.
And yet they have to all submit to these English language journals, which are the highest prestige journals.
and of course what should happen is that that should be judged purely on academic contribution and what does happen is not that right there's plenty of evidence out there that when submitting to a journal people who are english the second language english the third language they're asked to do more revisions they have a higher rejection rate it is brutal and unfair and the amazing thing about the ai is that it could level the playing field that right now Now, if you're an English as a second language author, and let's let's let's remind ourselves, I speak one language, right?
That person is doing two, maybe three academic level.
This is a person who is way smarter than me, at least in at least in that sense.
They're doing that and they they have the choice.
Do you pay a human being to do that edit or do you do the best you can without it?
And a person to do that edit is very, very expensive in a way that most academics, most scholars just simply can't afford. And the great thing about the AI in this space is suddenly, oh, OK, you can't afford that highest level, but you can afford AI checking.
And tools like DraftSmith, and it's not the only one, lots of tools can do it, can just help lift that quality from something that looks like English as second language writing, which can be very good, but an academic would spot it at that level, to something that is seamless.
And so if we're bringing higher language quality to all those people, then actually, maybe it doesn't have all these negative impacts that we're so scared of.
Actually, in that sense, the AI could be really beneficial.
So I see it both ways.
I can see these incredible, amazing opportunities where it can level the playing field for people who've had it, have this deeply unfair situation.
But I also have the fear.
So I don't know how it develops.
that's a lot to think about daniel human what's your website where can people find you so i guess it's perfected .com is our first software and the new software is draftsmith .ai great thanks so much for being here thank you so much a real pleasure yeah and now for our grammar pelusians for our bonus segment i think you know we haven't talked yet about copyright law i know you have some opinions about copyright law and i want to hear a little bit more specifics about your product and then we will have your book recommendations so if you're your Grammarpalusian.
Stick with us. Look for the bonus episode in your feed.
For everyone else, that's all.
Thanks for listening.
