Rubber duck editing with LLMs
Here are some prompts I'm using as I experiment with LLMs1 to see if they can help refine a draft of a talk I'm writing.
M: Here is a draft of a talk I wrote:
<draft>
It feels a bit scattered and disorganized to me. Please help me clean it up.
LLM: You've got a compelling narrative and some valuable insights here. However, it could be more tightly organized and concise for better impact. Here’s a revised and more structured version of your talk:
<clip>
M: Rather than rewriting it, can you just identify some flaws with the existing piece and recommendations for how to improve it? I don't want to reduce the length much, but I am happy to cut some pieces and add different lines of thought.
LLM: Certainly! Here are some observations and recommendations for how to refine your existing piece:
### Introduction
**Flaw:**
- Initial repetition
- Possibly too personal and less engaging for the audience who may not relate directly to your journey.
**Recommendation:**
- Streamline the introduction and reduce redundancy.
- Kickstart with a slightly broader hook that can capture the audience's interest more universally before diving into your personal connection.<clip>
M: Is the order of the talk good, or should it be rearranged?
LLM: The structure of your talk is generally logical, but it could benefit from some reordering to enhance clarity and impact. Here’s a suggested structure:
<clip>
M: Should I remove the discussion of [topic]?
LLM: The discussion about [topic] in your talk serves several purposes: it establishes your credibility, connects to your audience through a relatable experience, and sets the stage for your interest in research and documentation. However, the details can be streamlined to serve these purposes more efficiently. Here are some considerations:
<clip>
M: Thank you. Can you rewrite the talk to incorporate this feedback? Maintain approximately the same length, and do not change the voice or tone. If there are places where you think new ideas need to be incorporated, feel free to mark them with [TKTK] and a brief summary of what should be added if you are not confident you can match the tone or message.
Unsurprisingly, I still found the LLM was not at all helpful when it was trying to write or rewrite the talk. It tends to slip in to that very inauthentic “LLM voice” I so dislike, and the output is inevitably bland, uninteresting, and... well, ChatGPT-sounding. I didn't really expect this to be much good, and it wasn't, but I like to try it once in a while just to see if it spits out anything interesting.
While much of this comes down to the limitations of LLMs, I also think that most writers2 just aren't going to like any kind of writing that substantially isn't their own, especially when the topic is personal to them or where they are trying to express their own unique ideas.
However, when I asked “Is the order of the talk good, or should it be rearranged?”, the LLM provided an outline that really helped me to clarify the natural structure that had formed in the draft of the talk I had already written. From there, I was able to tighten things up quite a bit, and by the end it really resolved for me my feeling that the talk was a little bit meandering.
I also liked the format of the replies when I asked it to identify flaws, and when I asked if I should remove a portion. The flaws/recommendations layout was very clear and helped me think more about the potential weaknesses it had identified (some of which I agreed with, and some of which I didn't).
Similarly, when I asked it if I should remove a portion of the talk, it responded with a list of reasons I should keep it, and a list of potential upsides to removing it, before providing its suggestion. I ended up agreeing that it was worth keeping, though that was certainly a portion of the talk that needed some polish.
Altogether, the exercise reinforced my previous opinion: LLMs are terrible writers, but decent editorial rubber ducks.