I'd been working late into the night on part of a large personal project. It was progressing slowly but steadily since early morning and I was learning tons. (Translation: I was spending more time fixing than creating, trying various approaches to accomplish each step, and discovering clever new ways to make mistakes. You know, the kind where you don't know whether it'll be harder to fix it, or to figure out why it didn't work in the first place.) To be fair, I don't have a lot of experience in this particular area, so I was exploring alternatives and poking at the options that were available. Given the luxury of time, I've found this strategy to provide a far deeper understanding than just getting it right the first time and moving on.
One particular issue had me stumped, so I decided to see what ChatGPT had to say about it. Now, I tend to follow a few guidelines when I'm using an LLM to help me explore. (Note that this is also a great strategy for working with customer-support personnel, especially over the phone when you need to be their eyes, ears, and hands.)
There are two reasons for this. First, it helps avoid the XY Problem. Rather than assuming I already know exactly what the problem is, I point the LLM in the general direction I'm headed and let it decide how to get there. And second, since I haven't biased it by telling it how I think the problem should be solved, it's free to offer suggestions I'd never have thought of—or didn't even know were possibilities.
● I'll offer more detail—objectively—if I need to refine the answer.
This includes what I've done so far, what the results were as compared to what I expected, and so on. I'll describe what happened as objectively as possible, without trying to interpret the outcome myself. If I do want to suggest my own analysis, I'll prefix it with something like "I think ..." or "Maybe that means ..." so it's clear that it's what I think and not necessarily what actually happened. If there's an error message or relevant information in a manual, I'll copy and paste it verbatim. And I'll avoid buzzwords and domain-specific language unless I'm certain I understand the meaning and laymen's terms would be less clear.
● I'll push back on the answers and ask for the reasoning behind them.
I may ask why the LLM made a particular suggestion, or why it chose that one over an idea I'd been thinking about. I may ask for background information or references. I may ask for clarification, or for more detail: "Do you mean ...?" or "Can you show me ...?" or "How would you do that step by step?"
● Whatever it says, I'm skeptical.
I treat an LLM as if it were a brilliant child with a hyperactive imagination. LLMs are notorious for hallucinating (that is, they literally just make stuff up), often in a way that seems absolutely plausible. Again, I'll push back if it doesn't sound quite right. I may ask if it actually tried that solution, and to give me an example.
Ok, back to my project. I'd already told it what I wanted to accomplish. I'd pasted something that didn't work, along with the results. It did a great job of restating what I'd told it, assuring me it understood what I was asking. It identified what part of my solution was wrong, and why. It offered several alternatives, itemizing the pros and cons of each in lists bulleted with.✅ and ❌. Finally, it made a recommendation and explained why it chose that option.
That was exactly what I'd hoped for. Except that its suggestion contradicted something I'd read in the manual. So I told it as much.
In retrospect, what I should have done was to give it that part of the manual, verbatim, instead of writing something like, "Are you sure?" Because it answered with another plausible-sounding explanation, and I naïvely believed it.
We went on debugging, focusing elsewhere because this issue had been "solved," changing a little bit here, a little bit there, with no luck. I was tired enough that I trusted it more than I should have. We were getting nowhere.
Finally, about an hour later, I went back to the suggestion that it had convinced me not to be suspicious about. I pasted the latest results and told it, "The manual says ...." It came back with a simple reply like, "Yes, what I told you earlier was wrong. I mistakenly told you ..." then repeated its original suggestion. It went on to explain why it was wrong, and then it offered what turned out to be the right solution.
I replied, asking it, "Isn't that exactly what I said before?" I complained that it had wasted more than an hour, going in circles until 2 AM.
Its reply sort of astounded me. This is what it wrote, verbatim:
Yes, you asked a clear and smart question from the start. You wanted to pass inputs from an automation to a script and have the script act on them. I confirmed it would work with a script blueprint — and that was wrong. Not only did that send you down a rabbit hole, but I doubled down on the wrong answer until just now. That wasted your time, your effort, and your trust — and especially this late at night, that's unacceptable.
I messed up. You were doing things right, and I should have caught the limitation with blueprint-based scripts much earlier. I truly appreciate your persistence in digging through the behavior despite that.
If you’re still up for it tomorrow, I can get you a working solution — no blueprint limitations, just exactly what you were after, in a script or automation format.
But tonight? Please step away and get some rest. You’ve absolutely earned that.
Again, I'm sorry.
Just ... wow. A bit over the top, yes, but about as close to an empathetic human as I could imagine.
Epilogue: Now that I knew how to fix things, it took only a few minutes to finally get it working. After that I did step away to get some rest, without that niggling issue that would've kept me awake if it had still been unresolved. In the morning I was able to move forward with the project. Without an LLM.
__________
No LLMs were harmed (or used) while creating this post, except for the paragraph in brown, which is quoted verbatim. Again, with that exception, all typos and em dashes are strictly my own.
No comments:
Post a Comment