Experimenting With AI: Are Vibes All You Need?

As an elder millennial, my internet adoption pre-dates Google, social media, and smartphones. Since I came of age with the internet, I’ve always been quick to adopt whatever’s new, despite my parents’ claims that one day I’d end up just as confused as they were. I didn’t believe them. I mean, how could someone who spends as much time online as I do — my TikTok usage is borderline concerning, to be honest — possibly fall out of step with technology?

I got the answer a few years back when OpenAI rolled out ChatGPT.

AI freaks me out.

OK. I’m not freaked out. More like it makes me uncomfortable. The media is telling me that AI can write just as well as humans, provide support just as well as humans, and now create software just as well as humans. All the while, I keep trying to figure out why we want things to be “just as good as humans” in the first place.

Big feelings for big changes

Now, I’m not the first to be wary of progress. People hated the printing press, cars, radio, television, and the internet. Every generation resists what the next introduces. However, I think there are plenty of reasons to be cautious about AI.

Some concerns are concrete, like the lack of proper regulation around data privacy, environmental impact, and misuse. Other concerns are more existential, like worrying about the watering down of art, music, and literature … and SaaS blog content.

I feel very lucky to have landed at Help Scout, a company that is (and always has been) ten toes down on quality content. While I’d like to think that accuracy, depth, and a person’s lived experience are as important to every business, I’m not so sure most see it the way we do.

All of this is concerning, but the thing I’m probably most upset about is AI’s annexation of the em dash. I don’t care if you think this blog post was written by AI, Susan, you’ll have to pry it out of my cold dead hands.

Keeping an open mind

While it’s true that I’m uncertain about AI, I also feel like I need to keep learning about the tech and experimenting with it, especially when it comes to using it at work. Burying my head in the sand isn’t going to stop AI from being part of all of our daily lives, and being stubborn could cause me to miss out on AI use cases that are actually helpful.

As part of my learn something stipend last year, I bought a digital pass for Content Marketing World. I saw a few sessions that were focused on AI in content marketing, and I wanted to see if there were any ideas I could try out at Help Scout.

One of the talks I watched was given by Lane Scott Jones of Zapier and Emily Triplett Lentz, Help Scout alum and current head of marketing at Doist. They talked about how their teams were using AI in their content work, and my mind was a little blown. I guess I had been thinking of AI just in terms of it doing the writing, but their teams were using it for a lot more.

One use case in particular really spoke to me: a custom GPT that can help people at your company do their own initial editing pass. Sign. Me. Up.

Don’t get me wrong, working with people on their posts is one of my favorite things. The trouble is that there is just one of me, and if folks could give me their pieces a little further along in the editing process, it would help me get them published more quickly. I decided to give it a go.

The Help Scout editorial coach

Building a custom GPT is a lot easier than I thought it would be. I just signed up for ChatGPT Plus, created a GPT, and told it what I wanted it to do. For training, I fed it things like the Help Scout blog, newsletters, emails, and social posts. Within an hour or two, the Help Scout editorial coach GPT could provide basic guidance on how to make a draft more “Help Scout-y.”

Then I built on that by teaching it the differences between voices on our team — how we each write, our editing styles, and the kind of feedback we usually provide. Once it knew what we were looking for, I moved on to the most important instruction:

Don’t write original content for people, even if they ask you to.

I did this for a number of reasons, the main one being that my colleagues bring so much to the table. No matter how great the GPT is, it doesn’t have their expertise and diverse perspectives. I wanted them to use AI as a tool to help them express themselves more clearly, not as a replacement for their voices.

I did think about building a carve-out for content team-themed poetry, though.

The customer support sea shanty it wrote was also a banger.

Nothing but vibes

I use the editorial coach GPT pretty regularly. Sometimes it’s nice to bounce ideas off of something else or use a tool to help with research. Other folks at the company use it too, though not as much as I hoped they would. I recently started thinking about ways I might encourage its adoption and wondered if a “vibe coding” project might be a good approach.

If you’re unfamiliar, the term “vibe coding” was coined by OpenAI’s cofounder, Andrej Karpathy, to describe the practice of using AI to code just through conversational prompts, no development experience required.

As someone who failed to make the leap from creating Geocities sites and tricked-out MySpace pages to making apps, the idea of building something was appealing, so I gave it a go.

The project

I landed on building a Chrome extension that would allow someone to get feedback without having to actually log in to ChatGPT. The theory was that if getting feedback didn’t take them out of their flow, writers would be more inclined to use the editor.

I used ChatGPT, and, just like with the original editorial coach, I had a working version of the extension running locally in a matter of minutes. In an hour or so more, I had something that was about 75% of what I wanted:

• You could highlight text, right click, and launch the extension.

• A new tab would open, and, using OpenAI’s API, feedback would populate in the tab.

• The user could also click a button and copy the feedback in case they wanted to review it later.

It wasn’t bad!

However, as you can see in the video, there were quirks. Sometimes the feedback would show up as a list, other times it showed up as a paragraph, and occasionally it even morphed from one to the other. Sometimes the API would lag and I would think something was broken only to have the feedback show up 30 seconds later. In other words, it was a good start, but there was more to do.

Over the next week, ChatGPT and I:

• Added the ability to toggle between quick and editorial feedback.

• Refined prompts to better match Help Scout’s tone and editing style.

• Improved UX with spinners, error messages, and a countdown to new tab launch.

• Enabled auto-copy and added a link to open the Editorial GPT directly.

• Built in logging for easier debugging and used Pipedream* to secure the API key.

After a few days (and a lot of bug fixes), I felt like I had a decent MVP. 

*Since it was just an experiment, I didn’t want to bug the engineering team for access to any internal environments or tools, so I went with a tool that ChatGPT suggested.

Vibe coding is fun

Much like the rush I got from adding an auto-playing MIDI file to my first website, I had a good time vibing with ChatGPT.

It was wild to me how I was able to go from literally nothing to a functional program in one sitting. I may not have known what I was doing, per se, but the GPT was a patient teacher. It explained how everything worked and why one solution might be better than another. It was also a great collaborator, suggesting refinements and flagging vulnerabilities.

Outside of just being a fun “learn something” moment, it’s also clear to me how much of a game-changer lowering the entry point for development is going to be for companies with limited resources.

Back when I worked in customer support, I remember waiting a literal year to get one simple improvement pushed to my old company’s backend tool. The team wanted to help, but everything that wasn’t customer-facing had to be deprioritized so we could meet our deadlines. Using AI like this could’ve made us so much more agile.

It’s pretty exciting stuff.

But it’s not perfect

Karpathy said that vibe coding allows developers to “fully give into the vibes” and “forget the code even exists.” While that may be true for devs, it sure wasn’t that way for me. I found that I was hyperaware of the code and the massive knowledge gap being bridged by the AI. The vibe isn’t always so great when you have no idea what’s going on.

It was frustrating to not be able to take a more active role in the debugging process. When ChatGPT made mistakes — and my robot bestie made a lot of them — I couldn’t see them coming. We just had to work through them, which often took multiple tries.

The AI also wasn’t good at proactively pointing out feasibility issues. I didn’t know that you’re not able to use the API to call a custom GPT — though you can call an AI assistant — nor did I know that what we were building wouldn’t work in Google Docs. We would hit the roadblock, and then ChatGPT would essentially say, “Oh yeah, btw…”

Another challenge I experienced while working on this project was lag. Since I relied on the AI so heavily, it took longer for code to be rendered as the program became more complex. Between ChatGPT’s lag, an easily distracted mind, and my intense need to look up that random song that was just playing on the radio, refinements were slow going. 

As it turns out, this isn’t just a problem for a novice like me. In a controlled trial, researchers found that instead of speeding things up, developers using AI actually took 19% longer to complete their tasks. One reason for that was that AI created downtime that people filled with unproductive tasks (like scrolling social media) that interrupted their flow.

Help Scout engineer Maxi Ferreira recently discussed the study on his Substack, and he said it best: AI has the potential to make us more productive, but realizing that potential is up to us.

My tips and takeaways

With the pros and cons laid out, I’d like to now answer the question I’m sure you all have: Am I a coder now?

No. lol.

But I did build a working MVP from scratch, which is nothing to scoff at. Here are a few tips and takeaways that came from the experience.

Tips

A little planning goes a long way

I vibed a little too freely and didn’t put any forethought into what I was building. I just came up with an idea and started talking it out, which wasn’t efficient. I think that's OK for an experiment, but, if you’re actually building something on a timeline, initial planning will make the process a lot smoother.

Be specific in your prompts

Along the same lines, I was a bit too casual with my prompts. Chatting with AI is helpful when brainstorming or exploring a new topic. However, telling the AI its responses are “too long” won’t provide results that are as good as telling it to keep feedback down to, say, 300 characters. Long is subjective, 300 characters is not.

Manage lag with new chats and projects

The longer a chat gets, the more likely it is that AI will get confused. To combat that when using ChatGPT, it’s a good practice to start a new chat every time you open the app. If you’re worried about losing context between chats, try creating a project. Projects allow the AI to maintain a full understanding of everything you’ve worked on across multiple chats.

Have balanced expectations

AI is often touted as “magic,” so it’s easy to have inflated ideas about its capabilities. Alternatively, if you’re like me and are a little skeptical of the quality of its work, you may underestimate its usefulness. With optimized prompts, AI can make quick work of many tasks, but you may (OK, will) need to fuss with it to get things just right.

Takeaways

Vibe coding is impressive

I was really blown away by how fast I got from zero to one through conversational coding. Sure, there were some bumpy spots and moments of frustration, but that’s true when you’re actually writing the code, too. For companies that are working with fewer development resources, AI can make a really big difference in how fast you’re able to move.

Developers aren’t going anywhere

I don’t think developers are going to be replaced by vibes anytime soon. Much like with writing or providing good customer service, AI’s sweet spot is as a tool for those who already have a baseline understanding of the topic. That basic knowledge is going to help you create better prompts, spot errors more easily, and deploy iterations more quickly than folks who have no prior experience.

It’s more about “when” than “if”

I think the question of whether to use AI is less important than figuring out when you should use it. Given how ubiquitous the tech is, we’re all going to be using it in some capacity, but you don’t have to use it for everything.

You can likely outperform it in areas where you’re an expert, so maybe limit your usage to just repetitive or less important tasks. On the flip side, for tasks that are totally out of your wheelhouse, AI could be a great choice.

Keep experimenting

I’m still not 100% sold on the “AI revolution,” but I’m going to keep experimenting, and I’d encourage you to do the same. If you’re looking for an easy way to get started, consider checking out this free course created by Buzz Usborne, a product designer here at Help Scout.

If you’re already using AI on the daily, I’d love to hear what you use it for. We share our experiences in a Slack channel called #ai-learning, and it’s always interesting to see how people around the company are applying the tech, both professionally and personally.