Using LLMs as Glue in your apps
14 min read
AI is the new hot term that people are throwing the term on everything. But how can you best use this in your product? not just to sell your project.
Practicality
The most common and both easy to setup use for AI (LLMs) is to add a chat feature to your website and to attach a RAG to it. This is both an impressive and relatively obvious way to add some “AI” to your product or website; and you should definitely do it if it works for your use case, especially for help documents. There are a thousand videos, blogs, and paid tools out there on how to do just that, but I’m not here to beat a dead horse.
How can you use LLMs to actually improve your app in a meaningful and practical way?
It’s best to understand where LLMs can and should fit into any software, they help take fuzzy tasks and give them some level of solid predictability. LLMs can take human sounding language and actually make it do something!
Using the LLM as Glue
I have found in the products & projects I’ve worked on the best use cases were not in using LLM to “do everything” but to use them to stitch together processes that normally require a human to make a decision in the middle. Places that cant be solved with some simple if statements.
Use the LLMs as glue in your processes, to do fuzzy validation, to extract information from natural language, to summarize text, to add those finishing touches to an application to make the process “more human”.
Examples
Here are a few use cases that I think can give you some ideas to just streamline processes, or make things easier for everyone involved:
note: examples are generalized from my work experience and their exact nature has been changed
Travel Details Validation
As part of one of the wizard tools I worked on, a user needed to provide details needed for their travel so that it could be approved, booked and otherwise scheduled.
Previously this was kept to email, and that worked fine. Users would email what they needed and there would be some back and forth, but it would get done. This was inefficient, and lacked the tracking the company wanted for these expenses, the data would end up being keyed into other systems and information about the exchange would be lost.
Now, anyone who’s built software would tell you, this is a simple problem! it doesn’t need AI, just build a form! 10 or 20 fields and you’re good to go! And these people would be right, you can get the vast majority of use cases with a form that asks for who is going, the destination, a few dates, and has some check boxes for things like rental car needs or other usual items; and of course throw in a text box for anything “extra”. But like most things this only covers most cases.
And of course those that would be the exception to most cases are the ones paying the bills, the executive team, they would have trips with multiple legs, changes would need to be made mid travel, and other interestings things would need to be added to their trips. This ment the “extra” text box for these users was the majority of their travel requests. Now we could have build all the UI for these interesting and unusual requests, but this would have taken more work then the rest of the project combined and there is no way the costs would be approved, so they would have to just keep using the “extras” box, and any missing information and back and forth would need to go back to the old email system as before.
This isn’t really great if you’re trying to impress the executive team with your new travel system.
So how can we use an LLM as glue here? well they are great at taking the text in that fuzzy text box and confirming the user has actually put in all the information needed for that travel request before it goes to the people booking! and you can do it quickly with just a prompt (and about 5 cents in tokens).
You are a helpful travel assistant and you are in charge of making sure the you have everything you need to help arrange this user's travel.
# Your tasks
## Flights
- If they ask about a flight, please make sure they have indicated the following: time of day preference, the date they would like the flight, the departure and destination.
- Make sure they have indicated a seat type selection (window, aisle or no preference)
## Meals and Reservations
- If they ask for any reservations please make sure they have the number of guests and a preferred time indicated
- If they ask for food delivery, please make sure thy give how many people the food is for and what date and time it should arrive
... many more rules here
# Asking for feedback
- If you need the user to expand on any missing information, please ask in a courteous tone, and be specific about which items you need clarification on.
- Make sure you ask all of your questions at once, if there is more then one question put them in bullet points
If you have all the information you need or the user has indicated they have no special instructions call the travel_reviewed function
# The User's Request:
Name: {user_full_name}
{request}
also attached was a function call the LLM could call if it had no feedback for the user
With the above prompt we worked out about 50+ common mistakes as well as some items that would make the team’s job easier when booking / scheduling things. We were able to build this simple 1 extra step process into the UI in less then 3 days and were able to cut down emails about travel by 80%.
This did not require coming up with a complicated set of rules and dozens of UI elements to get the user to provide everything they needed, but just a text box with a followup “chat” style interface asking the user for clarifications and a 2nd text box. The original text, the AI response and the final clarifications would then be sent with the rest of the form.
Titling JIRA Tickets
For another process we had for an application, users could submit bug reports and feature requests. And as user’s do their titles were either “bug on X page” or would be very long, and additionally asking for both a title and a description of the issue was a bit more then user’s really wanted to do. So we came up with a quick solution that both helped our PM and PO triage these issues as well as improve the user experience. We would just make 2 quick LLM calls:
Summarize this ticket in 15 or fewer words so we can make a ticket tile. Only return the title and nothing else: {ticket_details}Summarize this ticket into 1-3 paragraphs to help us triage this ticket, only return the summary: {ticket_details}
We would call the first one for all tickets, and the 2nd for lengthy submissions. It’s hard to tell how much time this saves, but it only took an hour to implement and has definitely saved us at least that much time.
Booking Meetings
Lastly another interesting use case we had was helping user’s book meetings (as well as a couple other common tasks after a different meeting). Often our team would have a meeting with a client and there would be various tasks at the end of the meeting… these meeting notes would be contributed to by multiple people during the meeting and ultimately have a task list or 5 buried in there somewhere.
Now, this one doesn’t even need a UI if you wont want, you can just drop the notes into ChatGPT or your choice of LLM and ask it to extract a task list, but we wanted to go a little further. Our process generally was to add these “action items” to our global action items tracker so they could be re-assigned if needed and overall tracked.
So why not just add a text box to extract those items, and add them (with confirmation) to the action items list? Thats what we did! and we had the tool add back to the notes document what tasks it added.
Saving a Minute or 2
All of my examples above save 1 or 2 minutes here or there, but they are also super fast to build and deploy. Those couple minutes add up and more importantly they look impressive even if they didn’t take long to implement. Similar to what AI tools are doing with development with Cursor or Github Copilot, you should think about these tools not as replacing jobs or tasks but accelerating them, being a force multiplier.
A Word of Warning
As I’ve been rolling out and demoing AI tools I have found 2 big problems, that unfortunately I dont have a simple solution for; so keep them in mind as you build.
Trust
People seem to either trust the AI far too much or don’t trust it at all. You really do need to double check the AI’s work.
Scope Creep
Not unique to AI / LLM work, but people see it do 1 thing and want it to do ever increasingly difficult tasks. Anyone who has worked with these tools knows they lie or make mistakes far too often to be reliable for things that arent later reviewed by a human. People expect computers to be infallible.
As AI tools invade our everyday lives, and our everyday tools the computers are going to start becoming more and more “human”, and humans make a lot of mistakes.