I made a web app that enables all-to-all language communication between five different languages. This was a step function change from the previous setup (which used human translators who could translate from English to a given language) to a setup where a speaker in any of the five languages could be understood by any of the other languages. I’ll be sharing some of the things I learned during the process and some thoughts on how agentic coding changes software development and usage.
My journey to start building out this app started as most things do in my life: getting mad about something that seemed dumb.
In March, my local church congregation got together with a few other congregations for a conference (a “Stake Conference” if you know Latter-day Saint jargon). I was volunteered to help enable translation for the conference.
Our process for translating involved:
I was pretty frustrated, because it seemed like the technology should exist to eliminate all of this overhead. I had used large language models to translate between languages with success before, and I had also used voice-to-text within ChatGPT and Claude, and those seemed to pick up what I was saying pretty well. It seemed like all of the effort spent checking batteries, setting up zoom rooms, and coordinating languages might not have been needed. We were also spending some amount of money on buying new batteries for every conference and buying new transmitters to replace the ones that had broken, so it seemed like we could probably build a system that would substantially reduce the human effort needed from volunteers while also saving some cost.
I set up a web interface that uses the Gladia API (and an optional self-hosted whisper model for Haitian Creole) to provide language output for five different languages at once. The code is on my github.
As part of this, I also built a datset for and fine-tuned a whisper model for Haitian Creole. That is hosted on Hugging Face if you’re interested.
I think that enabling all-to-all translation was pretty cool. Our previous system only allowed people to translate from English to another language. This enforced restrictions on who could speak in meetings and how people could participate, which I think was harmful to unity in our church group.
This all-to-all translation functionality is only possible with recent advances in Artificial Intelligence; I don’t think there are more than 100 people on the earth who speak all of Mandarin, Portuguese, Spanish, English, and Haitian Creole, and I don’t think many of them would have been willing to come to our conference and do live translation.
As you might expect would be the case for a side project, I did not write that much code on my own here to build out this system. I also think that in the future, the amount of actual code written by humans will trend to zero for most applications. So, in lieu of talking about the coding work that I did on the project, I’ll share the things I did that will likely not be replaceable by AI in the near term.
I knew that any AI system would need to work with the human and technological infrastructure that we currently had. Our building has projector screens where we could share captions. I decided that if we could have a screen up front with all 4 languages needed, we could enable nearly everyone to understand what the speaker was saying with fairly minimal modification to the listening experience.
I also talked with other people who had done translation before, and they came up with some concerns that I addressed:
I modified my system to include speech-to-text so that people could hear audio coming out, and created a “listener” interface where people could see and stream audio coming out of sessions. I also added a profanity filter (turns out that profanity filtering is a lot more tricky in other languages than it is in English, but I added a blacklist of words that would likely never appear in a church talk).
Understanding the resources available and talking with people with past experience was a uniquely human job–nobody would have responded to emails or phone calls from an AI agent, and our organization didn’t have a centralized document with all of the infrastructure we have that might be relevant to translation.
As part of my investigation, I ran benchmarks on all of the languages that would be used in the meeting to determine whether the meeting would be feasible. I chose appropriate data to use for benchmarks (General Conference Talks, which are available online with matching text), wrote the prompt for the AI-as-a-judge, and verified that the benchmarks were successful by reading through outputs and analyzing whether they made sense.
I think that an LLM probably could have come up with benchmarks and run them, but I’m not confident that it would have made the right call at every decision point. There were a bunch of things in the benchmarking that needed a human, like:
I spent a good amount of time looking through webpages to see what was supported by different products. I found myself verifying rate limits for my plan, data egress related to my compute instance, language support by product, and parameter changes that I could use to make performance better.
To be fair, I feel like a coding agent would probably be theoretically able to do all of this on its own if I paid for the right tools. But, there were some product pages that were genuinely confusing (Assembly AI listed Haitian Creole in some places in their documentation but not in others). Also, there is something about giving an agent access to my passwords (and thus the ability to run up infinite API costs or delete my account) that makes me a bit nervous.
I spent a lot of time troubleshooting and testing the product before it was used, which I think was my biggest contribution to “building” the product. I noticed different issues (on mobile, certain buttons would disappear, transcription would hang for Haitian Creole local streaming, etc) that a coding agent would not have noticed on its own by just looking at the code.
I know that there are some AI systems for UI testing, but I did not use them in this project. Regardless, I think that there needs to be human user testing at some level to verify that systems work.
The Relief Society (the women’s organization in our church) was kind enough to let me try out my translation system during a regional conference with ~200 women in attendance. We recruited some women to help coordinate usage of the tool during breakout sessions, and provided training beforehand.
Unfortunately, we couldn’t anticipate how many women coming to the conference would need translation support, so it was hard to plan on exactly the scale of translation needs. In the end, there were about ten Spanish-speaking women, two Mandarin-speaking women, and two Portuguese-speaking women who were at the conference.
Translation happened after we broke into four breakout rooms, so I cannot provide feedback on how it went in all rooms. However, here is the experience for the room I was in:
I had provided people with the ability to join via a QR code on a piece of paper, which I thought would be easy enough, but in reality, this ended up taking 1-2 minutes per person (my assumption that all phones have working QR code scanning functionality was too heroic). In the end, we were not able to get all people in the room signed in before the breakout room moderators wanted to start. We ended up making the decision to split the room into two separate breakout rooms by languge in order to make the best use of everyone’s time.
So in short, the translation system was functional as I had intended it to work, but I did not consider a consider component in delivery (how to get people to actually join the system). As a result, I am not confident that the translation system actually provided any benefit to any attendees at the conference.
If I was doing this over again, here is what I would do differently:
This would mean sticking with my original idea for one TV screen with running subtitles, but I believe we would have had more success if we had changed the structure of the breakout rooms to match what my system could reliably provide.
I should have spent more time making sure that enough people leading breakout sessions were comfortable with the tool (and solving the problems that would come up in the “last mile” of providing the services to actual users). However, I’m also aware that doing this could result in a world with more time spent in training sessions than the counterfactual one with putting batteries into FM transmitters.
I have seen some people writing that they think SaaS is dead, or that it will be easy to build replacements for applications with high subscription fees. However, I don’t think that’s the case.
Yes, agentic coding tools did make it possible for me to build a cool translation tool on my own, but I don’t think that building your own alternative to an existing tool will make sense in most cases. I spent at least 10 hours testing and building out this system over the past few months, and if I value my time at the rate of minimum wage, it would make sense for me to pay $150 to a vendor to do this service (which probably only has a marginal cost to them of 3-4 dollars) instead of building it myself.
Agentic coding tools can satisfy user requirements, but they cannot easily identify the most intuitive ways for users to experience the tool. I believe that AI-enabled UI testing can verify that requirements are met, but I think it will be a while before AI-enabled tools can effectively understand what a user needs and deliver on those needs.
As agentic coding enables faster development, users need to provide feedback at a faster cadence to ensure the product is built in the right way. I think that a primary role of a “software developer” will be to ensure that feedback is coming in fast enough and integrated properly into agentic coding systems for the product to be built correctly.
I believe that as people get more comfortable with agentic coding tools, and as the remaining pain points around agentic coding disappear, we will be able to solve more real problems in people’s lives. I would love to see a world where people feel like they can participate fully in church meetings regardless of the language that they speak in. I think that as the cost of building software goes down, and as the capabilities of Artificial Intelligence continue to improve, people will be able to engage more deeply with the world and with their communities.