Episode Transcript
[00:00:00] When Language becomes Data how to measure Localization Impact By Yena Kolesnikova and Anastasia Urozhko Most companies want to measure localization impact, but the data rarely gives them a clean answer. Business indicators move for reasons driven by other teams such as user experience, UX updates, pricing, and market dynamics, and language's influence is absorbed into those broader changes.
[00:00:27] Localization becomes measurable only when each metric is tied to the specific behavior it is meant to explain. Imagine you have built a Russian marketplace app and are preparing to expand across the Commonwealth of Independent States. Kazakhstan and Uzbekistan are your first targets, both running on a mix of your own stock and local sellers. Kazakhstan's language environment is officially bilingual but uneven.
[00:00:50] Kazakh is the sole state language, while Russian holds a constitutionally protected role in official communication.
[00:00:56] Most consumer facing content, including labeling instructions and signage, is expected to include Kazakh. Uzbek is the only state language and labeling. Consumer information and advertising must be provided in Uzbek with foreign language text allowed only in parallel.
[00:01:13] Many companies localize only these critical areas and leave the rest of the interface in Russian. You launch in both markets with a Russian only interface.
[00:01:22] Acquisition looks strong, driven by early adopters and curiosity. Yet within a month, your core metrics fall below the Russian baseline. A student in Shymken grew up speaking Kazakh at home and uses Russian mostly for media. She can navigate a Russian user interface, but this requires deliberate effort, especially around prices, surcharges, and cancellation rules.
[00:01:45] In Uzbekistan, an older man speaks Russian but reads it slowly and treats it as risky when money is involved. If customer support responds only in Russian, friction Deepens.
[00:01:56] People avoid FAQs, send more tickets, misinterpret policies, and feel unheard when support cannot speak their language.
[00:02:03] What begins as minor effort becomes a practical and emotional barrier as tasks grow more complex.
[00:02:09] In analytics, this appears as an underperforming market. The first reliable signal is D1, D7 and D30 retention. Broken down by country and interface language, these metrics already exist in most analytics dashboards.
[00:02:24] In our marketplace example, D7 retention in Russia was about 80%.
[00:02:29] Kazakhstan with a Russian only interface hovered around 75% despite identical pricing inflows.
[00:02:37] After releasing the Kizzaq interface, D7 retention rose to 80% at 1 million active users. That five point lift brings back roughly 50,000 additional users in a single week. Churn shows the same dynamic from the opposite side. Retention shows who returns Churn shows who leaves when both shift immediately after localization, the underlying issue is comprehension, not product. Market fit conversion reveals where comprehension breaks in marketplaces. Seller onboarding is especially sensitive, registering a shop, accepting commission terms, linking payouts, and confirming identity. These steps are manageable in a familiar language when presented only in formal Russian Continuation collapses across our 200,000 monthly onboarding attempts in Uzbekistan, only 8 to 9% reached completion.
[00:03:30] After launching the Uzbek interface, completion rose to 11 to 12%. That 3.4-point lift added roughly 6,000 to 8,000 active shops each month. At about $5 in monthly monetization per shop, the language update produced 30,000 to $40,000 in monthly revenue. Once sellers are able to move through core actions, engagement shows how deeply they use the product.
[00:03:55] In the Russian only interface, sellers listed about 1.5 items per month and rarely used tools such as promoted listings, stock updates, bulk uploads, or analytics.
[00:04:05] After the Uzbek interface launched, sellers who switched listed around 1.7 items per month, which meant roughly 200,000 additional items each month at scale. The next signal is financial behavior, reflected in the average revenue earned per active user in marketplaces. ARPU rises when users understand financial decisions such as the cost of promoted listings, delivery surcharges, commission rules, refund conditions, and visibility tools. In a Russian only interface, these elements often appear in formal phrasing that users read slowly or treat with caution.
[00:04:41] In our case, ARPU in Kazakhstan hovered around $1.8 compared to Russia's 2.2 despite identical pricing.
[00:04:49] After releasing the Kazakh interface, adoption of boosts, add on services, and visibility tools increased and ARPU rose to 2.1. At 1 million monthly active users, that represents roughly $300,000 in additional monthly revenue. Search Engine Optimization, SEO, and App Store visibility follow the same pattern. A Russian only listing ranks well in Russia but loses relevance where users search in other languages.
[00:05:17] Before localization, the app barely appeared for core Uzbek search terms.
[00:05:22] After adding localized metadata, reworking keywords based on local search behavior, and updating screenshots to match the localized UI, organic traffic rose by about 30% within three months. We learned this the hard way when the Portuguese term modelogem was translated as modeling in our services catalog and then reused as a basis for other languages, where it was interpreted as 3D or fashion modeling rather than hairstyling.
[00:05:47] Our SEO in that category collapsed.
[00:05:50] Users searching for styling never found the service, and the product began appearing in irrelevant categories.
[00:05:56] These effects shape more than discoverability.
[00:05:58] When users cannot find or correctly identify your product, it influences how they perceive its relevance and legitimacy.
[00:06:06] Market share makes this visible in Kazakhstan and Uzbekistan. A Russian only interface positions the product as an import no matter how strong the features are. Local language competitors feel closer and more trustworthy. After launching the Kazakh interface, market share rose from 20% to 22% over several quarters. In a $50 million market, those 2 percentage points add roughly $1 million in annual revenue. With localization costs near $100,000, the return on investment becomes a matter of record rather than speculation.
[00:06:40] Localization does not replace pricing, logistics, or product strategy, but in emerging multilingual markets, it determines whether users will consider the product at all. When the interface speaks the user's language, the product finally gets a fair chance to compete, and business metrics reflect this through improvements in retention, onboarding, and revenue. But even when the numbers improve, they still cannot show how users experience the interface at the moment of decision.
[00:07:07] They do not reveal whether a price feels acceptable, whether a sentence sounds respectful or abrupt, or whether a confirmation screen inspires enough trust to enter a card number. UX research becomes a necessary complement to product data in these situations. It allows localization teams to observe how language is perceived before it becomes a percentage in a dashboard. Through structured tasks, interviews, and simple experiments, teams can explore questions that metrics alone cannot resolve, such as whether one phrasing feels more transparent than another, whether tone affects willingness to negotiate, and whether users interpret identical features differently depending on the interface language. A study we worked on explored whether interface language influenced how drivers evaluated price fairness. The product operated in both Russian and Kazakh across Kazakhstan, which made it a good place to observe how the same information might be interpreted through different linguistic frames. The study followed a survey based format. Drivers first chose the language that felt most natural and answered a few questions about their phone and app language. Then their responses guided which version of the interface they saw next.
[00:08:14] Monolingual drivers viewed the interface in their preferred language, while bilingual drivers were assigned to Russian or Kazakh at random.
[00:08:22] They then saw a localized ride request screen with distance, duration, and price, chose whether to accept, skip, or enter a new amount, and finally rated how fair the original figure felt on a fairness scale. Drivers using the Kazakh interface negotiated slightly less, and when they adjusted the price, their counteroffers were sometimes lower. Drivers using the Russian interface were more likely to raise their counteroffers or decline the initial price.
[00:08:48] Fairness ratings moved in the same direction, although only slightly. The differences were small but regular enough to suggest that language was one factor shaping how drivers evaluated the same numeric value during bidding. These findings suggest that interface language can shape how a pricing flow is interpreted, and once that influence becomes visible, it opens a wider set of questions for product and localization teams. If one language is associated with fewer negotiations or lower counteroffers the team may adjust how prices are framed, refine suggested bid ranges, or add clearer explanations of price formation. If another language is linked to more frequent counteroffers or rejections, it may require stronger anchoring, more explicit reassurance about fairness, or a more detailed copy about distance, demand, and availability.
[00:09:36] These hypotheses can be tested through familiar metrics such as acceptance rate, time to match, cancellation, and earnings per driver. When language begins to influence comfort with negotiation, it is reasonable to examine whether it also affects responses to refund rules, search explanations, safety prompts, or consent flows. UX research can surface early indications that language is shaping interpretation, and product analytics can measure whether revised copy, adjusted number formats, or added explanations change outcomes across language segments.
[00:10:09] The insight positions localization as a credible source of testable hypotheses and supports systematic experimentation, connecting people's perceptions of the interface with what dashboards eventually record.
[00:10:20] When a localization team lacks dedicated UX research, the effect of language becomes visible through other teams. Pain points during roadmap planning in the product division, we saw how heavily our support team was burdened by local markets. Repeated questions Most of those questions were answered only in a short FAQ written in English, which made it obvious that we needed a clearer, localized help center. I partnered with the UX writing team, including co author Anastasia Uroshko, to rebuild the Self Service Help Center.
[00:10:50] Together with support, legal and business development partners, we redesigned the Help Center's information architecture. The project set boundaries and highlighted risky phrasing in each market, surfacing the problems that generated the highest contact rate. We grouped countries and languages into Tier 1 and Tier 2 based on support load, how many of our active markets used each app language and the product's business share there. Spanish, for example, went into Tier one because it served users across Colombia, Peru, Chile, and Mexico, where we saw both high ticket volume and a large active user base.
[00:11:26] This let us release updates first in the languages with the greatest operational impact.
[00:11:31] Our first version used one shared structure across all services, which gave writers a stable framework with fewer duplicates, cleaner version control, and consistent terminology. We then checked this structure with real users through simple card sorting sessions, people grouped help questions into categories and named them. Their labels showed how users in each country mentally connect topics such as payments safety and legal issues.
[00:11:55] Where groupings differed, it became easier to see when a single global structure was enough and when local variations improved findability.
[00:12:03] Another way we improved search was by defining and localizing a glossary before writing the Help center content. The UX writing team created clear, consistent phrasing for key concepts such as service fees, disputes, and safety features, which kept entries concise and helped align markets. At the same time, we knew support content would keep changing as policies evolved and that many topics were repeated across categories such as driver or courier registration and restricted items for delivery. To keep everything maintainable, we stored articles in Google sheets so every team could review and comment in one place. We created custom segmentation rules so the translation management system could reliably tell headings, body text, and list items apart. No longer did it treat a single cell containing an entire article as one long segment or split content at full stops, which often cut sentences in the wrong places and could not handle headings without periods. That structure kept parsing consistently every time we updated and reuploaded content to the tms.
[00:13:06] Once the new structure and wording were in place, we tested several machine translation and large language model providers on a Help center content sample. The FAQ format turned out to be a good fit for mt. Entries contained enough context, segments were longer than typical UI strings, and the language was clear. With the new segmentation rules in enginemix, we could roll out updates across languages without restarting the process and cut translation costs for Help center updates by roughly half. When the localize Help center went live, the first visible impact was in payments.
[00:13:40] Previously, as Anastasia recalls, everything related to money was squeezed into a single generic article.
[00:13:47] After restructuring, that single payments entry became a cluster of short, focused articles. They explained what the service fee is when it is charged, how to top up the balance, and what happens to payments or refunds after cancellation.
[00:14:01] Once these localized entries went live, ticket volume on payment topics dropped by around 55% worldwide. The effect across the Help center was similar. We aimed for a 30% reduction in contact rate and approached 45%. Instead of agents resolved tickets, faster, escalations decreased, and many apparent product defects instead turned out to be linguistic friction. The Help center became a visible example of how design information architecture, consistent terminology, and a scalable MT post editing setup can translate directly into a lighter support load. Fixing support articles is the visible part of localization.
[00:14:39] You rewrite what confused users, localize it properly, and watched the numbers respond. What stayed with me was how many underlying issues shaped the experience long before any of this became measurable.
[00:14:51] Inconsistent terminology, tone drift, and unclear phrasing had been accumulating quietly for months, yet none of them appeared in dashboards without continuous monitoring. They were easy to miss, and we realized we needed a way to examine language with the same consistency that we apply to product performance.
[00:15:09] We turned to established models such as LISA qa, tas, dqf, mqm, and tqi, which all organized the review around error categories, severities, and weighting. The challenge was to adapt this structure to the realities of a ride hailing product and to the pace of a product team. We needed a model that reacted strongly to failures that mattered in this context and that stayed light enough to repeat on a predictable schedule. We built a localization quality index that uses these frameworks as a base and adapts them to our risk profile.
[00:15:40] In a ride hailing product, a mistranslated safety rule or an incomplete fair description affects users far more than a punctuation slip, so we weighted the criteria to reflect that difference.
[00:15:51] Accuracy, completeness, and clarity carry the greatest influence.
[00:15:56] Fluency and grammar follow, while punctuation, spelling, tone, terminology, and cultural appropriateness contribute at a lower but still meaningful level.
[00:16:06] The goal was to align each error type with the risk it creates for real users. The model follows the structure of tqi.
[00:16:14] Reviewers annotate errors by category and severity. Each issue receives points, and the total is normalized by word count and expressed as a score from 0 to 100. The main difference in our adaptation is how it handles high risk issues.
[00:16:28] We introduced a minimum quality baseline and added penalty rules for critical errors.
[00:16:33] If two or more critical issues appear in accuracy, completeness, clarity, terminology, or cultural appropriateness, the sample fails even when the average score appears strong.
[00:16:44] This approach stops high impact problems from being overlooked within a generally positive result.
[00:16:49] Before we could run the model regularly, we first needed a clear sampling method. Reviewing every translated string was neither practical nor useful, since it would turn the LQI into another heavyweight audit rather than a repeatable measurement.
[00:17:02] What mattered was capturing a reliable snapshot of the output. For the initial run, we calculated a representative sample from the full set of translated strings and selected it through random sampling. We then focused on weighting the criteria and testing the model on a pilot set across several languages. We compared the numeric results with expert judgment and adjusted the coefficients until they aligned. When linguists described a translation as good but uneven and the model returned an excellent score, we strengthened the relevant weights. And when clarity issues hurt comprehension but barely moved the number, we raised that weight as well. Once calibrated, we set the LQI to run on a monthly cycle. Each month's translations formed the new population for a representative random sample, which kept the workload manageable while still capturing quality shifts. Drops in specific criteria showed us what we needed to fix.
[00:17:54] Low tone scores led us to refine tone of voice guidelines, while terminology issues made us revise our glossary Focusing each cycle on the latest work made improvements easy to see and verify over time. When the calibration work was complete, we integrated the model into the crowdonlqa Add on reviewers annotate issues where they already work, and the plugin calculates error points automatically.
[00:18:18] We export the totals, apply the weights, normalize by word count, and compute the score without additional tools.
[00:18:26] With a regular schedule and a consistent sample size, these runs form a compact dashboard that shows how quality shifts over time and which categories drive those changes. The value of this index lies in how it is tuned, calibrated, and embedded into the workflow to support our product decisions. Taken together, these examples changed how we think about localization.
[00:18:46] Localization became a set of hypotheses that could be tested with the same care we apply to pricing experiments or funnel optimization.
[00:18:53] When localization teams are equipped with UX research tools, aligned quality models, and a few well chosen metrics, they can show how, where, and for whom words matter. That is the moment when language stops being an afterthought and becomes a visible part of product strategy. This article was written by Yana Kolesnikova. She is a localization manager at Yango with previous experience at Indrive and Kaspersky Lab. She specializes in building localization workflows, integrating machine translation and driving LQA processes, and enjoys mentoring through Women in Localization.
[00:19:27] And Anastasia Urozko. She is a senior content designer at QIC Digital Hub and a UX writer with over five years of experience.
[00:19:36] She specializes in crafting clear, intuitive interfaces and integrating content strategy into product workflows. Originally published in Multilingual Magazine. Edu2 52 June 2026.
