Parse or stream
Choose a format-specific parser and the consumption model your workload needs.
lokit.parse · lokit.streamThe Most Modern Localization Framework
The highest performance and most developer friendly localization framework written in Rust.
Cutting edge parsers, streamers, and interchange
handling for all major filetypes and a first class
type system.
• Free • Open Source • MIT License • Batteries Included
Format coverage
Every major format and localization interchange included
Each filetype has its own module imported from the top level. All of these work the same, convert to the same data structure and are extracted or regenerated the same way.
Application
Tabular
Documents
One predictable workflow
All modules work the same; filetype specific modules at the edges with a unified core
Work in one model
Inspect, transform, match, split, or store the same Data units regardless of origin.
BaseStructure · DataWrite or regenerate
Create a canonical output or preserve the original package and vendor envelope.
document.export · document.regenShared structure
All filetypes,
one data structure
All filetypes are unified in Lokit by one data structure allowing your same code that manages strings an XLIFF file to work on a DOCX file. Once the files are streamed or materialized, the same typed interface applies.
Working with one common model means less code, type safety, easy extraction and handling of segments, tags and metadata.
model.py
from lokit.types import BaseStructure, Data, TargetData
document = BaseStructure( source_locale="en-US", target_locale="fr-FR", data={ "checkout.submit": Data( source="Place order", target="Passer la commande", targets={ "fr-FR": TargetData( text="Passer la commande", ), }, extensions={"component": "checkout_button"}, ), },)Typed target shapes
Single-target and multilingual documents are documented as distinct shapes.
Stable downstream code
Matching and database APIs consume the same model the parsers produce.
Lokit is a full localization engineering and backend framework
Lokit includes everything needed for backend localization engineering
Strictly typed end to end
A py.typed package and one explicit model for units, targets, plurals, context, metadata, and inline content.
Materialized or bounded
Choose random-access documents, one-shot streams, parallel TMX, or async iterators without changing the unit shape.
Write new or regenerate
Create canonical output, or update translations inside an original envelope when vendor-specific content must survive.
Translation memory included
Load the same model into PostgreSQL with COPY staging and run exact, contextual, tag-aware, or trigram matching.
Performance
Unmatched performance
Lokit is build fundamentally different to other Localization frameworks using a fully async data streaming backend written in Rust.
TMX Materialization
Tested on materializing 50,000 segments from TMX
Median / Units per second / Cold RSS
| Framework | Runtime | Median time | Units / second | Peak cold RSS |
|---|---|---|---|---|
Lokit | Rust / Python | 0.240s | 208,143 | 80.4 MiB |
Okapi Framework | Java | 0.263s | 189,437 | 269.2 MiB |
translate-toolkit | Python | 0.652s | 76,586 | 135.7 MiB |
Package map
Top level import with extended modules
Each module guide shows the canonical API modules imported from the main lokit package.
Parse
lokit.parseMaterialize supported inputs into BaseStructure.
Stream
lokit.streamConsume large files through bounded iterators.
Export
document.exportWrite the common model to every supported output.
Regenerate
document.regenProduce localized files shaped like their sources.
Query
lokit.LokitNavigate, filter, fuzzy-find, and score matches.
Store
lokit.databaseBuild a PostgreSQL-backed translation memory.
Type
lokit.typesImport the stable public model and enums.
Explore the package
Convert massive, complex TMX/XLIFF files in one line of code
Lokit is in a late stage beta. There should be no more breaking changes before V1.
But it's still best to pin the current version.