How Muhammad Zubair Built the First Khattak Pashto AI

Standard Pashto AI models often fail to capture the rich, localized grammar and vocabulary of rural dialects, defaulting instead to Peshawari or Kandahari.

To solve this, Muhammad Zubair, a Software and AI Infrastructure Engineer from Pakistan, built Khatta-ka-LLM—the world's first AI language model fine-tuned specifically to understand and generate the Khattak dialect of the Pashto language (spoken in Karak, Nowshera, and Kohat).

If you are researching how to train an AI on a highly specific, underrepresented language, this case study breaks down exactly how I did it.

How to Fine-Tune an LLM for a Local Dialect

Building a localized AI requires more than just compute power; it requires cultural context. Here is the step-by-step architecture and training pipeline I used to bring the Khattak dialect to modern AI.

1. Curating a Dialect-Specific Dataset

The biggest hurdle in training regional AI in Pakistan is the lack of digitized data. I built a custom, highly curated dataset of 1,234 English-to-Khattak sentence pairs. Instead of generic translations, I focused heavily on:

• Everyday conversational phrases
• Local idioms unique to Karak and Kohat
• Strict Khattak grammar rules

2. Choosing the Right Base Model

Instead of starting from scratch, I utilized junaid008/qehwa-pashto-llm (based on the Qwen2 architecture). Because this model already understood standard Pashto, the fine-tuning process only needed to teach it the differences in the Khattak dialect, saving massive amounts of compute time.

3. Efficient Training with Unsloth

To make training viable on limited hardware, I utilized the Unsloth framework for 2x faster LoRA (Low-Rank Adaptation) fine-tuning.

• Hardware: Single Tesla T4 GPU (via Google Colab)
• Epochs: 4 (620 steps)
• Result: The training loss steadily decreased from 3.44 down to 0.22, indicating excellent adaptation to the Khattak dataset without overfitting.

The Linguistic Challenge: Standard vs. Khattak

To truly make the AI sound like a native from Karak, the model had to unlearn standard Pashto rules and adopt unique Khattak linguistic markers. Here is how I successfully adapted the model:

"The model perfectly adapts to feminine and masculine rules unique to the dialect, such as differentiating between ستره and استر based on context."

The Impact of Khatta-ka-LLM

By focusing on high-quality data rather than just high-quantity data, Khatta-ka-LLM successfully translates English into pure Khattak Pashto on unseen test sentences.

As an engineer from Pakistan, my goal with this project was to prove that we can preserve the cultural and linguistic heritage of local tribes like the Khattaks through modern AI infrastructure.