Open Source

Let’s say you’re feeling a bit under the weather. You’ve got a scratchy throat, a nose that resembles a leaky tap, and a cough that rivals a 2-stroke rickshaw without a silencer. You start by calling up your GP (general practitioner) who, with their broad training of common ailments, reassures you that it’s probably just a mild cold – nothing that a little rest, some tea, and binge-watching a few seasons of your favourite series can’t fix. ...

I think it was around 1999 when I first heard that Urdu is a low resource language. 25 years later Urdu is still considered low-resource despite having over 70 million native speakers. This is because large manually curated linguistic resources required for training are still not available for Urdu. One way around this barrier is to translate existing corpora from a resource-rich language (cough English cough) to Urdu. This seems like a chicken-and-egg problem though since good automatic translation systems require training resources. However, recent multimodal multitask models have reached a level where we can use their translations to set a baseline for more complex tasks like question answering. And we have done exactly that! ...

tl;dr I built a KWIC concordance of Iqbal’s Urdu Poetry - Check out iqbal.chaoticity.com ! In distributional semantics, just like in interpretation of law, a word is known by the company it keeps. Observing how a word appears in context of other words allows us to identify usage nuances and gain deeper insights into the meaning and connotations of that particular term. Traditional word indexes do not show this context and thus miss out on the most remarkable ability of the human brain – recognising patterns. Which is perhaps why, in the late 50s, Hans Peter Luhn (the father of hash codes!) came up with the idea of creating such an index for technical documents that allowed users to see keywords in their original context, facilitating efficient information retrieval. This innovative approach marked a significant development in the field of text processing and laid the foundation for subsequent advancements in concordance-based analysis. He called this concept Keyword in Context (KWIC – pronounced quick) and it caught on … quickly. ...