In my last post, I highlighted some problems that I face daily while using twitter in Urdu as well in English. A few days ago, I decided to experiment with the Twitter API and write my own client to fix some of these problems. You can see the result at www.twingual.com. It is a javascript only twitter client which supports neat Nastaleeq urdu fonts as well as transliteration.

It’s a work in progress and does not implement all twitter features. If you like it and want to see something you need everyday implemented, feel free to send a tweet.

Meanwhile, tweet away!

Update: BBC has now embedded a new font (BBCNasim) on their website which is quite good. In fact I don’t use this plugin any more myself. It’s not Nastaleeq but it is good enough.

Let’s face it. The font on the BBC Urdu website is not that good. When a friend complained about it on our alumni list, I thought of writing a small greasemonkey script to take  care of the problem. The results are pretty good, as visible in the image below. The left part is the site after installing the Urdu Naskh Asia type font provided by BBC and before installing the script (and I maintain, Aijaz, it is not good). The right part is after installing the script. Click on the image and you’ll get an un-scaled version.

To install the script, click the link below and follow the installation instructions given there. Currently, it works only on Chrome and Firefox.

BBC Urdu Font Changer @ userscripts.org

and the world is a bit better now…

There aren’t many tools which allow you to visualise sentences parsed with dependency grammars. Here’s a small tool which generates a PNG of the dependency graph of a given sentence using the Stanford Parser. You can generate the image for Einey’s quote below by following these steps.

1. Click here to download DependenSee.jar.
2. Download the latest version of the Stanford Parser.  I am using version 2.0.1 (For older versions, drop me an email)
3. Extract stanford-parser.jar and stanford-parser-2012-03-09-models.jar in the same folder as DependenSee.jar.
4. On the command prompt, run
   java -cp DependenSee.jar;stanford-parser.jar;stanford-parser-2012-03-09-models.jar com.chaoticity.dependensee.Main "Example isn't another way to teach, it is the only way to teach." out.png
   (If you are on *nix, replace the semicolon by a colon and make sure you have Arial installed. If you have an already parsed dependency output file, replace the sentence by -t input.txt .)
5. Open out.png and admire :)

I have added Part-of-Speech tags and very basic edge overlap management and might add more eye candy later (curved/coloured edges ?). You can link the library in your code as well. An example is given below. Comments and queries are welcome.


import edu.stanford.nlp.trees.*;
import edu.stanford.nlp.parser.lexparser.*;
import com.chaoticity.dependensee.*;
import java.util.Collection;
class Test {
public static void main(String []args) throws Exception {
String text = "A quick brown fox jumped over the lazy dog.";
TreebankLanguagePack tlp = new PennTreebankLanguagePack();
GrammaticalStructureFactory gsf = tlp.grammaticalStructureFactory();
LexicalizedParser lp = LexicalizedParser.loadModel();
Tree tree = lp.apply(text);
GrammaticalStructure gs = gsf.newGrammaticalStructure(tree);
Collection tdl = gs.typedDependenciesCCprocessed(true);
Main.writeImage(tree,tdl, "image.png",3);
}
}


This ‘thing’ took about 30 minutes to figure out. According to the WEKA documentation, if  you add a new Instance to an existing Instances object, String values are not transferred ! In case you are working on copying a dataset with a string attribute, you need to transfer the string manually. The code segment below copies the i^th instance from source to dest where the first attribute (at index 0) is a string attribute.


dest.add(source.instance(i));
dest.instance(dest.numInstances()-1)
.setValue(0,source.instance(i).toString(0));


This should come in handy for text classification using WEKA (and hopefully save your time).

Is google (finally) stemming Urdu? The last time I checked, there were doing something like a transliteration based search but in the screenshot below, you can see that searching for the phrase ان پڑھ چٹا shows some stemming is being used. Does anyone know anything?  Oh, and while I’m on this topic, I would also like to know why is it called چٹا ان پڑھ ?

For techies: I’ve been working on citation networks lately. You can visualize such a network as a graph. In this graph, the nodes represent publications (papers,articles etc) and the edges represent citations between them. The graph above was produced using the GraphViz. The data is from the ACL Anthology Network which contains publications from the publicly available ACL Anthology.

For non-techies: Oooooo! pretty picture!

While all the online English to Urdu translators that I have seen don’t really work that well (read suck), if we make use the overlapping vocabulary and grammar of Hindi and Urdu along with using Google’s translation API, things come out pretty decent (as mentioned in my previous post). Here’s a small 15 min first cut script which just uses English to Hindi translation and then transliterates from Hindi to Urdu. Feel free to use the code and do ping me if you improve something. This works as a Hindi to Urdu transliterator as well.

How are you?
آپ کیسی ہیں؟

(Thanks to عزت مآب جناب آغا علی رضا قزلباش رحمتہ اللہ علیہ who graciously sent me his term report on Hindi to Urdu transliteration, from where I’ve copied (and modified) the character mapping.)

I am thinking of using google’s English to Hindi translation and hooking it to a Hindi to Urdu transliterator to get an approximate English to Urdu translation. The Hindi to English transliteration provided by google has some errors which might not be there if we convert directly to Urdu. For example, on translating the sentence

It can be used in Urdu too,

we get the Hindi translation

यह उर्दू में इस्तेमाल किया जा सकता है

and the Roman transliteration of the Hindi translation

 yaha urdū mēṁ istēmāla kiyā jā sakatā hai.

If you notice the first word, it should have been transliterated to “yeh”. Instead, we get a phonetic transliteration which is made up of two letters ya and ha. Transliteration from Hindi to Urdu directly would have avoided that error. There’s a nice paper titled “Hindi to Urdu Conversion: Beyond Simple Transliteration”  which lists problems faced in simple character-to-character transliteration fromm Hindi to Urdu. Whenever I get some time, I’ll try to cook some javascript code quickly. Until then, the idea is open. Any takers?

I have a 1920×1080 desktop at work but when I use remote desktop to connect to home, it automatically resizes to my compact 1024*768 desktop. Most programs don’t seem to have a problem but I was working on Weka KnowledgeFlow and one of my flows, originally designed on the higher resolution, never showed a horizontal scroll. It might just be a java thing. In short, I had to look for a method to remote using a higher resolution than that of the local machine. Luckily, you can specify a custom resolution for the RDC using a command line switch ( more here ). The command line below gave be enough space to fix the flow. I hope this helps someone out there.

mstsc /w:1280 /h:1024

The picture above is my office machine when i was trying a horizontal flip. It works when you have many consoles open but the bottom part gets for browsing/coding, it’s not that great.

There are a lot of web apps around which report your twitter stats. But at times, it’s better to do things yourself. I haven’t done any fun coding for ages now so last night, I finally got around to making a small program to gather twitter word statistics. The fun part was to do everything using  unix tools.  Here’s a small script file which displays the 10 most used words in the tweets for any twitter id.  I have only tested it under cygwin so this is probably the best place to say “USE AT YOUR OWN RISK”.

Here’s how it works.

1. downloads all status information in a directory
2. extracts the status message lines
3. does some regex magic and filters stop words like the, a, an etc. ( haven’t  seen this done earlier anywhere but the join command comes in handy for processing stopwords)
4. displays the top 10 most frequent words (and emoticons)

Twitter assigns a limit to the number of messages that you can download (3200). Also, the twitter id timeline has to be public for this script to work. All you need to do is download the script file and stop word list, keep them in the same directory, run it with the twitter id in the command line and you’ll get the list of words with the frequency at the start of each line. For example,

$ ./tword.sh barackobama
161 watch
119 live
92 http://mybarackobamacom/livestream
81 health
63 reform
55 today
52 rally
48 #hc09
47 &
38 vote

The script takes time to complete so be patient. As you may have noticed, there are still html tags inside. You can remove them by piping in any html2text program. There’s a small perl script in the zipfile which does this processing. The output now brings in a new word “change”. You will, however, need to pipe this in the script after installing HTML::Entities though CPAN.

$ ./tword.sh barackobama
161 watch
119 live
92 http://mybarackobamacom/livestream
83 health
68 change
63 reform
55 today
55 rally
48 #hc09
39 vote

My list toppers as good, :D, time, day, twitter, read, hope, back, :p and make. I wonder if this makes me a happy person :)