This should come in handy for text classification using WEKA (and hopefully save your time).

Is google (finally) stemming Urdu? The last time I checked, there were doing something like a transliteration based search but in the screenshot below, you can see that searching for the phrase ان پڑھ چٹا shows some stemming is being used. Does anyone know anything?  Oh, and while I’m on this topic, I would also like to know why is it called چٹا ان پڑھ ?

For techies: I’ve been working on citation networks lately. You can visualize such a network as a graph. In this graph, the nodes represent publications (papers,articles etc) and the edges represent citations between them. The graph above was produced using the GraphViz. The data is from the ACL Anthology Network which contains publications from the publicly available ACL Anthology.

For non-techies: Oooooo! pretty picture!

How are you?
آپ کیسی ہیں؟

(Thanks to عزت مآب جناب آغا علی رضا قزلباش رحمتہ اللہ علیہ who graciously sent me his term report on Hindi to Urdu transliteration, from where I’ve copied (and modified) the character mapping.)

It can be used in Urdu too,

we get the Hindi translation

यह उर्दू में इस्तेमाल किया जा सकता है

and the Roman transliteration of the Hindi translation

 yaha urdū mēṁ istēmāla kiyā jā sakatā hai.

If you notice the first word, it should have been transliterated to “yeh”. Instead, we get a phonetic transliteration which is made up of two letters ya and ha. Transliteration from Hindi to Urdu directly would have avoided that error. There’s a nice paper titled “Hindi to Urdu Conversion: Beyond Simple Transliteration”  which lists problems faced in simple character-to-character transliteration fromm Hindi to Urdu. Whenever I get some time, I’ll try to cook some javascript code quickly. Until then, the idea is open. Any takers?

mstsc /w:1280 /h:1024

The picture above is my office machine when i was trying a horizontal flip. It works when you have many consoles open but the bottom part gets for browsing/coding, it’s not that great.

Here’s how it works.

1. downloads all status information in a directory
2. extracts the status message lines
3. does some regex magic and filters stop words like the, a, an etc. ( haven’t  seen this done earlier anywhere but the join command comes in handy for processing stopwords)
4. displays the top 10 most frequent words (and emoticons)

Twitter assigns a limit to the number of messages that you can download (3200). Also, the twitter id timeline has to be public for this script to work. All you need to do is download the script file and stop word list, keep them in the same directory, run it with the twitter id in the command line and you’ll get the list of words with the frequency at the start of each line. For example,

$ ./tword.sh barackobama
161 watch
119 live
92 http://mybarackobamacom/livestream
81 health
63 reform
55 today
52 rally
48 #hc09
47 &
38 vote

The script takes time to complete so be patient. As you may have noticed, there are still html tags inside. You can remove them by piping in any html2text program. There’s a small perl script in the zipfile which does this processing. The output now brings in a new word “change”. You will, however, need to pipe this in the script after installing HTML::Entities though CPAN.

$ ./tword.sh barackobama
161 watch
119 live
92 http://mybarackobamacom/livestream
83 health
68 change
63 reform
55 today
55 rally
48 #hc09
39 vote

My list toppers as good, , time, day, twitter, read, hope, back, :p and make. I wonder if this makes me a happy person

A friend of mine is carrying out research in classroom based e-assessment in developing countries such as Pakistan. The aim of the research is to assist primary school teachers with computer software that:

· Is aligned with the particular subject curriculum they follow in their schools.

· Provides pupils with challenges and interactive short quizzes and tests to take after completing a topic taught by the teacher in the classroom.

· Provides students with immediate and diagnostic feedback on their performance on each challenge or test they attempt.

· Helps teachers in identifying the individual pupils needing help in certain conceptual areas of the curriculum, in managing the overall classroom portfolio, and in assuring better teaching and learning within the socio-cultural context of their educational system.

You can help in two ways!

1. The project is still in the stage where available literature on formative e-assessment is being critically reviewed and any bright ideas relevant to the topic are welcome.

2. One very important aspect of the research is related to the kind of tools and technologies that should be utilized in developing the software product. This is done keeping into consideration the fact that the use of expensive tools, technologies and infrastructure does not help much in sustaining any change in the educational systems of developing countries. Therefore, Moodle (www.moodle.org) and other such open source learning management systems are being considered for the initial version of this project.

We need someone who can evaluate both Moodle and OpenMark, tell whether the functionality of diagnostic assessment used in OpenMark can also be integrated in Moodle, and how. Monetary remuneration is available for this activity.

The Open University (www.open.ac.uk) in the UK is one of the pioneers in establishing distance learning (also e-learning) programs for higher education. They are currently using OpenMark (https://openmark.dev.java.net/), their own open source Computer Assisted Assessment (CAA) system, as well as Moodle to develop formative assessment tests for their students enrolled in distance learning programs (http://labspace.open.ac.uk/course/view.php?id=3484&topic=all). You will find some documentation on this at http://labspace.open.ac.uk/course/view.php?id=3484&topic=all, and at http://labspace.open.ac.uk/mod/resource/view.php?id=381989&direct=1. After that we would need help in taking on the development of our own software from there.

If you know anyone who is interested, please leave a comment or drop an email to awais {at} chaoticity.com

1. a good mic (recommended but not necessary)
2. a quite room
3. any software which can cut and save wav files. I use Praat.
4. the console version of Praat, called PraatCon. This will be used to extract the formants from wav files.
5. about 30 mins.

Once you have all these things, here is what you do for recording the phone sounds

1. Start Praat. You will see two windows. You can close the other window but keep the one titled Praat Objects open.
2. In the menu, select New/Record mono sound. Select sampling frequency to 16000 Hz, press Record and start speaking in the mic. When you are through, press Stop. Speak all the phones you want in a single go but with silence in between. In the screenshot below, AA (as in father) is spoken multiple times.
3. Press Save to list and then Close. You should now an entry in the Objects listbox. Make sure the entry is select and then press Edit.
4. Select the phone for which you want to extract the formants by pressing the mouse button on the start of the phone and dragging it to the point right before it finishes. Please note that we will be using the mean formant value for a phone so it is essential that only the middle part of the phone is selected to ensure accuracy and no other noise or silence is present. The selected area will be highlighted.
5. From the File menu, select Write selected sound to WAV file, enter the name of the wav file and save.
6. Repeat step 4 and 5 for all the phones you want to save.

That’s it for the recording part. Alternately, if you already have a collection of sound files with a single phone, you can use that. You can download one such collection here. The zip file has 40 files where 2 speakers have spoken AA (as in father) and II (as in been) 10 times each.

Click here to download script and sound files.

Although you can use the GUI based Praat to find out the formant values for each phone, we want to automate the task. To do so, we will use the console version of Praat called PraatCon, and run a script to extract the formants. The small Praat script written below takes the name of a phone file as argument and outputs the first two formats F1 and F2 to the console.

form Display mean F1 and F2
   sentence filename
endform
filename_noext$ = replace$ (filename$, ".wav", "", 0)

Read from file... 'filename$'
select Sound 'filename_noext$'
To Formant (burg)... 0 5 5500 0.025 50
f1 = Get mean... 1 0 0 Hertz
f2 = Get mean... 2 0 0 Hertz
clearinfo
print 'filename$' 'f1''tab$''f2''newline$'

This script along with a small batch file (run.bat) which runs this script for all wav files present in the current directory, can be downloaded with all the sound files by clicking on the link below. To get this running, extract all files in a directory and place your downloaded copy of PraatCon in the same directory. The just click on the run.bat icon and watch the magic!

Click here to download script and sound files.

The sound files in the archive give this nice little graph when their F1 and F2 values are plotted as an XY scatter graph shown below. All this was done to construct a linearly separable dataset which can be given to students for training a perceptron to distinguish between two phones. Any suggestion/comments are welcome.

Click here to download script and sound files.