Google Sorry Page…

This is interesting. Google thinks that I am a bot . I am getting the following page no matter what I search…

Gmail and igoogle seem to be working fine… however, the search through igoogle’s page is failing too.

Political Compass Test

On appu’s recommendation I took the political compass test too. Here is where I stand:

Economic Left/Right: -1.38
Social Libertarian/Authoritarian: -3.59

I too found the test to be fairly comprehensive… however there were still a few questions which I either didn’t get or genuinely had an “it depends” answer. However, still a lot better than the majority of the polls I have seen.

Also, I thought I was more right than left… may be my definition of right is too different from what these guys think as right.

RSS ugliness

I find it ironical to post a criticism of RSS through a medium where RSS is so dominant itself but here it goes anyways

RSS is mainly used for communicating news (more often breaking news than editorials). Hence, timeliness of getting a news item through RSS is important. The mechanism by which RSS works is pretty simple itself. An RSS feed is a simple text file served by a web server. News readers keep ‘polling’ the subscribed feeds at regular time intervals to get updates. In general, news readers set their polling rate at highly aggressive levels to ensure timeliness of news delivery.

Many sites complain that RSS feeds on their sites get ‘polled’ at such high rates that the amount of bandwidth they consume is significant compared to the traffic to their main website. This is obviously not a good trend as bandwidth is scarce and expensive. Also, most of the transfers are just for the news readers to check if there was a news update or not, so its quite likely that a significant fraction of RSS hits aren’t even viewed. Some analysis on this can be found at Robert Scoble’s blog and Internetnews. The general workaround against the problem is to limit the size of RSS items or use an RSS distribution service.

It is amusing to note that RSS appeared and became successful at a time when well engineered and time tested alternatives already existed. The most prominent is e-mail. Its well tested, robust and most importantly a notification medium by design. Instead of messy polling, it is based on one time reliable delivery of the message directly to the interested party (I am talking about the SMTP protocol for mail delivery). There is no scarcity of email clients on all kinds of platforms and they have umpteen number of ways to automatically filter and organize one’s mail. Email has also evolved significantly over the years. Problems like duplicate delivery and authentication have been taken care of. Its sad to see the powerful and efficient system of email newsletters being abandoned in favor of RSS feeds.

Please note that the IMAP protocol for accessing is mail is a client protocol and does use polling. However, it sensibly restricts polling to a very small transaction with the client asking the server for a count of unread messages. This is way better than fetching the whole feed again as done by RSS. Having said that, it is known that IMAP can be pretty resource hogging itself. It is very much possible for a thin client interface for mail (webmail, remote login) to actually be more efficient than IMAP itself.

RSS is evolving too. There are services which allow you to cache your website’s feed on their servers. The service takes over the subsequent distribution load. This reminds me a lot about how DNS works. DNS is probably one of the most heavily used, scalable and design-wise underestimated protocols on the Internet. It also works by having (a big number of) distributed caches between authoritative content (in case of DNS, the host-name to IP address mappings; the feed content in RSS’ case). However, there are enough differences between the two to point out the problems with RSS’ general design. DNS data is reused (a lot). Hence caching it at a local server makes sense. Updates are infrequent and often non-critical making an elaborate delivery mechanism (like Email for instance) an overkill. DNS hence resolves to simple expiration of cache entries to refresh DNS data in the caches. In contrast, RSS data is discarded the moment it has been used hence polling and caching at a ‘blog service’ site makes little architectural sense other than offloading bandwidth usage. We already touched upon the timeliness and frequency of update aspects of RSS which is in stark contrast with DNS.

Overall, a step backwards in technology…

Pulseaudio + ALSA Configuration

Okay… if you have read my previous post on ALSA Configuration you will know that I am trying to solve my sound setup problem in linux. The basic problem can be summarized as:

• Use the sound cards at their native sample-rate to improve over their rather poor hardware re-samplers.
• Mix audio from multiple applications on the cards.
• Be able to easily designate audio cards on a per application basis at application launch time.

I had commented earlier that I decided against pulseaudio because of its lack of 32 bit signed integer audio support. I realized after some general use of my previous alsa configuration that even though some applications use 32 bit audio well, most other applications have problems. Typically mplayer and ut2004 (through aoss) couldn’t correctly figure out that the device was running at 32 bit precision and sent it 16 bit audio data. This caused both the applications to run audio at half the speed with the audio crackling all over. The fix was simple: change S32_LE in .asoundrc to S16_LE. However, that would be the end of any 32 bit audio on my system. I also realized that the only application which I used (heavily) with 32 bit audio was madplay (piped through aplay; madplay directly outputs to hardware alone). I grudgingly agreed to sacrifice that .

Pulseaudio configuration is pretty straight forward. I configured the /etc/pulse/default.pa for the two cards in the following way. I happily noticed that pulseaudio used a good resampler by default (speex-float-3 which has similar CPU usage as alsa’s plugin speexrate).

/etc/pulse/default.pa

 
#!/usr/bin/pulseaudio -nF
.nofail

### Load something into the sample cache
#load-sample-lazy x11-bell /usr/share/sounds/gtk-events/activate.wav
#load-sample-lazy pulse-hotplug /usr/share/sounds/startup3.wav
#load-sample-lazy pulse-coldplug /usr/share/sounds/startup3.wav
#load-sample-lazy pulse-access /usr/share/sounds/generic.wav

.fail

### Load audio drivers statically
load-module module-alsa-sink device=intelHDA sink_name=Spkr rate=192000
load-module module-alsa-sink device=iMic sink_name=HeadPh rate=48000

### Load several protocols
#load-module module-esound-protocol-unix
load-module module-native-protocol-unix

### Automatically restore the volume of playback streams
#load-module module-volume-restore

### Automatically restore the default sink/source when changed by the user during runtime
#load-module module-default-device-restore

### Automatically move streams to the default sink if the sink they are
### connected to dies, similar for sources
load-module module-rescue-streams

### Automatically suspend sinks/sources that become idle for too long
load-module module-suspend-on-idle

### Make some devices default
set-default-sink Spkr
#set-default-source input

To make alsa applications work directly with pulseaudio we just need to set the default pcm and ctl to the pulse alsa plugin (configuration shown later). The pulse plugin creates a new audio device called pulseaudio and maps its audio I/O to the default pulse sink which can be changed at runtime. The pulseaudio device also has two mixer controls; a “Master” for playback and a “Capture” for recording.

The problem with the above is that pulseaudio starts looking for mixer controls “Master” for every configured sink hardware device (with a fallback to “PCM”). The default alsa configuration doesn’t provide a “Master” control for either the intelHDA card or the iMic card. The “PCM” on iMic controls the hardware mixer directly and that for intelHDA controls a software volume for alsa’s default dmix configuration. Thus, the pulseaudio fallback to “PCM” works correctly for iMic (changing the pulseaudio “Master” control changes the volume) but not for intelHDA (changing pulseaudio “Master” control has no change on volume, but changing “Front” directly on the hardware using alsamixer -c does work). To work around this I configured two virtual softvol (software volume) devices for my two cards naming the softvol control “Master”. The configuration is shown below.

.asoundrc

pcm.!default {
        type pulse
}

ctl.!default {
        type pulse
}

# Create softvol devices for intelHDA and iMic.
# For intelHDA, its "PCM" control controls its default (not used) softvol.
# For iMic, its "PCM" control controls its hardware directly.
pcm.intelHDA {
        type softvol
        slave.pcm "hw:0,0"
        control.name "Master"
        control.card 0
}

pcm.iMic {
        type softvol
        slave.pcm "hw:1,0"
        control.name "Master"
        control.card 1
}

# It seems we require these dummies as pulseaudio tries to look up mixer devices.
ctl.intelHDA {
        type hw
        card 0
}

ctl.iMic {
        type hw
        card 1
}

# For UT2004 (which uses dsp0 through aoss), always go directly to headphone hardware.
pcm.dsp0 {
        type plug
        slave.pcm "hw:1,0"
}

ctl.dsp0 {
        type hw
        card 1
}

Note that the pcm names used (intelHDA and iMic) are used in /etc/pulse/default.pa as sinks. More information on the softvol alsa plugin can be found here. There is some nice documentation about the various alsa plugins here. There some good documentation about alsa is also present here, here and here.

To change the current default pulseaudio sink I rewrote the audio.sh from the previous post to the following:

audio.sh

#!/bin/bash

pulse_set_default_sink () {
        pacmd << EOF
set-default-sink $1
EOF
}

h=`cat $HOME/.dwm/status_audio`;

if [ "x$1" == "x" ]; then
        if [ "x$h" == "x[ Spkr ] " ]; then
                echo -n "[ HeadPh ] " > $HOME/.dwm/status_audio ;
                pulse_set_default_sink HeadPh ;
        else
                echo -n "[ Spkr ] " > $HOME/.dwm/status_audio ;
                pulse_set_default_sink Spkr ;
        fi
elif [ $1 == "up" ]; then
        amixer set Master 2%+ ;
elif [ $1 == "down" ]; then
        amixer set Master 5%- ;
elif [ $1 == "toggle" ]; then
        amixer set Master toggle ;
else
        echo "$0: Unknown command $1";
fi

v=`amixer get Master | grep "Front Left:" | egrep -o "[0-9]+%"`
m=`amixer get Master | grep "Front Left:" | egrep -o "off"`

if [ "x$m" == "x" ]; then
        echo -n "vol:$v "
else
        echo -n "vol:(M) "
fi > $HOME/.dwm/status_volume

Note that now environment based device settings are gone as the current default pulseaudio sink automatically maps to the global alsa default pcm device. Once an application connects to pulseaudio, it retains its sink mapping even after the default sink is changed. (Update: It seems that module-volume-restore remembers the volume and sink of a client and restores both when it sees the client again. This behavior is definitely useful but we need to disable module-volume-restore if we want our current default sink logic to work). Pulseaudio also support hot swapping an application’s audio from one sink to another. I am still thinking of a way to toggle the sink device of the currently focussed application using dwm shortcuts… that would be some nifty configuration .

ALSA configuration

I recently configured the sound setup on my home computer. The problem was as follows: I have two sound cards 1) intel HDA (inbuilt) 2) usb based iMic. (I needed two because I wanted to have both my headphones and audio system hooked up and ready to play audio without manually (un)plugging cables.) I wanted to be able to switch between the cards on a per application basis. Basically enough capability to be able to launch applications each with a playback device determined at launch time.

Also, both the soundcards are pretty cheap and hence (probably) lack good re-sampling engines on the card itself. Most of my music is in 44.1KHz and iMic probably re-samples them to 48KHz during playback. intelHDA supports sample rates as high as 192KHz and I think does a naive re-sampling to that rate too before playback. So besides simple device switching I also desired to use a decent resampler to convert to 48KHz or 192KHz before the signal is fed to the device.

And last (but definitely not the least) I needed some way to mix multiple streams without latency issues.

Pulseaudio fits the above requirements very well. It also supports per application volume controls and dynamically switching applications to different input/output devices. It packages good resampling algorithms and sports a lot of features through modules. However, it doesn’t support 32 bit integer audio (which intel HDA does). Once pulseaudio supports 32bit integer devices natively I plan to switch. Given the recent interest in pulseaudio by Ubuntu and Fedora projects, this should happen fast.

So I took the alsa route. The environment variables ALSA_CARD (or ALSA_PCM_CARD) can be set to the card number to switch the card being used. This gives us an easy mechanism to launch different applications with different audio devices.

Also, alsa by default configures a ‘dmix’ plugin to mix audio streams before sending it to the hardware. However, the default configuration (buried in /usr/share/alsa) puts dmix before the hardware device selection. dmix has to operate on data at a fixed sampling rate which by default is set to 48KHz. Moreover, to avoid lots of CPU usage, the re-sampling algorithm used is poor. I wanted to setup separate dmix plugins for each of my devices operating at their native frequencies and use good re-samplers for audio sample rate conversions. Fortunately, alsa has a set of plugins (package alsa-plugins in gentoo) which include good re-samplers. Note that with two dmix plugins setup, using ALSA_CARD to switch the hardware device will not (and should not) work. We need a mechanism to switch to the given dmix plugin instance using environment variables.

Without further ado, here is my ~/.asoundrc and audio.sh (which I use for launching audio applications / switching devices / manipulating volume).

~/.asoundrc

defaults.pcm.rate_converter "speexrate"
defaults.pcm.card_name "intelHD"
defaults.pcm.card 0

pcm.!default {
        type plug
        slave.pcm {
                @func getenv
                vars [
                        ALSA_CARD_NAME
                ]
                default {
                        @func refer
                        name defaults.pcm.card_name
                }
        }
}

ctl.!default {
        type hw
        card {
                @func getenv
                vars [
                        ALSA_CARD
                ]
                default {
                        @func refer
                        name defaults.pcm.card
                }
        }
}

pcm.dsp {
        type plug
        slave.pcm "intelHD"
}

ctl.dsp {
        type hw
        card 0
}

pcm.intelHD {
        type dmix
        ipc_key 1024
        slave {
                pcm "hw:0,0"
                format S32_LE
                rate 192000
        }
        bindings {
                0 0
                1 1
        }
}

ctl.intelHD {
        type hw
        card 0
}

pcm.iMic {
        type dmix
        ipc_key 1025
        slave {
                pcm "hw:1,0"
                format S16_LE
                rate 48000
        }
        bindings {
                0 0
                1 1
        }
}

ctl.iMic {
        type hw
        card 1
}

and audio.sh

#!/bin/bash

h=`cat $HOME/.dwm/status_audio`;
if [ "x$h" == "x[ Spkr ] " ]; then
        export ALSA_CARD_NAME=intelHD ;
        export ALSA_CARD=0 ;
        master=Front ;
else
        export ALSA_CARD_NAME=iMic ;
        export ALSA_CARD=1 ;
        master=PCM ;
fi

if [ "x$1" == "x" ]; then
        if [ "x$h" == "x[ Spkr ] " ]; then
                echo -n "[ HeadPh ] " > $HOME/.dwm/status_audio ;
                export ALSA_CARD_NAME=iMic ;
                export ALSA_CARD=1 ;
                master=PCM ;
        else
                echo -n "[ Spkr ] " > $HOME/.dwm/status_audio ;
                export ALSA_CARD_NAME=intelHD ;
                export ALSA_CARD=0 ;
                master=Front ;
        fi
elif [ $1 == "up" ]; then
        amixer set $master 5%+ ;
elif [ $1 == "down" ]; then
        amixer set $master 5%- ;
elif [ $1 == "toggle" ]; then
        amixer set $master toggle ;
else
        exec $1 ;
fi

v=`amixer get $master | grep "Front Left:" | egrep -o "[0-9]+%"`
m=`amixer get $master | grep "Front Left:" | egrep -o "off"`

if [ "x$m" == "x" ]; then
        echo -n "vol:$v "
else
        echo -n "vol:(M) "
fi > $HOME/.dwm/status_volume

Let’s go over .asoundrc first. BTW, just for the record, the alsa configuration format is one of the weirdest formats I have seen. It is not user friendly and it also lacks a good deal of flexibility. Anyways, alsa configuration is all about defining pcm (virtual devices for audio I/O) and ctl (mixer interfaces) virtual devices. A number of pcm virtual device plugins along with some details on setting them up is documented at alsa-project. I use the ‘hw’, ‘dmix’ and ‘plug’ plugins in the above configuration. The pcm.intelHD, ctl.intelHD, pcm.iMic and ctl.imic sections outline dmix configurations for each device. The ‘type dmix’ creates dmix plugin instances named ‘intelHD’ and ‘iMic’. Each requires unique (arbitrarily chosen) ipc_keys. The slave configuration is pretty much the hardware device at its highest possible capabilities in format and sample-rate (pcm “hw:n,m” uses the hw plugin to talk directly to the kernel device; BTW, we could have used any pcm virtual device here). I don’t know what bindings are, but I am guessing it binds channels between the slave pcm device and the pcm being configured. I use two channels and the above seems to work.

Note that the above creates dmix virtual pcm device instances with the same format and sample rates as the slave pcm. So intelHD dmix can take signed integer 32bit data at 192KHz and the iMic dmix can take signed integer 16bit data at 48KHz. My devices don’t support float 32bit data (which pulseaudio supports :/). If we directly use these pcm devices for audio playback our audio application will either need to do format conversion/resampling itself or fail with format/samplerate incompability kind of errors. Fortunately, alsa provides a ‘plug’ virtual pcm device (very weird name if you ask me) to do the format conversion and resampling automatically. It can also do all kinds of channel routing magic too but I didn’t use any such capabilities. I basically created the ‘default’ pcm device (used by all alsa applications in case no pcm device is explicitly stated) using plug. This frees applications to match my dmix (and hardware device ) format/samplerates. Moreover, the default pcm ‘plug’ virtual device uses the ’speexrate’ re-sampler for all resampling. The documentation at the url above says that plug supports a rate_converter argument for specifying the re-sampling engine to use. However, that didn’t work on my system. Fortunately changing the default resamler works. The speexrate resampler takes a little less than 20% CPU on a single core on my 2.4GHz Core2Duo. The CPU usage is reported as part of the audio application using the audio device. It would have been nicer on the CPU to be able to be able to create separate dmix instances for each sample rate in use by applications and then resample each of them to the hardware sample rate. But I guess that’s necessary only when one has a lot of simultaneous audio applications running and a lot of them at the same sample rate. Moreover, it will be a nightmare to configure that using the alsa configuration format :/.

The default pcm device looks up an environment variable ALSA_CARD_NAME to form its slave pcm device. I can give any pcm name here but the intent is to configure it to either “intelHD” or “iMic” to use the dmix pcms. Note the really weird syntax. I took me a few trials and inspiration from /usr/share/alsa/alsa.conf to get this right. I had to use two different environment variables for the defaut mixer (ctl) and default pcm. This is because I couldn’t figure how to use a slave with ctl.!default. (The !mark is for overriding a previous configured pcm).

This completes alsa configuration. Now let’s come to audio.sh. It looks up the status_audio file to check which audio device should be selected for the to be performed operation. If status_audio contains “[ Spkr ]” then intelHD is used, otherwise iMic is used. Its usage is as follows:

audio.sh <command>
runs the command with the selected audio device as the default pcm device. Basically, it sets up the the environment variables ALSA_CARD_NAME and ALSA_CARD and starts the command.

audio.sh up
Turns up volume by 2%. Note the usage of $master. The volume control that’s effective on the two devices are different. $master lets amixer use the appropriate control for each device.

audio.sh down
Turns down volume by 5%. (I use different values for up and down so that I can turn down very loud music really fast).

audio.sh
Selects the other device than the current. It writes to the status_audio file to preserve the setting.

audio.sh also writes the volume of the device being configured to status_volume. I use status_audio and status_volume for displaying audio status in dwm (which is a fantastic window manager, if you like a minimalistic and highly productive windowing environment).

“Illegal franchising” in India – 2/2

I hope you have read my first article on “Illegal franchising” in India. In this article, I will touch about another set of incidents which caught my attention. This is about a public sector company called Eastern Coalfields Ltd. where my father works. As you may have guessed, the company is all about coal and is situated in a very fertile coal belt in India. Our colliery has a relatively large production rate and churns out pretty high quality coal. However, because of a wide array of socio-political and economic reasons, ECL used to face a daily loss of Rs. 2 crores (I have no evidence or proof though). A big reason for this loss was because of inefficient workforce taking huge salaries. ECL has been able to recover from this situation using a policy of hiring freeze (again, I disclaim any knowledge of evidence). I am told that they are in profit now.

ECL coal mining sites are situated at quite some distance from major town centers. Hence, ECL provides their employees with a lot of benefits including housing, medical services and schooling for children. ECL sites are typically situated in remote areas where there is no dearth of land (or water and electricity, though their relative cost is minor). This gives opportunity to lots of non-ECL employees to settle here. Some of that is okay. You probably don’t want ECL to provide/control groceries, fresh produce, banking and transportation. In many cases, people in family relation with ECL employees (often living together), take up such jobs. Things become interesting when it comes to coal mining as an “alternate profession”.

The coal sites under the control of ECL are really very rich. In our colliery, there are places where manually digging the ground for some feet leads to coal. As you might imagine, given the hiring freeze, a large settlement of people and absence of law and order enforcement, a lot of people engage themselves in illegal mining. Its illegal because the coal is the property of the Govt. of India and these people obviousy have no license or contract with the Govt. for mining. If you visit our colliery in the evening when it gets dark, you will witness scores of piles of illegal coal dug manually and set on fire (Setting on fire burns the sulfur content in coal so that later it burns without smoke). The smoke is an obvious health hazard to all the residents in ECL but the Govt. has so far not cared to do anything about either the smoke or the coal stealing.

What makes matter even interesting is that the illegally mined coal is sold at a price lesser than what ECL charges for it. It might not lure heavy lifters who require large and consistent supplies. However, the small buyers love the competition and the price drop. The important thing to note is this phenomenon of a competitive, peaceful though illegal “franchising” of coal mining happening in the total absence of law and order enforcement. ECL obviously has more problems with efficiency to keep them occupied. However, like the railways, I see an important business opportunity missed.

“Illegal franchising” in India — (1/2)

In my two last visits to India, I have had an interesting insight which I would like to share with you. The following incident happened in Summer 2005 when I was coming to India. When coming to India I generally take a flight to New Delhi and take a train from there to my destination. There are a couple of fast trains from New Delhi towards Kolkata named Rajdhani Express which I generally prefer. They take about 2/3 the time most other trains do. However, these are few in number which leads to a ridiculous buildup of advance reservations. The seats fill up as early as one to one and half months before the date of the journey and as you can imagine, it’s really hard to program a trip that early.

I planned my trip to India in summer 2005 about a month in advance. I had little problem getting a good deal on the flights but I couldn’t get a confirmed ticket on Rajdhani for the appropriate day. I was able to secure a waiting list ticket. (A waiting list ticket automatically turns into a confirmed seat once a given number of confirmed tickets are cancelled.) I had a waiting list number of 1 which made me confident that given a month’s time it would be confirmed. I could not be more wrong.

Well, I cannot believe that there was not even one cancellation for an entire month. I was certain that there was something else going on. When I reached India I still didn’t have a confirmed ticket. I was beginning to worry. I had no place to stay, the train was in about four hours and I had no intention of travelling overnight without a confirmed seat. The magic of Indian economics begins now.

An old man walked up to me and said, “Do you need a ticket sir?”. I thought… what are the odds! Except when there are about a 100 of these guys probably coming up to people in all kinds of trouble and offering a solution. I told him that I needed a ticket for a train today. I thought he would laugh at me and say, that’s impossible and suggest a hotel to stay. (I think this might be what you might expect in the US or EU). This guy on the other hand took me confidently to an office on the other side of the road. I followed less out of hope and more out of curiosity.

Of course no service comes without money and this guy later said, “Saheb, my job is done. You can pay me whatever you think is appropriate.” I gave him Rs. 20 (think $3 in terms of cost of living). He seemed to be more than happy and he left. It might be possible that he also collected some money from the office he took me to. Okay, this office was a small room with a single window with a view of the railway station, was sparsely crowded and had an air-conditioner. Given the summer heat in New Delhi I would pay just to be in an air-conditioned room. This guy in the office was obviously selling railway tickets illegally but this was such a relief compared to the legal way of buying tickets in India (which is basically standing in a long queue in the heat for around half an hour and then presenting at the counter a form with specific details down to the train number to get a ticket. If you are missing a detail (say the train number) you are shoved out of the queue. Of course there is no easy way to find the train number of a given train, leave alone a sensible way to plan your entire journey which involves not only finding all the different trains you might need to switch but also finding seating availability in all of them). An air-conditioned office with an agent was such a relief…

I told him that I had a waiting list ticket for the Rajdhani with me which I needed to be somehow confirmed. I told him that this was for a train leaving in less than 4 hours. I thought he would be surprised at my audacity and expectation… but he continued very normally: “We will charge 30% extra over the ticket’s cost.”
I was surprised: “You mean you can get me a confirmed ticket on today’s Rajdhani?”
“Most certainly Sir. If you come after around twenty minutes I will have the ticket ready for you. I will need some advance payment though.”
“Sure.” I paid him a little and left still very surprised by what he claimed he could do. I went outside for some lassi (refreshing drink made using blended yogurt and rose essence). I thought about the number of hoops this agent would have to go through to get me the ticket. He would need to bribe people in the ticket counter to make sure he kept some seats reserved for premium sale such as mine. The person at the ticket counter has to make sure that those seats are reserved even though he doesn’t know the name or the age of the people who would eventually buy it. Only god knows how he does that… I guess some more money exchange is involved here too. Also, he is not the only guy selling tickets illegally. He has competition. So he needs people at various places in the station to see that his assets are not under risk. He maintains an office at a place which people would pay a lot of money for. He also has air-conditioning! And he doesn’t seem to have a crowd following in his office. Suddenly the 30% extra seemed too paltry… may be he is competing on these rates too. Hmmm… this seems like an already mature market running smoothly on the fundamentals on capitalism very strangely in the absence of law and order.

I came back after 20 minutes and sure there was my ticket… it was in every way a genuine ticket. When I boarded the train about three hours later my name was on the reservation charts, the Ticket Collector had my name on his list and the whole thing left me very amazed. I talked about this to some other people in India about this and they were surprised to see me so excited. They blamed the system and the government for mismanagement and rampant corruption and bribery (very rightfully). They however did not believe that there was something more interesting going on here. To put things in perspective, I did a little research on expedia.com (popular flights and hotels booking site in the US): A flight from Chicago to San Francisco costs about $223 dollars if you book tickets two months in advance, costs about $350 (57% extra) if you book it one month in advance and $501 (125% extra) for the next day. The same goes with hotel booking too. I got a Rajdhani ticket with only 30% extra 4 hours before the journey. Now, that is some capitalism. If only the thing was legal…

Disconnected Computing

When UNIX was built it was supposed to run on big machines with a number of users connected to it. You may think of it as one centralized system catering to many stateless “terminals” from which users could log in and work. This model was good because terminals could be made very cheap and all data, programs and processing power could be consolidated on the central computer. Moreover, this centralization forced people to share resources and costs.

Too bad the PC industry changed every thing. UNIX was a bit arcane for people to use. It might have been easy to slap a GUI on top of it to make it less arcane for office folks… however, that didn’t happen when UNIX vendors had the window of time to do it. The result, the PC industry took over the office market with a easy to use GUI on much less powerful computers selling at lower costs per piece. The last nail in the coffin for UNIX in the office was the software distribution rights for the Microsoft Windows operating systems which Microsoft kept exclusively for the PCs. It seems that the UNIX vendors failed to realize the importance of office automation or any such low end (but with a large market) compute needs.

The emerging PC industry (which grew by leaps and bounds) fragmented the compute resources. Now, a business firm would order many machines the net resource value of which would be huge. However, the resource available to an individual user was limited. This made PC manufacturers happy, but increased the costs for business firms (it seems that the UNIX guys didn’t take their chance to lower their prices… and were taken by a surprise by this whole GUIed office automation revolution. They are still here but their market is very small; usually huge data centers and backup back-ends.).

The coming age is seeing a new era of fragmentation. This time this is not the office but the computing needs of individuals which is up for taking. This makes a huge market consisting of personal digital storage, games, multimedia and communication. Often, the personal computer is seen as a digital hub for the integration of all of these. Lately, Microsoft, which has the hold on personal computing software platform is loosing its hold on this integration. In some sense, because of the openness of the platform (or the flexibility and the power present in today’s PCs one might say) multiple vendors are trying to take their share with this fragmentation. A good example is Apple with its iPod taking the music revolution with itself. I have seen my friends offloading their entire music collection from their PCs into iPods… a device which doesn’t come under the control of Microsoft or its conglomerate of PC makers. Another good example is the Internet and so called “web applications”. The backbone of the Internet doesn’t run under Microsoft’s control… nor do the majority of platforms on which web applications are developed. The server side is a huge market but probably not so huge as the client side and we have open HTTP suite of protocol specs for any web application. A large corporation such as Microsoft did have a chance to control all that… let’s say fortunately or unfortunately it managed to slip Microsoft’s attention. Some other areas of fragmentation include movies/video (courtesy google/youtube/TiVo) and gaming (playstation/nintendo taking some market share though Microsoft’s Xbox did and continues to take some share back).

What we should note is that this time the fragmentation is not induced by artificially induced market forces (yup… still living in the dream of free capitalism?). This time it is more because of the proliferation of mobile devices and the failure of the “everything will be connected by TCP/IP network” idea. It is impossible to think of scalability of millions of devices connected to a single centralized server for all its services. Let’s say that wireless bandwidth is not a technological hurdle but more of an economic/political hurdle. Its impossible to create high bandwidth wireless networks without substantial capital investment and its impossible for millions of devices from so many vendors to collaborate together for low latency high throughput wireless connectivity. Economy favors creation of small devices each with its own storage, compute power and network connectivity. The thin client model still holds for quite some stuff… notably email, e-banking and web applications. However, the network kind of already lost its battle for thin client for multimedia which is the major force in personal computing.

The point of the above is to convince you that we are already in a fragmented, disconnected and disintegrated personal computing world (which is only getting more fragmented). The point from this sentence onwards is to convince you that we don’t yet have the right tools platform-softwarewise to deal with it.

All our major computing platforms depend a lot on storage. The storage in an integral, inflexible part of the platform. Storage encapsulates “state” and fragmentation of state is not good. Given that our mobile devices also have storage and hence “state” with them we have to deal with the problem of synchronization of this state. The “state” of the devices are definitely not as (un)pluggable as the devices themselves. Let me illustrate using an example. Consider iPod and iTunes. iTunes needs to be installed on the PC so that iPod works with it. The device could have been more usable if it was not so. iTunes keeps its configuration synced with the iPod at all times. This often takes a lot of effort both on the software development side (think backwards compatibility) and the user side (think time taken to sync; potential errors) and is an inconvenience to both parties. It would have been way better if the iTunes software was present on iPod itself and would have worked from there… more precisely, the program resident on the same device and executing on the state present on the same device. Note that I didn’t put a restriction on where the program actually runs. Let’s take a rather unconventional example of a PC game. PC games are delivered on CD/DVD ROM discs. During game installation you typically install several GBs of data on your PC’s storage. When you play the game, your game state resides on the PC and it is generally inconvenient to carry this state around. (Note that game consoles generally use a small flash drive for all game settings which is a good first step. Game and console companies could also keep all this state on the network). I actually find this humorous, there is a market for gaming laptops which are like these heavy beasts which carry the game, its state and settings along with big screens and graphics cards primarily for the reason that the game and its state is not portable. Consider the following situation. The game is distributed on a flash device which also keeps the game’s state. When the PC discovers such a device it runs the game on the PC but reads program/data/state from the device and writes new state back to the device. You could then have a market of game parlors where the game parlor supplies all the (updated and most cutting edge) hardware and gamers bring in their game flash drives for LAN play etc.

So it seems that keeping programs and related data on separate storage devices does make sense. This makes even more sense for mobile devices which already have storage (with limited compute) resources on them. Some examples are addressbook on cellphones, photos on digital cameras, software packages on flash drives and we already saw examples of iTunes on iPod and games on flash drives/DVD combo.

Let’s see how we can move towards building such a disconnected system. Wireless is our friend here and something like wireless USB could be used for connectivity between devices. The essential idea is a “console” which serves as a compute and primary user interface. Lets imagine a powerful CPU (say Core 2 Duo) and a good GPU hooked up together with enough of chipset to support wireless USB. The rest of our devices will be hooked up using wireless USB. In fashionable style, imagine that the CPU, GPU, chipset and power are embedded in our display (since this will always remain the largest part of our system) [Well the iMac comes pretty close except that it includes hard drives and CDrom drives with it]. Our console boots off a regular CD ROM in the console (the idea being that this piece of systems software is not touched or modified at all but is easily replaceable). Additions to the system whether hardware or software is done through the wireless USB bus. Each “addition” to the system keeps its state (if need be) on itself. The power problem still persists but lets hope that we are not too annoyed by batteries

One very important thing that we get for free here is isolation of various software packages from each other. More precisely, a piece of software can touch only files in its own package drive (and the user’s personal storage devices).

One of the most important ways in which this system differs from the systems of today is the filesystem layer. If disconnected operation is to be encouraged then we cannot make the user wait for filesystem syncing and “safe device removal”. A device should almost always be in removable state, irrespective of whether programs are running off the device or operating on data on a device. This means that most of this interaction has to be transactional in nature than the regular “open file (which locks device), load data, assume that the data is not changed before writing and write it back after modification”. We now have “open file (which doesn’t lock device but helps the user know if the file is being operated on), run one or more transactions (during a transaction the device is locked), close file”. Adding (application level) transactions also allows us to share an application+data device between two consoles simultaneously.

Application level transactions may have many ramifications. The application knows the last consistent state of the app + data. So when the device is connected back to the console the application automatically opens in its previous state. Suppose you are working on a document at your home console and you now want to move to work you just need to snatch the flash device which contains the app and data and put it in your pocket and walk away. Those applications will disappear from your home console. At your office when you connect your device back the application runs again, detects its last consistent state and opens up your document.

Of course one might think of a slightly more “heavy weight” flash drive for your app + data (say your smartphone) on which you might be able to do some stuff even if you don’t have the console. Perhaps the biggest gift of this kind of a system will be lesser weight to carry as a laptop

High Precision Sleep

Recently, for my research, I needed to make a network traffic generator which could ouput CBR traffic at a given rate with exponentially distributed on off times. Its actually pretty strange I didn’t find anything as simple as that on the internet in the first place. (So if any of you guys need it mail me.)

The problem I ran in building it had to do with high presicion sleeping with an accuracy in microseconds. Now, we do have nanosleep and usleep (which calls nanosleep()). However, they are no where as accurate as microseconds. Most OS kernels keep their internal timer resolution to 10ms to avoid a lot of timing overhead. I recompiled my linux kernel to improve the kernel timer resolution to 1ms. Even after that I saw that usleep actually had an error of around 2000us!

However, it seems that the gettimeofday() is pretty accurate as it uses the kernel timer along with help from hardware level clocks to get an accurate estimate of the time. The accuracy of gettimeofday is supposed to be as good as microseconds. However, I saw that the overhead of a gettimeofday() (its a system call) call itself was about 5us. That was good enough for me so I built a high precision sleep using usleep and gettimeofday. Basically, I divide the time required to sleep into parts for which I usleep on and busywait over gettimeofday() respectively. The code is given below.

#include<unistd.h>
#include<sys/time.h>

#define BUSY_WAIT       3000            //in microseconds

unsigned long long int get_clock()
{
        struct timeval tv;
        gettimeofday(&tv, NULL);
        return (unsigned long long int)tv.tv_usec + 1000000*tv.tv_sec;
}

int high_pres_usleep_untill(unsigned long long int end)
{
        //The basic idea is to divide the delay into sleep and busy wait parts.
        unsigned long long int busywait, start;
        int sleep, delay;

start = get_clock();
        delay = end - start;

sleep = (delay / BUSY_WAIT) - 1;
        if(sleep > 0)
                if(usleep(sleep*BUSY_WAIT))
                        return -1;

//Now busy wait such that from start till the end is the exact delay.
        while(get_clock() < end)
                ;

return 0;
}

If you require even greater sleep accuracy do take a look at the TSC register way of getting the current time. Its available on higher pentiums and it is a register which increments at the rate of the CPU clock. As CPU clocks are well over a GigaHertz you get nanosecond accuracy using that. The only hurdle that comes with it is calibrating it against a reference clock before you start doing anything with it (and some multiprocessor issues).

DRM

I got a bit enthused over recent Slashdot post on DRM and its comments. Slashdot also had some very good discussion on it. I thought I would blog about my thoughts on the same.

The motivation for DRM is pretty simple. Media content companies are afraid that the internet and the digital revolution make it very easy to replicate content. Given the relatively little price for such illegal copies of content (movies, music), these companies will go out of business if they don’t try to restrict this copying. Its debatable whether their business really suffers from piracy or not. However, content piracy is becoming a big problem for these companies as is evident from loads of content on P2P networks, IRC networks and BitTorrent.

One should also note that there are many other legitimate consumers who actually pay up and want the freedom to copy what they bought over as many types of devices as they want and share it with their near and dear ones. Without this freedom, these people (and I) really don’t see a reason to actually buy content. Let’s say that its because the cost of music and movies is anyways pretty high and the quality has been going downhill for some time now. DRM frustrates these customers. You end up paying the same (or perhaps even more; there has to be quite some engineering effort to get the DRM thing working smoothly) for content which you cannot share with people and which you cannot use with all your devices. This is really sad. Its pretty appalling that the content companies are forgetting this simple business principle: “keep your regular customers happy”.

The obvious question that comes up is – if not DRM then what? I would like to ask the content creation companies: Why do you think it is so unrealistic to manage the piracy crime using the techniques traditionally used for stopping crime; the police? Do you think that encumbering all media playback technologies with DRM is less costly than monitoring piracy and killing the real culprits? Why to you sing along with technology monopolies like Apple, Microsoft, Sony et.al. knowing the fact that they will never like to make DRM seamless and interoperable because of the simple fact that content creation is not their business? Given the proliferation of so many technologies to playback content (let’s count: Protected AAC, MP3, Protected WMA, flash players, mini hard drive players, so many third party programs to load music on devices, different DRM policies for each kind of device, Format wars, Codec Wars, Media Center wars… ) don’t you think that the average customer will be confused and will hold back shelling out money for stuff? Apple is a company which gets this “simplicity requirement” quite correctly and look at what it has made it in terms of brand and customer satisfaction. I ask you content creation companies, do you really think technology hype will sell your content? What’s in for you in all this mess?

I really love music. I buy CDs because I know that its a hell of a standard. *Any* device plays it. You can rip it and play it on your computer. Make playlists of your favourite songs. Sometimes I even do a salesmanship act by actually trying to convince my friends that some music I heard is great. Some of my friends take pains to even write reviews about recent movies and music. Don’t you content companies think that all of this will be lost the moment confusing technology comes home? Do you really think that my friend would actually buy another album of an artist if I didn’t hook him up with a favourite song from my library?

Why would you do this to yourself?