Tamil

Information Processing in Tamil

Recent Efforts Towards Standardisation

Current Standards for Tamil

Need for the Proposed Standard for Tamil

Design Goals of the Proposed Standard

• A tamil font defined very much like the roman font such as Times or Helvetica, once installed in the system, can be used on all software packages supported by the respective OS without the need for additional software/hardware intervention. It is likely that over 90% of tamil computing is in the form of simple word-processing of plain text. The encoding standard must be such as to be readily implemented in most of the widely used computer platforms (UNIX, Windows and Mac). The input of tamil materials will be in all these three platforms. On the internet, the information exchange may involve any of the three OS (sender could use a Windows PC, the recipient a Mac and the intermediate mail server can be Unix-based).
• Fortunately in the last three years, procedures have been developed for production of fonts with identical encoding scheme that work under these different platforms. Information exchange via email and WWW has also been perfected that, no serious problems are anticipated in rapid implementation of the proposed scheme on all three OS. Tamilnadu Govt. is willing to undertake the task of producing one such tamil font and distribute it free on internet. Free distribution of a handful of such fonts will not deprive the software market. There will always be a need for specially designed fonts for professional usage (in publishing houses), very much the same way the font market still exists for roman fonts (Adobe and others continue to make millions marketing roman fonts!)

2. The encoding is at the 8-bit bilingual level, using a unique set of glyphs and the usual lower ASCII set (roman letters with standard punctuation marks) occupying the first 128 slots.

• Why a 8-bit bilingual scheme?

i) Almost all of the European languages (representing several hundred million population!) currently employ such 8-bit bilingual scheme, commonly known as ISO 8859-X schemes. Such 8-bit schemes are proven standards widely implemented by all major computer platforms. So, in terms of identification and implementation, the scheme is rather straightforward even for non-tamil speakers.
ii) A 8-bit scheme with lower ASCII part in the first 128 slot can facilitate enormously the smooth flow of information across the internet in all of the commonly used protocols (SMTP, FTP, HTTP, NNTP, POP, IMAP,..) All non-tamil speaking personnel entrusted with communication flow (postmasters, system administrators,..) can easily follow the content, its originator, destination etc.
iii) Tamilnadu as a constituent state of India works under a bilingual scenario with both English and Tamil as the languages for official communications. With a single font it will be possible to correspond in either or both of the languages. ISC-II standard of the Govt. of India is also defined in a similar way.
• What does it mean by a unique set of glyphs?

Tamil has far too many alphabets to be accommodated as a single glyph in the 128 slots left. So, depending on the complexity of the character (and its rendering) the scheme may use one, two or three bytes to define a single alphabet. But the choices of glyphs are such that, each of the 250+ tamil alphabets (uyir, mei and uyirmeis) are represented by one and only one way.
In the past, Tamil language used alternative glyphs for some of the tamil alphabets (e.g forward kombu/kokki to write lai/Nai/nai, Raa, Naa and Naa, referred to as ORNL). A unique definition scheme implies that there is no place for these old style characters in the encoding scheme.
What about character encoding as in ISCII and UNICODE?
If the glyph encoding scheme is UNAMBIGUOUS in defining the resulting character set, then it does not really matter if one choose to encode glyphs or characters. Defining a unique set of glyphs leading to a unique definition of all of the 250+ tamil characters makes the glyph encoding scheme unambiguous. Defining glyphs also defines the rendering part of the characters. The fact that we already have successful functioning of several tamil fonts in the market is a clear proof on the validity or implementation part of the approach. As mentioned under (1), the glyph encoding scheme allow design of self-standing simple fonts. Defining characters alone and leaving the rendering part to the software (as in UNICODE and ISCII) require dedicated expensive softwares. Most of the rendering methods currently employed in these schemes require modern font handling techniques that are available only on state-of-the-art computers. Unicode fonts and Apple Multi-lingual package (with Devanagiri) can be used only on the latest generation computers with Power PC chips and current OS software !!

• For centuries tamil language has grown with several grantha characters added on. The usage of these grantha characters along with pure tamil ones is so deep-rooted in the day-to-day usage of tamil by the common man. Hence the inclusion of these grantha characters becomes essential under the above criterion. Both ISC-II and UNICODE recognise this situation and have provided specific slots for a number of grantha characters.
• Unlike many of the tamil fonts and software packages that leave out rarely used tamil alphabets (such as ngu, ngU, nyu, nyuu), the present scheme ensures their presence. This has been done so that multimedia and softwares for teaching tamil can display all of the tamil alphabets without exception.

• Why unicode compatibility?

The glyph choices are to be such that, a one-to-one correspondence table between the alphabet/character definitions under the present scheme and UNICODE / ISCII can be established. A draft of one such mapping table is presented herewith in the annex section..
• There are major advantages by ensuring this requirement:

i) The end-user can have a choice in the storage format to be either the present 8-bit scheme or the unicode/iscii format. So files can be exchanged readily between users of these different standards without loss of integrity of the file;
ii) Secondly the present glyph encoding scheme can happily co-exist with the more sophisticated Unicode/ISCII schemes and even can make way for smooth transition to unicode at a future date. Indian language Packages for Unicode and ISCII are very expensive and have started appearing in the market only very recently. It is still largely under-explored domain for fool-proof implementation.
• What does unicode compatibility means in terms of glyph choices?

Both Unicode and ISCII scheme include a number of tamil numerals. So the present scheme need to include these tamil numerals. Else there cannot be a one-to-one correspondence between these forthcoming standards.

• The encoding scheme should leave at least 4-5 slots free for special use by software developers. None of the standard softwares written specifically for tamil will use the characters that are placed in these slots in "search" or "sort" type routines. However, use of this *special* characters in archives and other digital libraries is not encouraged so as to prevent mis-interpretation of their 'values' or 'meanings'.
• What are possible uses for such private use area?

Several possible usages can be envisaged for these private slots:
a) replacement of straight quotes by the corresponding curly quotes, as is the current default case in most of the Microsoft, Claris softwares for word-processing, graphics,...( vacant slots #145-148 strongly recommended for this purpose)
b) diacritical markers for writing transliterated/romanized form of tamil
c) old style tamil characters such as lai/Nai/nai or Raa/Naa/naa.
d) "escape slots" through which software developers can bring many special characters - such as those required for recording/processing of old classical tamil texts still in palm-leaf manuscript level.

• One-to-one correspondence table in the character definition as per the proposed standard with the popular tamil fonts/DTP packages will ensure smooth transition and recovery of all the archived tamil text materials produced till this date. There exist already conversion softwares that allow inter conversion of tamil text files prepared using different font encoding schemes. One such conversion software based on the proposed tamil standard can be made available to promote rapid and smooth transition to the new standard.

• The character encoding standard will have no restrictions on its use. It can be used freely for both commercial and private purposes.

Practically all of the tamil fonts and softwares that are currently in use world-wide are the recent work of individual authors and hence are subject to copyright protection to the authors. The copyright protection to authors is very clear with DTP packages. But when it comes to fonts, the scope (what can be subject to copyright and what is not) is very hazy and protection vary from country to country. So it is desirable to develop a true international standard that will be labelled clearly to be in public domain.
The proposed "encoding" is in public domain - i.e. no one needs to seek permission or state credit to implement a font based on the encoding. But the "implemented font" may or may not be copyrighted by the developer - this is entirely the developers descretion.

• It is very likely that with the widespread growth of a true international standard for tamil, large databases (library catalogues, electronic telephone directory, land/property registry, inventory of materials in departmental stores etc. etc.) are built based on tamil script. Routine usage of these databases often require search or sort routines. The encoding scheme should be such as to allow development of softwares for these without unnecessary demand for huge computer memory or processing capacity.

• What about the special needs of publishing houses that require a high quality output. Can a glyph encoding guarantee this?

As mentioned earlier, with 128 available slots it is not possible to keep all of the 250+ tamil characters as single unique glyphs. If we use the frequency of occurrence of various tamil alphabets in a typical tamil text as a guide, with the proposed glyph choices, nearly 90% of the text will consist of tamil characters appearing as full glyphs without invoking of kerning or other font handling techniques!
• Kerning is a routine font handling technique now available in most of the common computer platforms/OS. The present scheme envisages using kerning techniques to generate only two series of uyirmeis (ikara and iikara varisai). As a right-end modifier, the ikara and iikara varisai uyirmeis can be rendered fairly precise on all platforms. So it is likely that over 98% of the tamil characters can be rendered easily on screen and in print without loss of quality. Techniques such as pair-wise kerning can handle even the residuals adequately.
• Also, sophisticated software packages for professional publishing houses (that use high end computers ) can always invoke font/glyph substitution and bring in a single glyph for the required character in question rather than using a composite. Provisions have been made with escape slots for this purpose.
• What about the display in Point-of-Sale (POS) terminals?

Excessive invoking of kerning can pose problems for character display in POS terminals. As stated above, the kerning is invoked in only two series (ikara, iikara varisai). Even in these case the kokki falls apart in primitive POS systems, the tamil text should be still readable. Since the screen is constantly re-written, there should not be any problem to display all characters. Even in the character only terminals certain characters cannot be rendered legibly (Na, ha, ksha, sa may not fit in the usual 8 x 12 cell).

• There are several popular methods of input for tamil and these are considered under different keyboard layouts: classical tamil typewriter, romanized and phonetic or transliterated. Several Keyboard editors that allow input according to these different methods have already been developed and these can be readily adapted to include the proposed encoding scheme as the reference chart for the font in question.

• One possibility would be to agree on a supplementary ding-bat type font for exclusive usage amongst the tamil community - one that contains symbols such as OM, religious symbols, arrows, greek symbols etc. If the all tamil web pages use these two (one official tamil font and a second de-facto standard dingbat style font ), we can easily add some color and liveliness to the world of tamil computing.

Proposed Standard Code for Tamil

 Glyph choices for Slot positions 0-127 /rows 0-7:

Notes on Implementation of 8-bit encoding scheme