Python's sys.stdout loses encoding

When you use Python with sys.stdout you might run into a problem where sys.stdout.encoding suddenly becomes None. This happens due to the fact that upon using a pipe or redirection, at least under Unix, it falls back to not knowing anything about the target. In order to work around this you can add a fallback to use locale.getpreferredencoding(). So if you use encode() on a string you can do something like:

from locale import getpreferredencoding

text = u"Something special"

print text.encode(sys.stdout.encoding or getpreferredencoding() or 'ascii', 'replace')

This is how we currently use it within Babel as well for printing the locale list.

SQLAlchemy and simple WHERE clauses using AND

These posts use code from the Trac project. I'm using the question mark notation for in-place variable substitution, this is where you normally use either direct variables or an indirection mechanism.

If you have done SQL before you are familiar with the syntax such as:

SELECT name FROM auth_cookie WHERE cookie = ? AND ipnr = ?;

So, how does one do this with SQLAlchemy?

With SQLAlchemy (SA in short) you first declare a schema within Python:

auth_cookie = Table('auth_cookie', metadata,
        Column('cookie', String, primary_key=True),
        Column('name', String, primary_key=True),
        Column('ipnr', String, primary_key=True),
        Column('time', Integer))

Next you import this schema (living within Trac as trac/db/schema.py) as follows:

from trac.db.schema import auth_cookie

This allows direct manipulation using direct calls to auth_cookie. So for a SQL select we need to extend our code as follows:

from sqlalchemy import select

This allows us to build an almost equivalent statement as follows:

statement = select([auth_cookie.c.name], auth_cookie.c.cookie==?)

To add the AND clause SA has a very simple function to add into your code:

from sqlalchemy import and_, select

This allows us to extend the previous statement as such:

statement = select([auth_cookie.c.name], and_(auth_cookie.c.cookie==?, auth_cookie.c.ipnr==?)

Similarly there's an or_() function as well, which works exactly the same.

Now the difficulty arose due to the fact this SQL query changed its WHERE-clause depending on an if/else. The regular case was the first statement we created, the other case added the cookie's IP number into the equation. So how to deal with that?

statement = select([auth_cookie.c.name], auth_cookie.c.cookie==?)
if self.check_ip:
    statement.append_whereclause(and_(auth_cookie.c.ipnr==?))

As you can see, depending on whether or not check_ip is set it changes the statement in-place and expands the WHERE-clause with an AND for ipnr.

Office 2003, Visual Basic editor and AppLocale

So I was working with a Japanese .xla (Excel add-in) file. I needed to look at something in the source so I fired up the Visual Basic editor within Excel. Upon investigating the form and the various captions it turns out that the Visual Basic editor only displayed them in gibberish (typical decoding issues) or question marks (substituting the .notdef glyph for codepoints). So it seems the Visual Basic editor is either not multi-byte capable (typing directly a string in Japanese into the caption yielded question marks) or it is bound to the locale of the system.

I then remembered AppLocale and fired up Excel through it, setting it to think it is on a Japanese system. Then within Excel I proceeded to start the Visual Basic editor and, sure enough, the text was showing me the Japanese I needed.

I am not sure if I should find this lame or understandable.

Wah Nam Hong (華南行) in Rotterdam

Here in Rotterdam we have a Chinese supermarket called in Dutch phonetic Cantonese 'Wah Nam Hong', which in Jyutping (waa4 naam4 hong4) stands for the hanzi <span lang="zh">華南行</span>. Literally translated <span lang="zh">華南</span> stands for South China and matches the obvious Cantonese heritage. The <span lang="zh">行</span> stands for a profession or business line.

What is interesting to me is that in Japanese (<span lang="ja">日本語</span>) you read <span lang="ja">華南</span> as <span lang="ja">かなん</span> and it means South China as well. However, <span lang="ja">行</span> would be <span lang="ja">こう</span> or <span lang="ja">ぎょう</span> and has not retained the profession/business line meaning at all.

IA32 and the 4 GB memory problem

Found some information in two of Intels chipset datasheets about this issue:

The board utilizes 4 GB of addressable system memory. Typically the
address space that is allocated for PCI Conventional bus add-in cards,
PCI Express configuration space, BIOS (firmware hub), and chipset
overhead resides above the top of DRAM (total system memory). On a
system that has 4 GB of system memory installed, it is not possible to
use all of the installed memory due to system address space being
allocated for other system critical functions.  These functions include
the following:

BIOS/firmware hub (2 MB)
Local APIC (19 MB)
Digital Media Interface (40 MB)
Front side bus interrupts (17 MB)
PCI Express configuration space (256 MB)
MCH base address registers, internal graphics ranges, PCI Express
ports (up to 512 MB)
Memory-mapped I/O that is dynamically allocated for PCI Conventional
and PCI Express add-in cards

And the other note:

Memory between 4GB and 4GB minus 512MB will not be accessible for use
by the operating system and may be lost to the user, because this area
is reserved for BIOS, APIC configuration space, PCI adapter interface,
and virtual video memory space. This means that if 4GB of memory is
installed, 3.5GB of this memory is usable. The chipset should allow the
remapping of unused memory above the 4GB address, but this memory may
not be accessible to an operating system that has a 4GB memory limit.

The Beauty of Irony

I needed to look up something within a XHTML specification over at the W3 Consortium website. So I went to the XHTML2 Working Group Home Page. I was greeted with various encoding issues. Trademarks showing up as â„¢ character sequences. Now, normally when you see a page with an  or â at the start of a strange sequence you can be fairly certain it is a Unicode encoding, typically UTF-8. So at first I thought my auto-detect within Firefox was not turned on, checked it, no, it was definitely on. Selected Unicode as encoding myself and, indeed, the page displayed normally. So I checked the page's source. I was lovingly greeted by the following:

<?xml version="1.0" encoding="iso-8859-1"?>

I am sure most of you can appreciate the delightful irony that the organization that has a multitude of XML-based standards and specifications, which almost always use UTF-8 as default encoding, encode a page wrongly. Yes, mistakes are human, but to see something like this on the W3C site...

Science Fiction and Space Battles

Something I never understand between various science fiction movies and series is the difference in how space battles are conducted. Take for example Star Trek, you see all kinds of ships in fights, but contrast this to Battlestar Galactica, Space: Above and Beyond or Star Wars and there's one striking difference: the absence of fighter and bomber ships fighting alongside capital ships. In Star Trek you will see, for example, the Enterprise in direct battle with various other ships, but aside from some shuttles they have no fighter ships or otherwise. In Star Wars next to the Imperial Star Destroyers, Corellian Corvettes and Dreadnaughts, Mon Calamari cruisers, and Nebulon-B frigates you have a multitude of TIE fighters, TIE interceptors, TIE bombers, X-Wings, Y-Wings, A-Wings, B-Wings, and YT-1300 like the Millennium Falcon. Battlestar Galactica had its Vipers and other ships. I wonder what this, in my opinion, fundamental difference signifies and where it comes from.

Laughter amidst the Stillness

One of the reasons why I enjoy buddhism so much is the occasional (and sometimes even frequent) making fun of being too serious:

いざさらば
雪見に転ぶ(ゆきみにころぶ)
所まで(ところまで)

A typical 俳諧の連歌 consisting of a 5, 7, 5 metre by Matsuo Bashō (松尾 芭蕉).

This could be translated as such:

Now then, lets go out
to enjoy the snow... until
I slip and fall!