for (ii in imap) delete imap[ii]

instead of delete imap,

which should keep the largest allocation of imap’s hash, and indeed gives you a bit more gawk speed.

Oddly, my mawk is not faster with these rewrites:

loui@loui-desktop:~/gawk$ mawk -W version
mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan

compiled limits:
max NF 32767
sprintf buffer 1020
loui@loui-desktop:~/gawk$ awk -W version
GNU Awk 3.1.8
Copyright (C) 1989, 1991-2010 Free Software Foundation.

BEGIN {
for (i=2; i in ARGV; i++) {
inf = ARGV[i]
outf = inf “n”
delete imap
I=0
while (getline < inf) {
if (!imap[$1]) imap[$1] = ++I
if (!jmap[$2]) jmap[$2] = ++J
print imap[$1] ” ” jmap[$2] ” ” $3 > outf
}
}
for (v in jmap) print v > “vocab”
}

will save you 10-30% in my gawk, depending on the length of your files.

Now, if Arnold Robbins would give us back the pre-allocate hash size option at the command line, we could get that faster using extra wide hashes.

Note that I am taking advantage of the serial file processing. I am a huge fan of awk/gawk, data pipieline processing, and pragmatics, such as how much time it takes you to write, debug, maintain, explain, etc.

Since Stallman showed me gawk in 92, I’ve been a huge fan. Good to see awk/nawk/gawk/mawk get some well deserved respect. Solaris nearly killed nawk by giving us the worst implementation in history.

A few language extensions could speed this up even more. Array-in and array-out would reduce the file-i/o overhead. ENDFILE and BEGINFILE would remove an annoying conditional. For long lines, a parse-to-n-and-quit, like the split in bash and perl, would also save time, since $n, n>3, is never referenced. It’s possible that a two-stage “i in array” test would be faster than hash-collision-chain traversal, where the first stage is a bloom filter.

In fact, this is another 10% faster…

BEGIN {
for (ifile=2; ifile in ARGV; ifile++) {
inf = ARGV[ifile]
outf = inf “n”
delete imap
I=0
while (getline outf
}
}
for (v in jmap) print v > “vocab”
}

awk “experts” abhor the use of BEGIN for the main loop, but the fact is, you get a lot more control, you can process multiple streams, you can increase readability and correctness, AND you can popularize the language for more general scripting use.

So happy to see discussions like this.

> This is a known limitation: mawk’s format for %d is limited by the format.
> The limitation is done to improve performance.
>
> You can get more precision using one of the floating formats (and can construct
> one which prints like a %d, e.g., by putting a “.0″ on the end of the format).
http://code.google.com/p/original-mawk/issues/detail?id=23

total_size, total_count, average: 2147483647 50586 2493242
total_size, total_count, average: 1.26123e+11 50586 2.49324e+06

Sadly, the exponential notation doesn’t work for the report’s audience. Is there some workaround for this? With careful direction I don’t mind editing the source code, but it’s really not my forte.

Again, thanks for all the useful comments.