mercuric.net

After seeing Georg Zoeller’s phenomenal talk on development telemetry at GDC, I had a bunch of ideas about data and interesting things to do with it.  I’m still working on my streamlined bug reporting tool for NWN2 (ran into a wall dealing with IPC between C# and C++).  This post, however, is about a much flashier, much more finished project.

In the presentation above, there were several screenshots of useful, good-looking visualizations of data, including heatmaps of things like crash locations and gameplay activity.  I wanted to be able to make pictures like that!  Problem is, I couldn’t find any code or libraries to do quite what I wanted, and rather fighting with what was out there, I decided to roll my own.

I wrote the core code a while ago, so I don’t really remember enough to post-mortem that.  Much more recently, though, I felt unsatisfied with the “hacky” (to be generous) interface of one button in a form that ran the whole generation process when you clicked it.  It was time to revisit the project.

Most of the interface was just a matter of pulling several parameters to the HeatMap constructor out of code and into the GUI, but it makes me a lot happier to have a tool that other people can reasonably use.  At this point, it’s abstract enough to be used for many applications beyond the original use case of plotting deaths on a NWN area minimap.

The finished GUI.

Besides that, I’m very happy with my approach to scaling the gradient for maximum contrast over the map.  The data set mapped in this screenshot, for example, has a massive concentration of points at the gate near the bottom left, and relatively sparse concentrations almost everywhere else.  The simplest approach to scaling the gradient is a linear transition from minimum to maximum heat values, but this results in a hot point at the large concentration and cool-looking points over the rest of the map.

I fought with a few different approaches before I realized that I could approach most data sets as being vaguely normal distributions (albeit often skewed left or right).  This means percentiles are meaningful, and I can map them to particular points on the color gradient (e.g. dark blue is the 20th percentile, green the 60th, etc.).  Rather than some of my dead-end approaches, which required manual tweaking to get a map that looks good, this allows the program to calibrate automatically to the shape of the data set, leaving just a few intuitive parameters to twiddle until the user is satisfied.

In the interest of minimizing the number of times the wheel must be invented, the (C#) code is available here.  Do whatever you want with it, though I’d love to hear from you if you find it useful!

(As an aside, WordPress seems to change my C# tag to C++ whenever I save the post.  Suppose I’ll do without for now.)

I will try my hand at this blogging thing, so that perhaps my escapades can prove instructive to others.

As to the most recent escapade, I recently found myself fighting with Shorewall.  A hastily hand-drawn, but informative, diagram of the network:

184highnet

odin acts as bridge, router, and single point of catastrophic failure.  artemis and cobblebox have enough services running that I want them to have public IPs, so they are bridged to the cable modem segment.  Everything is also connected to the LAN, which odin NATs from its bridge interface to eth2.  odin provides DNS, DHCP, etc. to the LAN.  So far, so good (except that dhcpd still won’t update bind properly…).

Yes, this is far more complex than it strictly needs to be, but it means I can make pretty graphs and figure out how close I am to making Comcast angry.

mrtg eth0-week

I soon found that I need to be able to forward ports on odin’s external interface (br0) to machines on the LAN.  It’s easy with a two-interface machine: just say, for example, “any traffic in zone net with a destination port of 5121/udp goes to 192.168.10.2:5121″, or, in shorewall-speak (/etc/shorewall/rules):

#ACTION        SOURCE  DEST                    PROTO   DESTPORT        SRCPORT ORIGINAL_DEST
DNAT           net     loc:192.168.10.2:5121   udp     5121

Let’s give it a shot on the three-interface setup.  Oops, still can’t connect from dmz (the zone with artemis and cobblebox).  How about adding this, then?

DNAT           dmz     loc:192.168.10.2:5121   udp     5121

Great, now it works!  Except… oh shit, I can’t connect to anything else on port 5121 now.  WTF?

Turns out that those rules apply to any traffic in those zones with a destport of 5121, meaning all my outgoing traffic on 5121 was getting rewritten to go to 192.168.10.2.  To stop this, we use the original destination in the rules.  Because the original destination (read: the address of br0) is obtained via DHCP, though, it’s bad mojo to specify it in the file.  The solution to that is to modify /etc/shorewall/params.  The net result is:

/etc/shorewall/rules:
#ACTION         SOURCE  DEST                    PROTO   DESTPORT        SRCPORT ORIGINAL_DEST
DNAT            net     loc:192.168.10.2        udp     5121            -       $BR0_IP
DNAT            dmz     loc:192.168.10.2        udp     5121            -       $BR0_IP
/etc/shorewall/params:
BR0_IP=$(find_first_interface_address br0)

Yes, all this information was clearly in the FAQ if I had bothered to read, but my attention span is oh hey, I didn’t know WordPress could do that