A Story about Data, Part 2: Abandoning the notion of normality

Continuing on with my work, I was just about to conclude the non-normal data of the distribution. However, I remembered reading about different transformations that can be applied to data to make it more normal. Are any such transformations likely to have any effect on the normality (or the lack thereof) of the score data?
I’d read about the Box-Cox family of transformations: essentially proceeding through powers and their inverses, in the quest to improve normality. I decided to try it, using the Jarque-Bera statistic as a measure of the normality of the data.
Continue reading A Story about Data, Part 2: Abandoning the notion of normality

Basis: Plotting arbitrary coordinate systems in Ruby-Processing

One of the first things I realised while working on visualisations in Processing is that a lot of the work required in setting up coordinate systems and plotting them is somewhat of a chore. Specifically, for things like parallel coordinates, multiple axes, each with its own scaling, I initially ended up with some pretty ugly custom code for each case. I did look around in the Libraries section of the Processing website, but didn’t find anything specific to manipulating and plotting coordinate axes.
Continue reading Basis: Plotting arbitrary coordinate systems in Ruby-Processing

Data interactions in parallel coordinates: 40x-60x speedup

This is an update on the visualisation post on parallel coordinates. Understanding the Processing model made me realise that it probably wasn’t a good idea to draw all the samples each time draw() was called. Of course, every refreshed call of draw() does not clear away the previous frame’s graphics, so that just makes it easier. In the end, I went and explicitly drew only the samples which were under the current mouse position.

The speedup is obvious and massive: whereas the previous version worked well with only 300 samples, the current one processes 18000 samples without breaking a sweat. At 29,000 samples, there is a bit of a slowdown, but only just a bit, you wait 1 second for the highlighting instead of 6-7.
Here’s the new video, using 18k samples. Notice how much denser the mesh is.

Data interactions in parallel coordinates

Processing is growing on me. Inspired by the different and (very) interesting data visualisation examples I’ve seen, I decided to take a shot at interacting with the parallel coordinates that I generated here. Of course, I had to reduce the number of samples for this demonstration; it’d slow to a unholy crawl otherwise. For this video, I’ve taken 300 samples. The interaction is essentially a mouse-hover highlighting of any sample(s) under it. I fiddled with the colors a bit, but decided that a white-on-greyscale scheme would show up better.
Of course, I still haven’t gotten around to labeling the axes. This I’ll probably pick up next. But as the video demonstrates, there’s a lot to Processing than meets the eye.

PS: By the way, the actual demonstration ends around the halfway mark; I was trying to figure out how to stop the bloody recorder.

Getting ActiveRecord to behave nicely with Ruby-Processing in JRuby

Really, all I wanted to do was use Processing from Ruby. jashkenas has kindly written a gem which does just that. There was only a slight wrinkle: I wanted to pull my data from MySQL through ActiveRecord. Well, JRuby makes this process slightly more interesting than usual, so I document the process here. To start off with, install the gem with:

`sudo gem install ruby-processing`

Go into the directory where the ruby-processing gem is installed, and from there into {ruby-processing.gem.dir}/lib/core. In my case, this was /usr/lib/ruby/gems/1.8/gems/ruby-processing-1.0.9/lib/core.
Once inside there, you’ll find a file named jruby-complete.jar. Get rid of it, because we’ll be replacing it with a fresh (and different) version of jruby-complete.jar. Download the 1.6.3 version of JRuby-complete jar file. Rename it to jruby-complete.jar and put it in place of the jarfile we just deleted.

One step remains: this jarfile does not contain the activerecord-jdbcmysql-adapter gem. Install that with:

`java -jar jruby-complete.jar -S gem install  activerecord-jdbcmysql-adapter --user-install`

You’re good to go now. One more thing, just remember to replace the ActiveRecord adapter string with “jdbcmysql” and allow usage of that gem in your code with:

`require 'arjdbc'`

.