Visualizing TVTropes - Part 3: Proper batching

The story so far

Yesterday we were discussing our (admittedly) somewhat ghetto, not-quite-batched Neo4j implementation.

The long and short of it is that I was initially attacking the import process as one would on a JDBC client for a relational database. Query for these values here, create those if some don’t exist, insert relationships here, etc.

That seems to be woefully inefficient in Neo4j.

Visualizing TVTropes - Part 2: Neo4j optimizations

Welcome. You may want to first catch up on Part 1 of this little experiment.

All done? OK then.

Visualizing TVTropes - Part 1

Introduction

This is the first part of a series of articles on a small experiment I’m building. The intent is to crawl TVTropes, to find possible relationships between tropes and the material they appear in, as well as shared concepts.

I want to be able to not only visualize a concept and the tropes that it links to, but also how they relate to each other. A bonus would be being able to query how far away a concept is from another - for instance, how many steps we need to take before we can go from Cowboy Bebop to Macross Missile Massacre.

Chasing after NoSuchMethodError: com.google.common.io.ByteStreams.limit

This one bit me a bit hard and, seeing as I could find only one reference for it in the context of Clojure and Leiningen, I thought a small write-up would help.

After including some ClojureScript code on my project, lein started barfing when I attempted to build an uberjar with any optimization settings other than none.

Exception in thread “main” java.lang.NoSuchMethodError: com.google.common.io.ByteStreams.limit(Ljava/io/InputStream;J)Ljava/io/InputStream;, compiling:(/private/var/folders/64/ckdtdxm14059n9wh8rhf1mvw0000gn/T/form-init527739448955044836.clj:1:123)

There was only one obscure reference I found, from an IRC chat now a couple of years old. My first thought was that something in the ClojureScript code was throwing off the compiler, but it seemed odd that this would trigger a call to a specific non-existent method.

Other searches - outside of the lein context - made me realize it was actually a dependency conflict, and only triggered on uberjar because since there wasn’t any ClojureScript code deployed before the compiler hadn’t had to be invoked.

Calling

lein deps :tree

showed there were multiple versions of com.google.guava being referenced, with the one used by ClojureScript being 18.0 but a much older 11.0.1 being referenced by urly. Seems like lein either doesn’t realize there’s a conflict or defaults to the older library, which was causing the error when ClojureScript attempted to build.

ClojureScript vs. CoffeeScript for Processing sketches

DECEMBER 2015 UPDATE: I’ve run some tests with the latest ClojureScript and made some minor changes to the code, which significantly improved performance. Read this post for updated numbers. Keeping this post around for archival reasons.

The set up

This whole thing started not as a performance test, but as me experimenting with ClojureScript and Quil while reading Matt Pearson’s Generative Art. As such, it is not the most scientific of comparisons, and instead born out of my notes when exploring how to do sketches with ClojureScript for the web.

Consider the following example:

CoffeeScript

CoffeeScript is a huge improvement, but underneath all that synctactic sugar it’s clearly still Javascript:

Extending types from Clojure while working on the REPL

On the datatypes chapter of Programming Clojure 2nd edition (page 155) there’s the following bit where the CryptoVault is extended to support the default input streams:

The calls to spit/slurp didn’t work on my tests at first, even reloading the namespace.

UnitySteer acceleration smoothing changes

UnitySteer currently has an acceleration smooth rate parameter on TickedVehicle. It’s used to indicate the percentage of the velocity that is actually applied every frame to the current smoothed acceleration.

There’s two major issues with this:

• It’s framerate dependent, making vehicle behavior effectively non-deterministic.
• It makes it impossible to describe acceleration in the terms that people are used to thinking of them, for example 0 to 60 in 5 seconds.

The latter could be handled as an independent behavior or control, which gradually increases the maximum speed for the vehicle, but the former is a definitely something I want to address. And since I’m going to be working on that, I might as well implement the gradual acceleration as well.

The approach I’m thinking of following is adding two properties to AutonomousVehicle for maximum acceleration and deceleration per second. These would be evaluated on CalculatePositionDelta, so that the vehicle implementation can control its own acceleration style.

The only issue with this is that at the point that method is called, we have already assigned a speed to the vehicle when UpdateOrientationVelocity is called. This means the vehicle is supposedly moving at the full velocity, even if not necessarily all of it is being applied.

Overall, it looks like the necessary changes then are:

• Remove accelerationSmoothRate altogether, to eliminate the framerate dependent behavior.
• Add properties for maximum acceleration/deceleration per second to AutonomousVehicle.
• Not set Speed immediately, and instead save it on DesiredSpeed parameter (which is currently pretty much unused).
• When CalculatePositionDelta is called, we gradually lerp Speed to DesiredSpeed based on the time delta.
• Keep the Velocity property on AutonomousVehicle as is – defined as a function of the forward vector and the current speed.

This also has the advantage that it lets us simulate a bit of inertia, where it takes a vehicle a while to stop. It will be a major change and will have an effect on how your agents behave, so keep it in mind before upgrading.