Ben Wendt's blog

File Integrity Monitoring in Python

Ben Wendt — Mon, 08 Sep 2025 20:00:00 -0400

Some Anti-virus software will monitor that certain key files, for example important binaries or the kernel, have not been modified. The way to do this is with file integrity monitoring. The general idea is to store hashes of files, and continually re-hash the files to make sure nothing has changed.

Let’s look at how this might look in python.

def monitor_files(path, monitor_log_file="log.pickle", hashes=(hashlib.sha256, hashlib.blake2b, hashlib.md5, hashlib.sha3_512)):
    if os.path.exists(monitor_log_file):
        with open(monitor_log_file, 'rb') as handle:
            monitor_log = pickle.load(handle)
    else:
        monitor_log = {}

    if not os.path.exists(path):
        raise Exception(f"Path does not exist {path}")

    new_log = {}


    for root, dirs, files in os.walk(path):

        for file in files:
            print(root, dirs, file)
            file_hashes = []
            for hash_ in hashes:
                m = hash_()
                with open(f"{root}/{file}", "rb") as f:
                    m.update(f.read())
                file_hashes.append(m.digest()) 
            if file in monitor_log:
                if file_hashes != monitor_log[file]:
                    print(f"hash mismatch for file {file}")
            else:
                print(f"new file {file}")
            new_log[file] = file_hashes
    with open(monitor_log_file, 'wb') as handle:
        pickle.dump(new_log, handle)

Notes about this code:

There should be a size check. The file system can quickly give the file size and if there’s a mismatch between expected and actual, it’s clear that the file has been modified.
Whole files are being hashed. This is fine for small files, but for big files, it’s better to store a bunch of small hashes and short-circuit the execution if a mismatch is found. For example, there’s no need to hash through a 100GB file if the first byte is wrong.
Multiple hashes are being computed. This makes it very difficult to perform a hash collision attack, as the probability of finding an attack that can satisfy multiple hashes simultaneously is very low.
Using pickle is a bad idea. Firstly, pickle is susceptible to arbitrary code execution, so if this code is running in an untrusted environment, the serde is an attack vector (however minor). It’s better to use a WORM storage medium or ship valid hashes to a trusted remote machine. Ideally the trusted hash is generated on a trusted machine to begin with.

Here’s a version that does the incremental hashing:

@dataclass
class FileHashes:
    whole_file: bytes
    chunks: list[bytes]
    window_size: int
    file_size: int

def rolling_fim(file_path: str, window_size:int = 256, monitor_log_file="rolling_log.pickle", hash_ = hashlib.blake2b):
    if os.path.exists(monitor_log_file):
        with open(monitor_log_file, 'rb') as handle:
            monitor_log = pickle.load(handle)
    else:
        print("warning, no old file integrity log exists.")
        monitor_log = FileHashes(b"", [], window_size, os.path.getsize(file_path))

    if not os.path.exists(file_path):
        raise Exception(f"Path does not exist {file_path}")

    if monitor_log.window_size != window_size:
        print("warning window size changed, expect all comparisons to fail.")

    new_log = FileHashes(b"", [], window_size, os.path.getsize(file_path))

    if new_log.file_size != monitor_log.file_size:
        print("file size changed. Files do not match.")
        return

    m = hash_()

    chunk_num = 0
    with open(file_path, "rb") as f:
        while True:
            n = hash_()
            chunk = f.read(window_size)
            if not chunk:
                break
            m.update(chunk)
            n.update(chunk)
            new_log.chunks.append(n.digest())
            if len(monitor_log.chunks) > chunk_num and monitor_log.chunks[chunk_num] != new_log.chunks[chunk_num]:
                print(f"mismatch found in chunk {chunk_num}")
                break
            chunk_num += 1
    new_log.whole_file = m.digest()
    if monitor_log.whole_file != new_log.whole_file:
        print("mismatch in whole file hash")

    with open(monitor_log_file, 'wb') as handle:
        pickle.dump(new_log, handle)

Notes on this code:

Switched to use one hash, blake2b. Blake2b is generally preferred for hashing large files due to its speed. We could implement a similar logic for computing multiple hashes of each chunk of the file to make collision attacks more difficult.
Added the file size check here, because why not.
The code computes a series of hashes of given window sizes of the source file. If one fails, short circuit execution.
Also compute a whole file hash, this is also to make collision
attacks more difficult.

This code is for illustration purposes only, but several improvements have been identified that, if implemented, could make this system more production ready. Further improvements would be needed to prevent a compromised machine from always returning known good hashes, such as remote verification.

Tell A Story To Write Better Code

Ben Wendt — Fri, 29 Aug 2025 20:00:00 -0400

Programming could be described as the process of giving a computer instructions. There are many ways to write programs that have within some tolerance the same function. Because of this arguments and philosophies arise as to what way code should be written. When starting out a lot of folks tend try to “golf” their code, make it as short as possible, while others may aim for best performance. I’m of the opinion that code should be written (given a certain functionality) as a document that communicates both the intent of the author and the functionality of the program.

My favorite technique for this is to have your code tell a story. My old boss Jack Neto advocated for this at an old job at a company that no longer exists, but I’m quite sure I had heard about it before then.

The idea is start with a function body that is essentially a script of your procedure. Fill it with nothing but function signatures of steps that will happen in that procedure. By reading this function, you’ll see exactly what the program does. You can iterate this so that the called functions are also little “scripts” of steps given by function calls, but eventually you get down to the level of granularity where the function bodies are short definitions of functionality.

The technique ends with a very readable and maintainable product, with the added bonus of making the writing process easier, testing easier, and it’s more fun. My favorite part about this technique is that you can spend a long time writing code, then go click run and generally just have a few minor things to fix up and it works.

Let’s look at some examples.

First I will preface with some versions of bad code to illustrate why this is a useful technique.

Imagine you asked a GPT to write a program to have your robot take your dog for a walk. It might write something like this:


def take_dog_for_walk(path, robot, dog):
    """Walk a dog and pick up poop."""
    # Traverse the path
    for waypoint in path:

        # go to the next waypoint
        while not robot.at_location(waypoint):

            # move along vector
            robot.step_in_direction((waypoint.x - position.x, waypoint.y - position.y))

            # use vision to detect poop
            if dog.pooping:
                while dog.pooping:
                    # a short delay
                    sleep 1

                # look at the poop and pick it up
                target = robot.locate_object(POOP, dog.position)
                robot.pick_up_object(target)

                # set boolean to true.
                robot.holding_poop = True

    # dispose of poop if necessary.
    if robot.holding_poop:
        robot.dispose_waste()

…Or something like that. It tends to write things very procedurally in one big function with lots of comments. Basically it’s the coding style that books have been written warning against, and many developers have always spent their careers fighting against.

Now lets rewrite it as a story.


def walk_dog(robot, dog, path):
    for waypoint in path:
        walk_dog_to_point(robot, dog, waypoint)
    dispose_poop(robot)

def walk_dog_to_point(robot, dog, waypoint):
    while not robot.at_location(waypoint):
        move_toward(robot, waypoint)
        wait_for_dog_if_needed(robot, dog)
        pickup_poop_if_needed(robot, dog)

This part of the code is super readable. All it does is tell the story of the walk. You barely have to understand python to understand this. This code is naturally decomposed, and has single responsibility.

Another great thing is if you want to use an LLM, you’ll now have a better outcome because something like co-pilot can often take the function signature and guess at the body of the function. You’ll of course have to verify that it’s right, but it’s a smaller piece and more logically coherent so that should be easier.

Let’s fill in the mock implementations of these functions based on the earlier silly code to fill in the details.

def move_toward(robot, waypoint):
    robot.step_in_direction((waypoint.x - position.x, waypoint.y - position.y))

def wait_for_dog_if_needed(robot, dog):
    if dog.pooping:
        while dog.pooping:
            sleep 1

def pickup_poop_if_needed(robot, dog):
    target = robot.locate_object(POOP, dog.position)
    if target:
        robot.pick_up_object(target)
        robot.holding_poop = True

def dispose_poop(robot)
    if robot.holding_poop():
        robot.dispose_waste()

Another added bonus is these smaller functions are naturally easier to unit test.

There’s a maxim in computer science “code is read more than it’s written”. One of my professors in university said to comment your code because often it will be you who doesn’t remember how it works.

In my experience if you write code as a document for reading as much as a list of instructions (or whatever) for the computer, it will be far more readable still. Formatting code as a story is the best way to communicate your intent with future readers. And who knows, maybe it will be you.

Generating fractals from Dask Distributed

Ben Wendt — Tue, 08 Jul 2025 20:00:00 -0400

Dask distributed is a python package / distributed processing framework that gives the ability to parallelize code execution across a cluster.

For a simple example, observe:


def double(x):
    x * 2

scheduler_host = "whatever"
client = Client("tcp://{scheduler_host}:8786")
scattered_inputs = client.scatter(list(range(20)))
futures = client.map(double, scattered_inputs)

results = client.gather(futures)

Of course I had to do a mandelbrot viewer. A mandelbrot fractal is great for parallelization because each pixel it computed independently of all others. The “heat map” generated by the mandelbrot fractal algorithm is basically the same as how long it took to compute each point!

from dask.distributed import Client
import dask
import numpy as np
import matplotlib.pyplot as plt


max_iter = 180

# reference implementation...
def get_mandelbrot_pts(x_y):
    iterations = 180
    x0, y0 = x_y
    x = float(0)
    xtemp = float(0)
    y = float(0)
    iteration = 0
    while ((x*x)+(y*y) < 4) and iteration < iterations:
        xtemp = x * x - y * y + x0
        y = 2 * x * y + y0
        x = xtemp
        iteration += 1
    return iteration

def inputs(w, h):  
    minx, maxx, miny, maxy = (-0.8, -0.7, -0.2, -0.05)

    inputs = []
    for x in range(w):
        for y in range(h):
            x0 = (maxx-minx)*float(x)/float(w) + minx
            y0 = (maxy-miny)*float(y)/float(h) + miny
            inputs.append((x0, y0))
    return inputs

w = 800
h = 600

# Dear reader, please don't hack me now that you have my internal IP
client = Client("tcp://192.168.1.163:8786")
scattered_inputs = client.scatter(list(inputs(w, h)))
futures = client.map(get_mandelbrot_pts, scattered_inputs)

results = client.gather(futures)

array_2d = np.array(results).reshape((w, h))

plt.axis('off')
plt.figure(figsize=(20,20))
plt.matshow(array_2d, fignum=1, aspect='auto')
plt.savefig('heatmap.png', bbox_inches='tight', pad_inches=0)
plt.close()

This will give you the standard fun image:

This only tells half the story though. We haven’t talked about running the scheduler and compute nodes!

On my local network I have many computers. Just for fun, definitely not performance (this particular problem runs better on one machine), I hooked up the following computers as dask compute nodes:

My 2019 MBP i7
My Steam Deck
A 2012 i7 Mac Mini
Two raspberry pi 4s
For fun, an Orange Pi Zero 2W
My creaking old i5 PC

(I don’t have a new-ish computer)

Enter dependency hell. Dask works on a “pickle everything” serde model, but pickle is notoriously picky about sending data between versions. Dask is super cool, but if I could make one criticism its that you can’t inject your own serde to use for IPC.

To sidestep dependency hell (not today, Dependency Datan, Dependency Jesus, take the wheel) I decided to run everything in docker. (Podman actually, but yes, docker is Dependency Jesus) Since all my CPUs were basically idle I think the overhead of running in docker is fine, especially for a toy implementation. Here’s the docker commands:

For the scheduler, only run once:

alias docker=podman
docker run --network=host ghcr.io/dask/dask dask-scheduler

Get your scheduler’s IP of course, and then run this on any node you want to be a compute node:

alias docker=podman
docker run --network=host ghcr.io/dask/dask dask-worker $SCHEDULER_IP:8786

All in all, it seems like a fun way to run code distributed. I can see how certain problems that are very parallelizable would be good to run this way, as compared to using something like spark. You don’t really need to learn a different mental model of distributed computing for this framework. But, having said that, the number of problems where this would be a performant solution may be limited, especially with being limited to python.

One drawback is that there doesn’t appear to be any managed service for dask in the big cloud providers, which means if you wanted to use this in production, you’d need to manage your own k8s cluster, which is another level of pain and complexity that people prefer to side-step.

Branch Coverage, A Cautionary Tale

Ben Wendt — Sun, 06 Jul 2025 20:00:00 -0400

Unit testing your code is the ultimate route to developer sanity. A well written test suite is a patchwork proof of the correctness of your system. The more complete your patchwork, the more you can rest easy knowing your code will do what you intend it to do.

The typical measure of the completeness of a test suite is line coverage. A profiler running while the test suite is run checks every line of code to see whether it has been run or not. In theory, the closer you get to 100%, the better tested your code base is, and the more easily you can sleep at night.

This of course isn’t always true. For example, consider the following code:

def contrived_example(data, a, b):
    encoding = "utf-8"
    if a:
        encoding = "not a real encoding"
    if b:
        encoding = "iso-8859-1"
    return data.decode(encoding)

def test_contrived_example():
    assert contrived_example(b"test", True, True) == "test"


test_contrived_example()

Congratulations, you have 100% line coverage! But what happens when you do this:

def test_contrived_example_2():
    assert contrived_example(b"test", True, False) == "test"

test_contrived_example_2()

Oh no, an error:

LookupError: unknown encoding: not a real encoding

This is because the all possible branches weren’t checked by test_contrived_example, as shown by test_contrived_example_2.

But sometimes branch coverage isn’t enough either. Consider this example:

def my_function(a, b):
    if a >= 3:
        a -= 3
    if b > 0:
        b /= a
    return b

def test_small_a():
    a = 1
    b = 2
    print(f"{a} {b} => {my_function(a, b)}")
    assert my_function(a, b) == 2.0

def test_big_a():
    a = 1
    b = 2
    print(f"{a} {b} => {my_function(a, b)}")
    assert my_function(a, b) == 2.0

def test_small_b():
    a = 1
    b = 0
    print(f"{a} {b} => {my_function(a, b)}")
    assert my_function(a, b) == 0.0


tests = [test_small_a, test_big_a, test_small_b]

for test in tests:
    test()

This “test suite” gives 100% branch coverage of our example function, but there’s still special case handling that can cause an exception:

my_function(3, 2)

Yields

ZeroDivisionError: division by zero

So even with 100% branch coverage we cannot be sure that our code is clear of errors. This is where the peer review process is important. More eyes on a code base will reveal these edge case issues that aren’t caught by code coverage.

Rotary MIDI

Ben Wendt — Fri, 27 Jun 2025 20:00:00 -0400

Using the BLEMidi.h library, you can get your esp32, the goat of dev boards, to be a midi server (instrument), or client (it’s complicated).

It’s pretty easy to get your esp32 to play notes:

#include <BLEMidi.h>


void setup() {
  Serial.begin(115200);
  BLEMidiServer.begin("Basic MIDI device");
}

void loop() {
    if(!BLEMidiServer.isConnected()) {
        return;
    } 
    BLEMidiServer.noteOn(0, 80, 127);
    delay(100);
    BLEMidiServer.noteOff(0, 80, 127);

}

But that’s not really an instrument. I am not a musician, but I wanted something I could play. In walks my joystick unit I used in my last post about making an automated mouse clicker. Out walks this musical abomination:

If you connect the pin outs as directed, this code will turn your joystick into an esp32 controlled octave pointer. Probably the worst musical instrument ever. As I said, I’m no musician, but I tried to record myself playing the easy intro part of ode to joy (E-E-F-G-G etc.) on this and it was well beyond my ability.

Regardless it was a fun little project making a MIDI “instrument”.

/*
GND    GND
+5V    3.3V (preferred for safety)
VRx    GPIO 34
VRy    GPIO 35
SW    GPIO 27
*/

#include <math.h>
#include <BLEMidi.h>
#include <Arduino.h>

#define VRX_PIN 34
#define VRY_PIN 35

void setup() {
  Serial.begin(115200);
  BLEMidiServer.begin("Basic MIDI device");
}

void loop() {
  if(!BLEMidiServer.isConnected()) {
    return;
  }
  float x = analogRead(VRX_PIN) / 2048.0 - 1.0;
  float y = analogRead(VRY_PIN) / 2048.0 - 1.0;

  float theta = atan2(y, x);
  float degrees = theta * 180.0 / PI;
  float magnitude = sqrt(x * x + y * y);

  // don't play a note unless the joystick is far enough from center.
  if(magnitude > 0.5) {
    int note = map( (180 + degrees) / 3, 0, 360, 52, 76);

    Serial.print("Note: ");
    Serial.print(note);

    BLEMidiServer.noteOn(0, note, 127);
    delay(100);
    BLEMidiServer.noteOff(0, note, 127);
  }
}

Cookie Clicking

Ben Wendt — Wed, 16 Apr 2025 20:00:00 -0400

Cookie clicker is a video game where the goal is to click a cookie many times. Each click increments a click counter, and as the number of clicks increases power-ups are unlocked that accelerate the counter incrementing rate. Clearly this is a silly and futile endeavor. But not all silly and futile endeavors are entirely pointless. Enter the Arduino leonardo, an IC controller that can act as a keyboard and mouse.

the device in question

My idea was simple, use the arduino to send a continuous stream of mouse clicks. One thing I learned from working with this thing in the past is that it’s good to have a failsafe mechanism. It’s not fun to have a device sending hundreds of mouse clicks with no way to turn it off. So there’s a connection between pin 8 and ground. When that is connected, mouse clicks will be sent continuously at whatever x-y coordinate is currently on screen. To stop it, just unplug that jumper wire. If I had a more well-stocked parts bin I would have used an on-off switch.

So here’s code that emulates the mouse with the joystick (with a left click option using joystick press) and the continuous click when the jumper is attached.

#include <Mouse.h>

int horzPin = A0;
int vertPin = A1;
int selPin = 9;         // Joystick button
int togglePin = 8;      // Jumper cable pin

int vertZero, horzZero;
int vertValue, horzValue;
const int sensitivity = 200;

int invertMouse = 1;

bool lastTogglePinState = HIGH;

void setup()
{
  pinMode(horzPin, INPUT);
  pinMode(vertPin, INPUT);
  pinMode(selPin, INPUT_PULLUP);
  pinMode(togglePin, INPUT_PULLUP);  // Jumper defaults to HIGH

  Serial.begin(9600);  // For logging

  delay(1000);
  vertZero = analogRead(vertPin);
  horzZero = analogRead(horzPin);

  Mouse.begin();
}

void loop()
{
  // Joystick movement
  vertValue = -(analogRead(vertPin) - vertZero);
  horzValue = analogRead(horzPin) - horzZero;

  if (vertValue != 0)
    Mouse.move(0, (invertMouse * (vertValue / sensitivity)), 0);
  if (horzValue != 0)
    Mouse.move((invertMouse * (horzValue / sensitivity)), 0, 0);
  bool currentToggleState = digitalRead(togglePin);
  if (currentToggleState == LOW) {
    Mouse.click();
  }
  if (currentToggleState != lastTogglePinState) {

    lastTogglePinState = currentToggleState;
    delay(100);  // Debounce
  }
}

When you plug this in you’ll get the uncanny clicker achievement in the game from clicking so fast.

Using ollama to make an epub synopsis

Ben Wendt — Wed, 08 May 2024 20:00:00 -0400

Like everyone else, I do my best to read many books, and like many, I find it to be a challenge, not just in finding the time, but also I found after you’ve read enough books, especially non-fiction, a lot of the book seems like filler. Different books on the same subject will cover the same material, or the author will be needlessly verbose in covering a topic. I thought ‘if I just want to read this book for the content, not the style, could I shorten it with an ai tool to ease some of the pain points?’

Now, synopsisizing has a long history, and has met with it’s share of detractors. For those wanting the sense of accomplishment of finishing a book, this isn’t for you. On the other hand, Coles Notes was once a staple of classrooms and campuses, and certainly helped many students achieve their goals. A manager I had a couple years ago was a proponent of using a synopsis site, I forget the name, but it served a similar purpose.

Anyways, I thought I would try making a tool to generate a shortened version of a text. The idea is to turn a chapter into something more like a page of text. My target genre is lighter non-fiction books and books about management, which I would like to read some of to help me with work.

llama3 is a gpt model. It’s pretty good, and you can download the full model and weights. A few weeks ago I was trying to install it, but ran into the usual cuda issues on my macbook, and had some annoying dependency issues derail me on my old linux machine that has a 1060 6GB that I use for AI stuff. So my momentum was gone. Then the other day, someone posted on the #random channel at work about ollama, which comes with a mac installer. I’ve always been a huge proponent of avoiding dependency hell. I’ve been pulled down into that morass regularly for decades. I’ve spent more time manually copying around specific versions of dll files than I can remember; that’s just how it was in the .net 1.0 days.

So, I ran the ollama installer. Once it is running you can do ollama run llama3 and chat with llama, as if you were in the instagram app. This can be useful, but running on my macbook, it’s actually much faster to run the inference on facebooks servers that I’m sure have hefty GPUs. But for my purpose I had another idea.

I can open an ebook in python. These contain a collection of xml documents, one per chapter. I can create batches of paragraphs and get a synopsis of each by prompting the ollama ai.

Here’s some code.

from itertools import islice
import ebooklib
import requests
from bs4 import BeautifulSoup
import json

# I found this code on stack overflow for making
# batches from a collection:

def batched(iterable, n):
    "Batch data into lists of length n. The last batch may be shorter."
    it = iter(iterable)
    while True:
        batch = list(islice(it, n))
        if not batch:
            return
        yield batch

def gen_prompt(author, title, all_text):
    """this is under the assumption that llama3 may already
    know something about the book, so it's a hint to add
    the title and author. I did not validate this assumption."""

    return f"generate a synopsis of this excerpt from '{title}'" "by {author} with no preamble and without referencing the " "author or the excerpt: {all_text}"

to_shorten = 'my_book.epub'
book = ebooklib.epub.read_epub(to_shorten)

# larger means less to read, but you'll hit a limit of what
# llama can handle if this is too big.
text_batch_size = 10

author = book.get_metadata('DC', 'creator')[0][0]
title = book.get_metadata('DC', 'title')[0][0]
print(f"author: {author} title: {title}")

for item in book.get_items():
    if item.get_type() == ebooklib.ITEM_DOCUMENT:
        print('==================================')
        print('NAME : ', item.get_name())
        content = item.get_content()

        # probably not the best choice.
        y = BeautifulSoup(content)

        # pretty sure this title logic will depend on which ebook
        # you are reading.
        title = y.findAll('h3')
        if title:
            print(f"{title[1].text}")
            print(''.join(['=' for l in title[1].text]))

        # I don't know enough about epubs to say whether they
        # all use the p tag.
        ps = y.findAll('p')

        batches = batched(ps, text_batch_size)
        for batch in batches:
            all_text = " ".join([p.text for p in ps])
            prompt = gen_prompt(author, title, all_text)

            # thanks ollama
            url = "http://localhost:11434/api/generate"
            body = {
                "model": "llama3",
                "prompt": prompt
            }

            x = requests.post(url, json = body)
            generated = ""

            # the response contains JSONL essentially.
            # One word per response object line.
            # Convert it back to readable.
            for dat in x.text.split("\n"):
                if not dat:
                    continue
                js = json.loads(dat)
                generated += js["response"]

            print(generated)

Overall the result is pretty good. I’m still trying to find a prompt that won’t start half of the outputs with things like "Here's a synopsis of the excerpt:", so it will read better, but overall I’m quite happy with the results. Whether I’m happy enough to get this running on my 1060, or if I’ll ever use it again remains to be seen. But I did use it to get the “super gist” of a book I had already read half of, and it worked quite well. I mentioned in my intro how I find many of these kinds of books to be quite repetitive and low-density for information. This leads me to doing skimming. I think for me, if I pay close attention to a synopsis rather than skimming, the result is about the same. This post must be leading by far in justifications and rationalizations per whatever, but that’s the nature of AI.

An Intro to Data Build Tool (dbt) With Hive

Ben Wendt — Mon, 25 Mar 2024 20:00:00 -0400

DBT has support for a wide variety of databases. So far, I’ve been using Google’s bigquery as the database for all of my DBT blog posts, so let’s try something different. For this demo I’ll use Apache Hive

Development Environment Setup

Apache hive has a quickstart guide that doesn’t quite work, so I will provide the steps I took to get it going in podman.

First, a very important step (I could imagine people wearing shirts with this):

alias docker=podman

Next, pull the image:

docker pull apache/hive:4.0.0-alpha-2

Next, run the image:

export HIVE_VERSION=4.0.0-alpha-2
docker exec -it hive4 beeline -u 'jdbc:hive2://localhost:10000/'

This will put you into a beeline shell (a hive CLI). Keep this open.

Setting up DBT.

This is based on the dbt hive setup. First, edit ~/.dbt/profiles.yml. Add something like this:

hive:
  target: dev
  outputs:
    dev:
      type: hive
      host: localhost
      port: 10000 # match your docker run from before.
      schema: my_schema

Now do pip install dbt-hive, then dbt init ${your_project} and choose hive for the adapter. Edit your new dbt_project.yml and make sure your profile name matches what is in the profiles file.

Now you should be able to do whatever you want in dbt. For my example, I set up a quick seed called randoms.csv:

id,random_value
1,2

This is a personal in-joke with myself. I had a professor in university [Dr Marco Polannen], who, when teaching us about randomness said “is 2 random?” (The point was that a single number can’t really be random, randomness is a property of a sequence where it is the shortest definition of itself). So after naming my model “random” I thought 2 would be a good seed value. Anyways… now you can run dbt seed. If all goes well, you can return to your beeline shell and run:

select * from my_schema.randoms;

And see something like:

+-------------+-----------------------+
| randoms.id  | randoms.random_value  |
+-------------+-----------------------+
| 1           | 2                     |
+-------------+-----------------------+

And that’s it. You could extend this starting point to add tests, models and so on. Have a nice day.

An Intro to Data Build Tool (dbt) Hooks

Ben Wendt — Tue, 19 Mar 2024 20:00:00 -0400

DBT supports hooks, which are a mechanism for running operations at different points of the execution of your DBT DAG. This is useful for running operations like adding a record to an audit table, running specific reports, and more.

Here I will show an example based on my “enriched movies” example from my previous blog post. Consider the use case where you want to export a table to your data lake after populating it using DBT run. You could do something like the following:

{{ config(
  post_hook = "EXPORT DATA
  OPTIONS (
    uri = 'gs://bq_movie_export/movie_export/*.json',
    format = 'JSON',
    overwrite = true)
AS (
  SELECT *
  FROM dbt_bwendt.enriched_movies
  ORDER BY movie_title
);
"
) }}

with movies as (
    select id as movie_id, title
    from {{ ref('movies') }}
),
actors as (
    select id as actor_id, name
    from {{ ref('actors') }}
),
movie_actor_mapping as (
    select id, movie_id, actor_id
    from {{ ref('stg_actor_movies') }}
)
select movie_actor_mapping.id,
    movies.title as movie_title,
    actors.name as actor_name
from movie_actor_mapping
join movies using (movie_id)
join actors using (actor_id)

Note the use of the config macro with a post hook specified. The action performed by the post hook is:

EXPORT DATA
  OPTIONS (
    uri = 'gs://bq_movie_export/movie_export/*.json',
    format = 'JSON',
    overwrite = true)
AS (
  SELECT *
  FROM dbt_bwendt.enriched_movies
  ORDER BY movie_title
);

This is a bigquery specific export option that will write the table to the specified GCS location. If you were to look in the generated file, you would see this in JSONL format:

{"id":"3","movie_title":"Avatar","actor_name":"Sigourney Weaver"}
{"id":"5","movie_title":"Paul","actor_name":"Sigourney Weaver"}
{"id":"2","movie_title":"Speed","actor_name":"Keanu Reeves"}
{"id":"1","movie_title":"Speed","actor_name":"Sandra Bullock"}
{"id":"6","movie_title":"Terminator 2","actor_name":"Linda Hamilton"}
{"id":"4","movie_title":"The Matrix","actor_name":"Keanu Reeves"}

Bigquery’s export automatically writes these out in a way that will be easy to read into another distributed processing engine like dataprow or dataflow.

An Intro to Data Build Tool (dbt)

Ben Wendt — Tue, 12 Mar 2024 20:00:00 -0400

DBT is a tool to simplify populating the relationships between different tables. With DBT, you can specify the queries used to create your tables, as well as parameterizing portions of those queries. You can also add data tests.

Introduction: Getting to Know Data Build Tool (dbt)

Ever wished for a smoother way to handle your data without the headaches of complex ETL processes? Enter Data Build Tool, or as we fondly call it, DBT. It makes data transformations and modeling feel like a breeze.

DBT takes a refreshing approach to data pipelines, letting you express your data transformations in good old SQL. No need for fancy jargon or convoluted workflows—just simple, straightforward SQL magic.

In this article, we’ll take a relaxed stroll through the world of DBT, from setting it up to some cool tricks it can do. So grab your favorite drink, kick back, and let’s dive into the world of DBT together!

Installation of Project Bootsrapping.

To get started with dbt, the recommended method for installation is to use pip.

pip install dbt-{your adapater}

I am using bigquery, but there are adapters for all the major database engines, including snowflake, postgres, cassandra, mySQL, SQLServer, SQLite, Oracle, Athena, Redshift and many more.

So for me, the install was done with:

pip install dbt-bigquery

This mat take a while because it has to install some GCS dependencies, such as grpc. You can then confirm your DBT installed correctly with:

dbt --version

Now you will be ready to start your own DBT project. I have been keeping a GCP console open to bigquery to verify my changes.

First navigate to the parent folder of where you want to store your DBT project, and then run:

dbt init

This will start a configuration wizard which has fairly reasonable defaults. When the configuration is complete, cd to your new folder and you will be ready to start working with DBT.

DBT Profiles

DBT init will have created a file named dbt_project.yml in your project root. Take a look! This will have all the info you entered in the setup wizard. It’s also where you can configure connections to external sources of data.

We can add a reference to the public dbt-tutorial dataset in GCP by adding this to dbt_profile.yml:

jaffle_shop:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: oauth
      project: dbt-tutorial
      dataset: jaffle_shop

Your profile is also the place where you configure your materialization strategy. The default is view. You can configure this by:

models:
  my_dataset:
    # Config indicated by + and applies to all files
    +materialized: table

You can override this materialization config on a per-model basis in your model.

Database Seeds

DBT supports database seeds. These are described as:

Seeds are CSV files in your dbt project (typically in your seeds directory), that dbt can load into your data warehouse using the dbt seed command. Seeds are best suited to static data which changes infrequently.

Seeds are not meant as a way of loading database dumps, but it does seem like it would be pretty easy to abuse this.

Here are some seeds I will use for my demo.

seeds/movies.csv:

id,title,year 
1,Speed,1994
2,Avatar,2008
3,The Matrix,1998
4,Paul,2012
5,Terminator 2,1992

seeds/actors.csv:

id,name
1,Sandra Bullock
2,Keanu Reeves
3,Mickey Rooney
4,Sigourney Weaver
5,Nick Frost
6,Linda Hamilton

seeds/stg_actor_movies.csv:

id,movie_id,actor_id
1,1,1
2,1,2
3,2,4
4,3,2
5,4,4
6,5,6

Note that this is closer to “loading a dump” than proper seed loading, but this is for a demo, so that is fine.

If you now run:

dbt run

And look in your database, you should see these three tables.

Models

The models/ folder is where you will put your model files. You can put a file in here such as:

flowers.sql:

select 1 as id, 'flea bane' as plant
union all
select 2, 'showy tick trefoil'

After a DBT run, these will appear in your database. It would be more common to do something like this:

stg_customers.sql:

    select
        id as customer_id,
        first_name,
        last_name

    from {{ ref('jaffle_shop', 'customers') }}

Note the use of the ref function here. DBT will resolve this reference to the table you set up in your profile; note how the first parameter jaffle_shop matches the profile name, and customers is the name of the table in that public dataset.

Macros

In the macros/ folder, you can define jinja macros that will be accessible from your models. For example, you could make a good customers macro called good_customer.sql:

{% macro good_customer(number_of_orders) %}
CASE when {{ number_of_orders }} >= 3 then
    true
    else
    false
end
{% endmacro %}

This is needlessly verbose but serves to illustrate how to use a macro. Assume then that we use this macro in model called customers.sql:

with customers as (
 select * from {{ ref('stg_customers') }}

),

orders as (

    select * from {{ ref('stg_orders') }}

),

customer_orders as (

    select
        customer_id,

        min(order_date) as first_order_date,
        max(order_date) as most_recent_order_date,
        count(order_id) as number_of_orders

    from orders

    group by 1

),

final as (

    select
        customers.customer_id,
        customers.first_name,
        customers.last_name,
        customer_orders.first_order_date,
        customer_orders.most_recent_order_date,
        coalesce(customer_orders.number_of_orders, 0) as number_of_orders,
        {{ good_customer('customer_orders.number_of_orders') }} as is_good_customer
    from customers

    left join customer_orders using (customer_id)

)

select * from final

Here you see the reference to the good_customer macro. You can see the generated sql by looking in the target folder. So for this model, you could run cat target/compiled/my_proj/models/customers.sql, where you would see something like this:

with customers as (
 select * from `my_proj`.`my_dataset`.`stg_customers`

),

orders as (

    select * from `my_proj`.`my_dataset`.`stg_orders`

),

customer_orders as (

    select
        customer_id,

        min(order_date) as first_order_date,
        max(order_date) as most_recent_order_date,
        count(order_id) as number_of_orders

    from orders

    group by 1

),

final as (

    select
        customers.customer_id,
        customers.first_name,
        customers.last_name,
        customer_orders.first_order_date,
        customer_orders.most_recent_order_date,
        coalesce(customer_orders.number_of_orders, 0) as number_of_orders,

CASE when customer_orders.number_of_orders >= 3 then
    true
    else
    false
end
 as is_good_customer
    from customers

    left join customer_orders using (customer_id)

)

select * from final

Note how the case statement has been inlined from the macro.

Schemas

In the model folder, there is a file called schema.yml that will contain the schemas of all of your tables in yaml format. This is not auto-generated. You should be going in here and populating any fields you create. So for my customers table, I have:

models:
  - name: customers
    description: One record per customer
    columns:
      - name: customer_id
        description: Primary key
        tests:
          - unique
          - not_null
      - name: first_order_date
        description: NULL when a customer has not yet placed an order.

Note the presence of the tests key. This is where you will define your data tests, including the ones I’ve shown here for column constraints, but you can do more interesting stuff as well.

A DBT test passes when the query it represents returns zero rows. So imagine I made a macro called test_does_not_contain.sql like this:

{% macro test_does_not_contain(model, column_name, unwanted) %}
select {{column_name}}
from {{model}}
where contains_substr({{column_name}}, '{{unwanted}}')
{% endmacro %}

(Note that test macro names must begin with test_.)

I can then use this test for my fields by adding it in the schema, like so:

- name: customer_orders_by_name
    description: a report on how many orders come from customers per first name.
    columns:
      - name: first_name
        description: the 1st name.
        tests:
          - unique
          - not_null
          - does_not_contain:
              unwanted: 😂
      - name: total_orders
        tests:
          - dbt_expectations.expect_column_values_to_be_between:
              min_value: 0
              strictly: false

This will now throw an error if any first names contain the laughing smiling emoji ‘😂’.

Note the second added test. After installing the dbt_expectations package you can use its tests, and this one does range checking. You can install the package by running dbt deps --add-package dbt_expectations:1.0.0.

Data Tests

For larger tests, such as data quality tests, you can add a test in the tests folder, like this:

select 
    order_amount_cents
from
    {{ ref('order_amounts') }}
where order_amount_cents <= 0

As before, DBT expects zero rows to be returned from a test for it to pass. You can configure the expected number of rows for a failure or warning. You can run all the tests with dbt test. Tests will likely fail if you haven’t first populated their underlying tables. Tests also run when you do dbt run.

Analyses

If you want the goodness of re-usable code to generate SQL without necessarily using it to populate a table, you can put SQL in the analyses folder. For example you could make movies_by_year.sql:

select year, count(*)
from
    {{ ref('movies') }}
group by year

Then run

dbt compile

Then inspect the file:

cat target/compiled/my_proj/analyses/movies_by_year.sql

This will give you something like:

select year, count(*)
from
    `my_project`.`my_dataset`.`movies`
group by year

Python models

DBT also supports python models. You could make something like transformed_customers.py:


def my_transform(name):
    return f"{name} {name[0].lower()}"

def model(dbt, session):
    dbt.config(materialized="table")

    df = dbt.ref("customers")

    pdf = df.to_pandas()

    pdf['name2'] = pdf['first_name'].apply(my_transform)

    return pdf

You don’t have to use pandas here. In fact, is probably quite often a bad idea. df will be a spark data frame.

This will apply your custom transformation logic to each row and save that in the new table. Note that the python logic will be run through whatever flavour of spark is convenient for your platform. So since I’m using bigquery, that means dataproc.

Conclusion

And there you have it -— your crash course in Data Build Tool (DBT). It’s like the Swiss Army knife of data, simplifying your life and making building and managing data projects a whole lot easier.

As you venture into the world of DBT, remember that you’re equipped with a powerful tool that’s trusted by data professionals worldwide. Whether you’re a seasoned analyst or a newcomer to the data scene, DBT offers a friendly and intuitive platform to work with.

So take a deep breath, relax, and dive into the wonderful world of DBT. Experiment, explore, and have fun with it! After all, data doesn’t have to be daunting -— it can be downright delightful with DBT by your side.

Here’s to embracing data adventures with a smile—dbt style!

Finding Sator Squares

Ben Wendt — Sun, 12 Nov 2023 19:00:00 -0500

I had a visit with my mother recently where she introduced me to the idea of Sator Squares. It’s a five letter acrostic, popular in ancient Rome, and originally rediscovered during the excavation of Pompei and Herculaneum. It has the interesting property that transposition of the matrix is an identity operation.

Image from wikipedia

The drawing she wrote of it has been sitting on my counter for ages, and I’ve been meaning to code it ever since I first saw it.

Here’s a definition of whether or not a collection of five words (assumed to be five letters, which is safe because I’m re-using my wordle dictionary):

def is_sator2(w0, w1, w2, w3, w4) -> bool:
    return w0[1] == w1[0] and w0[2] == w2[0] and w0[3] == w3[0] and w4[0] == w0[4] \
        and w1[2] == w2[1] and w1[3] == w3[1] and w1[4] == w4[1] \
        and w2[3] == w3[2] and w2[4] == w4[2] \
        and w3[4] == w4[3]

I tried using itertools.permutations to check every option, but it was horribly inefficient, due to checking so many things that aren’t even possibly sator squares.

Here’s an optimization that can rip through thousands of words per second:

# word_map is a mapping of first letters to words that start
# with that letter.
def look_for_sator(search_in, word_map):
    for word0 in search_in:
        print(word0)
        for word1 in word_map[word0[1]]:
            for word2 in word_map[word0[2]]:
                if word2[1] != word1[2]:
                    continue
                for word3 in word_map[word0[3]]:
                    if word3[1] != word1[3] or word3[2] != word2[3]:
                        continue
                    for word4 in word_map[word0[4]]:
                        if word4[1] != word1[4] or word4[2] != word2[4] or word4[3] != word3[4]:
                            continue
                        if is_sator(word0, word1, word2, word3, word4):
                            print("======")
                            print_sator((word0, word1, word2, word3, word4))

This will give nice outputs like:

hippo
======
hippo
idler
plead
peace
order
======
hippo
inlet
plant
pence
otter
======
hippo
inlet
plant
penne
otter
======
hippo
inlet
pleat
peace
otter
======
hippo
islet
plant
pence
otter
======
hippo
islet
plant
penne
otter
======
hippo
islet
pleat
peace
otter

An uninformative error in bigquery

Ben Wendt — Tue, 24 Oct 2023 20:00:00 -0400

Just a quick note about an uninformative error I saw in bigquery the other day and was having trouble finding on google. If you see the error:

MaterializedView is required for DerivationSpec

When trying to create a materialized view in bigquery on google cloud platform (GCP), it means you didn’t specify the query, or including an empty query. It’s an easy mistake to make. Maybe this will help someone some day.

Schemas and SqlTransform in Beam

Ben Wendt — Thu, 13 Jul 2023 00:00:00 -0400

Beam has support for working with collections of data that conforms to a schema, and you can use SQL Transforms to transform this data. This feels a bit more like working with data in spark, but beam does not have the same level of optimization.

Here’s some imports. I’ll leave these in because beam’s docs are not great and I had to copy these out of a youtube video when I was learning to do this:


import apache_beam as beam
from apache_beam.transforms.sql import SqlTransform

import typing

from faker import Faker

Here is my schema class. Beam wants it to be a named tuple.

class Person(typing.NamedTuple):
    person_id: int
    name: str
    fave_color: str

And you have to register a coder for the class:

beam.coders.registry.register_coder(Person, beam.coders.RowCoder)

And here’s an example showing some usage of the schema and a transformation:

with beam.Pipeline() as p:
    fake = Faker()
    people = (
        p |
        "get ids" >> beam.Create(list(range(10_000)))
        | "to people" >> beam.Map(lambda person_id: Person(
            person_id=person_id,
            name=fake.name(),
            fave_color=fake.color_name()
            )
        ).with_output_types(Person)
        | SqlTransform("""
            select fave_color, count(*) as `COUNT`
            FROM PCOLLECTION
            group by fave_color
        """)
        | "print" >> beam.Map(print)
    )

Solving Spelling Bee and Letter Boxed

Ben Wendt — Mon, 14 Nov 2022 19:00:00 -0500

I’ve written many times about solving wordle. I still play wordle most days, but recently I’ve moved on to playing other daily games available on the NYT site, spellling bee and letter boxed. So let’s solve those too.

Spelling bee is just about spelling as many words as you can, but they all have to inclue the central letter. The ideal is to get a pangram, i.e. a word that includes all the available letters.

Here’s a regular expression that will find all possible solutions to spelling bee, assuming clhawbe are the available letters and a is the mandatory letter:

grep -E '^[clhawbe]+$' /usr/share/dict/words | grep a

This doesn’t solve the pangram but it should be clear how to extend to solve for that.

Letter boxed is played on a box with three letters on each side. The aim is to spell words that use up all the letters within five turns, but you cannot use two consecutive letters from the same side of the square. And each new word must start with the final letter of the previous word. Here’s a python that gives all the words you can spell. First, we’ll set up the dictionary and the letters we have to work with:

from english_words import english_words_set

english_words = list(english_words_set)

groups = (('r','h', 't'), ('m', 'a', 'd'), ('u', 's', 'o'), ('i', 'w', 'c'))

Here’s the function that can determine whether a word can be spelled given the rules of the game:

def can_spell_word(word, groups, exclude_group=None):
    if word == "":
        return True
    for i, group in enumerate(groups):
        if i == exclude_group:
            continue
        if word[0] in group:
            return can_spell_word(word[1:], groups, i)

    return False

Ages ago I was talking programming with some colleagues. One mentioned having trouble thinking about how to write recursive functions. A second co-worker recounted something one of his professors had told him:

The trick to writing recursive functions is to write it like it’s already been written.

I love that expression.

Anyways, here’s iterating through the dictionary to find the words you can spell:

found_words = []
for word in english_words:
    if len(word) < 3:
        # game rule of minimum letters.
        continue
    if can_spell_word(word, groups):
        found_words.append(word)

I didn’t bother extending this to finding which groups of words can solve letter boxed in a given number of turns, or the least number of turns but it should be very doable from here.

Safe Navigation in Python

Ben Wendt — Wed, 12 Oct 2022 20:00:00 -0400

In a chain of method calls, what happens when one of the calls returns a null? The next chained call will throw some kind of reference error. But what if you don’t want to deal with a reference error? What if you want the null value to be the answer? That’s where a safe navigation operator is useful.

Ruby has a safe navigation operator &.. This is really useful in database applications. For example, consider a database model of a tree, so you might have something like:

tree.trunk.branches[2].branches[5].twigs[2].leaves[1]

But what if your tree doesn’t have that leaf, or that branch, or if it has no trunk? But also you don’t mind “null” being the answer to your query. Well that’s where safe navigation is useful. Observe:

tree&.trunk&.branches&.[2]&.branches&.[5]&.twigs&.[2]&.leaves&.[1]

This would return null if any of the calls in the chain return null, without having 9 levels of short-circuiting.

Note: I’m ignoring the law of demeter in this post.

But python does not have a safe navigation operator. One has been proposed, but currently it doesn’t exist.

it is however possible to roll one. Consider an input document like this:

<!doctype html>
<html>
    <head>
        <title>Something</title>
    </head>
    <body>
        <article>
            <h1>My article</h1>
            <div>
                <p>Some text <b>yelling <i>curiously</i></b>.</p>
            </div>
        </article>
        <article>
            <h1>My article</h1>
            <div>
                <p>Some text <b>yelling <i>curiously</i></b>.</p>
            </div>
        </article>
    </body>
</html>

Reading the article with beautiful soup, you can access various html nodes using a chained dot notaion:

import bs4
from typing import List, Optional


with open("something.html", "r") as f:
    soup = bs4.BeautifulSoup(f.read(), "html.parser")

print(soup.body.article.div.p.b.i)
try:
    print(soup.body.article.div.p.c.i)
except AttributeError:
    print("that did not work.")

Here we can read the contents of the b tag, but not the c tag, since the latter doesn’t exist. But if we have the function we talked about:

def safe_navigate(soup: Optional[bs4.BeautifulSoup], tag_list: List[str]):
    if tag_list:
        return safe_navigate(getattr(soup, tag_list[0], None), tag_list[1:])
    else:
        return soup

print(safe_navigate(soup, ["body", "article", "div", "p", "b", "i", "text"]))
print(safe_navigate(soup, ["body", "article", "div", "p", "c", "i", "text"]))

Now our document query works in both cases, the query with the b gives the expected result, and the query with c returns None.

A Tale From Microservice Hell

Ben Wendt — Tue, 20 Sep 2022 20:00:00 -0400

Ages ago I was working as a contractor on a data engineering team, working on a knowledge graph platform that was built on a service architecture.

The platform had dozens of services. Each service had its code in its own repository. We used containers for deployment, testing, and so on. This meant we had dozens of containers to migrate over. Several of the containers were built on top of other containers, for example there was a base python container, and a base java container. All of these containers had images hosted on docker hub.

A decision was made to change container registries. This meant we would have to go in to every repository, update the container deployment in the CD code, then once that was merged, update the container pull code and make a separate pull request in an infra repo to update the container location there. And because we had several base containers, that meant there were several more containers that we would need to deal with, and we would need to handle those base containers first.

All told, each repo had two or three PRs needed to update the container repository, and across the dozens of repositories this added up to about 40 pull requests. I decided that the 40 subtasks would overwhelm JIRA so all tracking was done in a big spreadsheet with rows for each container image and columns for each task state.

I eventually got through the migration, but it was pretty painful. The process put a large PR overhead on the team for the several weeks the job took to complete. The mechanical part of updating everything was manageable, but the hardest part was tracking down all the different stake holders to determine which images were in use, and which were not (and did not need to be migrated). Even with my best efforts there we still missed a couple images that we had to go back and migrate later.

Running a shell command for each entry in a PCollection

Ben Wendt — Sat, 16 Jul 2022 20:00:00 -0400

I was preparing for something I needed to do at work with beam, namely running a shell command for something that isn’t possible to run natively in the beam runtime, and did not find much documentation for it. It’s relatively straight-forward but I have not “blogged” in a while so here goes.

Cowsay is a notoriously difficult algorithm, which no one outside of the original author has been able to optimize to work in less than O(n^n) time. As such, the best way to access the mind expanding functionality is to use the binary:

import apache_beam as beam
from subprocess import run

with beam.Pipeline() as p:
    cowed = (p | 
        "sayings" >> beam.Create([
            "You're damned if you do, damned if you don't.",
            "Takes one to know one."
        ]) |
        "cow it" >> beam.Map(
            lambda x: run(["cowsay", x]).stdout
        )
        | "output" >> beam.Map(print)
    )

This will give this delightful output:

 ________________________________________
/ You're damned if you do, damned if you \
\ don't.                                 /
 ----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
None
 ________________________
< Takes one to know one. >
 ------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
None

Another application would be to draw upon feature-rich applications, such as ffmpeg:

import apache_beam as beam
import glob
from subprocess import run

files = glob.glob("*.mov")

with beam.Pipeline() as p:
    cowed = (p | 
        "sayings" >> beam.Create(files) |
        "cow it" >> beam.Map(
            lambda x: run(["ffmpeg", "-i", x, f"{x}.mp4"]).stdout
        )
        | "output" >> beam.Map(print)
    )

Of course you would tweak the ffmpeg command line settings for whatever you are doing here.

Anyways, pretty straight-forward but thought it was worth “blogging” about.

Transferring Musical Likes Between Services

Ben Wendt — Sat, 21 May 2022 20:00:00 -0400

After using spotify for many years, I decided to try switching my music service a few months ago. I was finding that any time I put on shuffle, spotify would eventually direct me back to funk and soul music from the seventies. I generally love those styles of music, but for a service that is supposed to provide discovery, it wasn’t really working for me.

I compared the various services available in Canada on the following criteria:

Library
Sonos Support
iOS Support
Mac Support
CarPlay Support
Google Home Support

Tidal met most of my needs, so I gave it a try. It’s a good service, kind of lacking in the polish I got used to with spotify, but the library was good and I could listen on most of my devices. I even found it has an app for my roku TV, which I never really used. I was a bit disappointed with the CarPlay app, but it was okay. The real sticking point was no Google Home, so when my trial ended, I signed up for a youtube music trial.

One thing I will say for tidal was the discovery was like a bresh of fresh air coming from spotify. I found a lot of really great stuff on there.

Youtube Music doesn’t have a real mac app, but there is a “web native” app, which is basically what “chrome apps” are now, from what I can tell.

When I left spotify, I exported my thousands of Liked songs with the hope of importing them to whatever service I settled on. Unfortunately Tidal’s API is worse than almost any I’ve seen, and Youtube Music doesn’t even have an api. There is, however, an unofficial API for Youtube Music called ytmusicapi.

It’s a bit of a kludgey library, working by pretending to be a client in a browser, but overall it worked great. Youtube Music seems to have some kind of protection in place to prevent abuse so if you just fire off a tonne of requests using the library, you will see a lot of timeouts. The usual way to deal with something like this would be to use a retry decorator But I decided to roll my own, which I thought was nifty and worth sharing:

def retry_backoff(fn, args, i=1):
    try:
        return fn(*args)
    except ReadTimeout:
        time.sleep(10 * i)
        return retry_backoff(fn, args, i+1)

Then you can call this with something like:

retry_backoff(ytmusic.rate_song, [video_id, 'LIKE'])

Obviously would be more useful if the Exception type and sleep multiplier were configurable, but it worked for me. I got all the Likes imported. I may move off youtube music, maybe even starting a new spotify account, but for now I am happy.

Doing some biological modeling

Ben Wendt — Mon, 02 May 2022 20:00:00 -0400

At my new job our focus is to uncover scientific insights to accelerate and improve scientific research. I’m a data engineer by trade, but the science side of the business bleeds in to what I do on a fairly regular basis. Without the scientific and research background that many of my peers have, I feel the need to do some self improvement in those areas. So I was happy to find a series of videos on youtube titled Computational Biology with Python (Modeling Gene Networks) created by Mike Saint Antoine

The video series centers around modeling ordinary differential equations and stochastic differential equations using python, specifically for gene expression, and the central dogma of molecular biology. This was great for me, because that is a topic that has come up at work, so it was a great chance to bridge my history of working with code, and my ancient (university days) history of working with math.

The central dogma of molecular biology is that information flows from DNA to mRNA to proteins. That is DNA is transcibed into mRNA and mRNA expresses proteins. I think.

DNA ➡ mRNA ➡ protein

This is modeled roughly as:

mrna_change = k_m - gamma_m * m
protein_change = k_p * m - gamma_p * p

So the amount of mRNA has a constant upward change, but it slows down as more mRNA is present. And the amount of the protein goes up proportional to the amount of mRNA present and decreases proportional to the amount of protein present.

I went ahead and coded the model for myself before the videos showed a solution. Here’s my code:

m_0 = 1.0
p_0 = 1.0
gamma_m = 0.4
gamma_p = 0.6
k_m = 0.5
k_p = 0.9

def next_m_p(m, p):
    m += k_m - gamma_m * m
    p += k_p * m - gamma_p * p
    return (m, p)

m = m_0
p = p_0
mps = [(m, p)]
for _ in range(100):
    m, p = next_m_p(m, p)
    mps.append((m, p))

This gives a plot like this, which the next video showed how to do with odeint:

The course goes on to talk about more complex systems and finishes with a stochastic model of a 3 gene oscillating system with I think was called a Goodwin Oscillator.

Way back in my university days I took a course in fourth year called Mathematical Modeling, with professor Kenzu Abdella. In those days I did almost all of my coding in Maple, which is basically Canada’s mathematica. I haven’t used it in 15 years or so, but I’d say it’s a functional language used mostly for symbolic manipulation and equation solving. I spent many late nights working in maple trying to do my year-end project for that course.

I was trying to replicate and hopefully improve the Lotka-Volterra predator prey population model. The idea of the model is that predator and prey population levels are dependent. An over-population of prey leads to higher levels of predators, which leads to a population crash in prey, then the predator population crashes. So both are cyclical, but there is a lag between the cycles. Honestly I remember that finding a data set of predator and prey populations was more challenging than the math or coding aspect. And I’ve found that finding good data is often the hardest part in a modelling exercise.

Because of my experience with this model, I decided to try my hand with knowledge from the video series on the Lotka- Volterra model with the techniques from the video.

First, here’s how I would do it before having watched the videos:

# started with the values from wikipedia for these
# then fiddled until I got something reasonable.
alpha = 0.0075
beta = 0.133
delta = 0.10
gamma = 0.10
x0 = 0.5
y0 = 0.1

def next_xy(x, y):
    return (x + alpha * x - beta * x * y, 
        y + delta * x * y - gamma * y)

Which gives this:

I’m not a population biologist or whatever, but I think this looks pretty nice.

And here’s how the problem would be solved using the technique from the videos:

def sim(variables, t, params):
    x, y = variables
    alpha, beta, delta, gamma = params
    dxdt = alpha * x - beta * x * y
    dydt = delta * x * y - gamma * y

    return([dxdt, dydt])
# I think the wikipedia values worked here.
alpha = 0.66
beta = 1.33
delta = 1.0
gamma = 1.0
params = (alpha, beta, delta, gamma)
t = np.linspace(0, 50, num=100)
y0 = [1,1]

y = odeint(sim, y0, t, args=(params,))

Which gives this plot:

I tried to do the stochastic model but I couldn’t get it working. I noticed that Mike Saint Antoine has a couple videos about Lotka-Volterra, so I’ll probably check those out at a later date and see what I was doing wrong.

Diabolical Answers In Wordle

Ben Wendt — Mon, 11 Apr 2022 20:00:00 -0400

I’ve blogged several times about wordle before. It’s a great game for someone like me who loves words. I don’t think I’m a great wordle player, but I love thinking about the intricacies of the game. One bugaboo in wordle, which I’ll call “diabolical answers” is when you reach a point where you have four letters correct and there are many possible answers. The worst of these would have more than six possible words, meaning you aren’t guaranteed a win even though you’re almost done.

I’ll start with an example. The word “soare”, which I blogged about earlier as being a great first guess, is also a diabolical answer, and this happened to me the other day. Assuming all but the second letter are correct, other possible answers include:

scare
share
slare
snare
spare
stare
sware

Admittedly some of these are rare and don’t occur in wordle’s dictionary.

I am still working on learning more Apache Beam, so I thought I would try to find these diabolical words using that tool. Here is the algorithm:

import apache_beam as beam
from wordfreq import word_frequency

# should probably use the actual wordle dictionary.
dictionary = "/usr/share/dict/words"

def is_lower(x):
    return x.lower() == x

def is_fives(text):
    return len(text) == 5

def to_missings(word):
    retval = []
    for i in range(len(word)):
        s = word + ""
        s = s[:i] + "_" + s[i + 1:]
        retval.append((s, word))
    return retval

def remove_rares(x):
    return word_frequency(x, 'en') > 1.0e-06

with beam.Pipeline() as p:
    fives = ( p |
        beam.io.ReadFromText(dictionary) |
        "is lower" >> beam.Filter(is_lower) |
        "is fives" >> beam.Filter(is_fives) |
        "is common" >> beam.Filter(remove_rares)
    )

    various_lists = (fives |
        "create keyed missings" >> beam.FlatMap(to_missings)
        | "unique those" >> beam.Distinct()
    )

    groups = (various_lists |
        "group by missing chars" >> beam.GroupByKey()
    )

    filtered_groups = (groups |
        "only baddies" >> beam.Filter(lambda x: len(x[1])> 5)
    )

    output = (filtered_groups |
        "hopefully found something" >> beam.Map(print)
    )

Note the use of the wordfreq library, one of my faves, to remove rare words, based on the heuristic threshold I decided on of 1.0e-06. This gives a really pleassing (to me at least) list of diabolical answers 😈:

_arry: barry, carry, harry, larry, marry, parry
_atch: batch, catch, hatch, latch, match, patch, watch
_erry: berry, derry, ferry, jerry, kerry, merry, perry, terry
_illy: billy, filly, hilly, milly, silly, willy
_itch: bitch, ditch, fitch, hitch, mitch, pitch, witch
_ound: bound, found, hound, mound, pound, round, sound, wound
_ater: cater, eater, hater, later, mater, water
_over: cover, dover, hover, lover, mover, rover
_olly: dolly, folly, holly, jolly, molly, polly
_ight: eight, fight, light, might, night, right, sight, tight, wight
gra_e: grace, grade, grape, grate, grave, graze
sha_e: shade, shake, shale, shame, shane, shape, share, shave
sta_e: stage, stake, stale, stare, state, stave

Some of these have 6 or more entries, which would make them very frustrating.

Messing Around In Beam, π Day, And More Wordle

Ben Wendt — Sun, 13 Mar 2022 20:00:00 -0400

Apache Beam is a distributed programming framework, mostly designed as a counterpart to the DataFlow service in Google Cloud Platform. In the past I’ve done a fair bit of work on pipelines in Spark and with a service architecture, but I’ll be needing Beam for my new job, so Ive been playing around a bit with that.

I’ve may have written a bit about it before, but I think of calculating π using the Monte Carlo Method to be kind of like the “hello world” of data pipelines. A quick review of the algorithm:

Select a large number of random x, y ∊ [0, 1]² from the uniform distibution.
Take the sum of squares of each couple.
The proportion of sums of squares that are less than one will (very slowly converge to π/4).

This works because finding random points withing this quarter arc is analagous to finding the area of the unit circle. Here’s the code in beam:

import apache_beam as beam
import numpy as np

def in_circle(pair):
    if (pair[0]**2 + pair[1]**2) < 1:
        return 1
    else:
        return 0

with beam.Pipeline() as pipeline:
    n_samp = 100_000
    X = np.random.uniform(0,1,2 * n_samp).reshape(n_samp, 2)

    x = pipeline | "create xs" >> beam.Create(X)

    number = x | "calculate it" >> beam.Map(in_circle) \
        | "sum up" >> beam.CombineGlobally(sum)

    number | "writing pi value" >> beam.io.WriteToText("number.txt")

This gave me a value of pi of π is 3.14136, which any piphile would tell you is way off. The Monte Carlo Method is great for a close enough answer, but it’s a terrible way to calculate digits of π.

I’ve also written recently about wordle. I’ve been playing every day for months, and I often get the urge to write a program to cheat for me. I have thus far resisted the urge to cheat, (unless you count writing a program to give a good first guess cheating), but I have gone back several times to re-think word choices.

The other day I had a clue of _o_us and I discounted several letters to get there. Off the top of my head I thought of “bolus” and “focus” as options. It ended up being “bonus,” but it took me a few minutes to remember this common word.

Here’s a beam that calculates some of the words that may match in this scenario.

import apache_beam as beam

# my favorite unix file
dictionary = "/usr/share/dict/words"

def to_lower(x):
    return x.lower()

def is_fives(text):
    return len(text) == 5

def match_greens(pattern):
    return lambda x: all([x[idx] == letter for idx, letter in pattern])

def non_matches(excludes):
    return lambda x: all([ex not in x for ex in excludes])

with open("beam-output.txt", "w") as f:
    with beam.Pipeline() as p:
        words = ( p |
            "read in words" >> beam.io.ReadFromText(dictionary) |
            "to lower" >> beam.Map(to_lower)
        )

        fives = (words |
            "filter to five letter words" >> beam.Filter(is_fives)
        )

        # sample of letters I had tried and discounted.
        excludes = ['t', 'r']

        without_exclusions = (fives |
            "remove exclusions" >> beam.Filter(non_matches(excludes))
        )

        # sample "green letter" pattern _o_us
        pattern = [(1, 'o'), (3, 'u'), (4, 's')]

        matches = (without_exclusions |
            "get green matches" >> beam.Filter(match_greens(pattern))
        )

        (matches 
            |"do output" >> beam.Map(print)
        )

And I get these output words with my dictionary:

bogus
bolus
bonus
cobus
comus
conus
copus
focus
fogus
hocus
kobus
locus
momus
mopus
nodus

As I mentioned before, wordle does use common words, so it would make sense to either filter or sort these based on word frequency. But as a quick learning exercise for beam it was decent. One aspect I am really enjoying about beam so far is that it basically forces you to document every step of your pipeline, because you can’t use the same operator twice without doing the "explanation" >> step thing, which is really nice.

BrainF Interpreter In Scratch

Ben Wendt — Wed, 09 Mar 2022 19:00:00 -0500

Since late 2012, I’ve been a parent. My oldest is an absolutely avid coder. He loves the scratch programming language, and has made an incredibly bevy of spectacular games, originally by following griffpatch tutorials, and then moving on to his own stuff.

As a way of sharing time with him, I do some scratch of my own. It’s a fun way of spending time together and encouraging his interest. Of course, as a professional software developer I like to explore what I can accomplish in a given environment. So I thought I would implement the esoteric langage BrainFrig in scratch.

BrainEff is a simple language, with only eight commands. From wikipedia:

> Increment the data pointer (to point to the next cell to the right).

< Decrement the data pointer (to point to the next cell to the left).

+ Increment (increase by one) the byte at the data pointer.

- Decrement (decrease by one) the byte at the data pointer.

. Output the byte at the data pointer.

, Accept one byte of input, storing its value in the byte at the data pointer.

[ If the byte at the data pointer is zero, then instead of moving the instruction pointer forward to the next command, jump it forward to the command after the matching ] command.

] If the byte at the data pointer is nonzero, then instead of moving the instruction pointer forward to the next command, jump it back to the command after the matching [ command.

BrainFart languages operate similar to a turing machine, with the idea of a single pointer that moves along a tape, making modifications and performing some basic conditional logic. Translating this into scratch means setting up a List variable, and a variable for the pointer.

Then initialize the memory:

We then need a program to operate on. I thought it would be cute to have Scratch Cat ask you for your program:

This is then read into the program variable:

Each unit of scratch code is called a block (they look a bit like lego blocks, and they stick together similarly). Each block is a blocking operation, that is, no further connected blocks are run until that block is finished. The ask block is no different in this regard. Scratch has two means of code separation, by message broadcasting, and by making custom blocks. A custom block is analagous to making a function.

After a program is entered, the new program message is broadcast, this will cause this code to run:

This code handle the “parsing and lexing” part of the interpreter, basically. Next is just the code that runs the functional parts of the input program, alters the memory, and creates output. Here are the parts for moving the pointer around on the tape:

Here’s the code for incrementing and decrementing the memory at the pointer location:

Here’s the IO code:

And finally, the tricky bit: the looping code.

This is the second time I’ve written a BrainFrag interpreter, and it’s been fun both times. BrainFrick is a funny language, as it’s probably a lot more common to write an interpreter for it than it is to write a program for it to run. I got a good chortle out of my coding companion when I showed him I had implemented another programming language in scratch, even if I had to bowdlerize the name of the language when I was talking about it.

Automatic Word of the Day

Ben Wendt — Wed, 02 Mar 2022 19:00:00 -0500

Since 2007 I’ve been keeping up my own personal word of the day blog. I started it back in the heyday of google reader. In those days I was subscribed to several word of the day blogs and wanted to do one of my own. I’ve gone through phases of posting more and less. Some years I posted every day, and toward the late teens I had single digits of posts for several years running.

It’s fun. My favorite words to add are ones that I come across while reading something, and I either really like the word or have to look it up. Words that arise organically have a personal touch that is appropriate for blogging.

But in the years where I posted every day, I mixed and matched between organic words and wikipedia crawling. I forget the exact criteria, but I had a userscript that I would run in a browser that would look for interesting words, those being wikipedia articles that matched several criteria:

Single word title, with optional parentheses.
No proper nouns
Probably some kind of length limit.
There were probably other criteria.

It would be interesting to share what I was doing with javascript in those days, but alas, whatever that user script did, it’s lost to the sands of time. Being totally honest I think it was on a work computer, and this was back in the days when a work computer was a big box that sat in an office, not a laptop in your own home.

My enthusiasm for the project waned over the years, but never left. So last year I was looking to revive the blog. I thought an interesting way to automate it would be picking random words from the dictionary with in a given word frequency range.

from wordfreq import word_frequency
from PyDictionary import PyDictionary
from english_words import english_words_set
from random import choice
from wiktionaryparser import WiktionaryParser
import wikipedia
from datetime import datetime

dictionary=PyDictionary()
english_words = list(english_words_set)
parser = WiktionaryParser()

def find_interesting_word(max_freq=5e-07, min_freq=7e-10):
    freq = 2 * max_freq
    while freq < min_freq or freq > max_freq:
        word = choice(english_words)
        freq = word_frequency(word, 'en')
    return word, freq

Here you see that an interesting word is one whose frequency lies in the range 7e-10 to 5e-07. I found these bounds by trial-and-error, and even so most of the output words aren’t great, so pick a bunch of them:

def find_interesting_words(num=7):
    return [find_interesting_word() for i in range(num)]

Example:

[('incubate', 2.88e-07),
 ('rattail', 3.24e-08),
 ('recondite', 5.37e-08),
 ('okra', 4.27e-07),
 ('headdress', 4.57e-07),
 ('Grosset', 8.13e-08),
 ('hyperbola', 1.07e-07)]

Aside from the proper noun “Grosset,” these are basically all suitable for “words of the day.” But a word of the day isn’t just a word, you also need a definition, and I liked to include a picture. I found a function that grabs images for a given query from wikipedia, and use PyDictionary and WiktionaryParser to give definitions:

def li(x):
  return f"<li><i>{x['partOfSpeech']}</i>: {' '.join(x['text'])}</li>"

def html_of(word):
    out_code = f'<p><b>' \
      '<a href="https://en.wiktionary.org/wiki/{word}">{word}</a>' \
      '</b></p>'
    meaning = dictionary.meaning(word)
    if meaning:
        for (typ, defs) in meaning.items():
            out_code += f'<p><i>{typ}</i>'
            lis = ''.join(['<li>' + x + '</li>' for x in defs])
            out_code += '<ul>' + lis + '</ul>'
            out_code += '</p>'
    else:
        meaning = parser.fetch(word)
        out_code += '<ul>'
        for defs in meaning:
            lis = [li(x) for x in defs['definitions']]
            out_code += ''.join(lis) + '</ul>'
        out_code += '</ul>'
    wikiimage = get_wiki_image(word)
    if wikiimage:
        out_code += f'<p><img width=720 rel="{word}" src="{wikiimage}" /></p>'
    return out_code

I see a few issues here, but this was meant as a proof-of-concept, so that’s fine. After running a few tests, I found I was getting results like this:

europium

Noun a bivalent and trivalent metallic element of the rare earth group

This is fine, but I feel it’s a bit soul-less. Had this worked out I would have liked to add blogger API access, including scheduling and tagging, but I wasn’t happy enough with the results. After reflecting on what I wanted the blog to be, I decided to just make a better effort of manually posting things, and I’ve been doing a much better job since last year.

What is Canada's Largest Transitless Municipality?

Ben Wendt — Mon, 21 Feb 2022 19:00:00 -0500

Happy 2s Day.

I was chatting with my brother the other day, and he opined that I may live in Canada’s largest municipality that doesn’t have a municipal bus transit system. Of course I was interested to find out which is the biggest Canadian city without buses, so here we go.

I was very pleased to find that pandas will read an html page and output a list of dataframes that it was able to parse from tables. Pair this with wikipedia keeping a list of Canadian municipalities with public transport and we have a good start:

import pandas as pd
import requests

transit_url = "https://en.wikipedia.org/wiki/Public_transport_in_Canada"
r = requests.get(transit_url)
df_list = pd.read_html(r.text)
transit_municipalities = df_list[0]

	Name	Municipalstatus[3][6]	County[15]	Incorporationyear[16]	2021 Census of Population[15]
	Name	Municipalstatus[3][6]	County[15]	Incorporationyear[16]	Population(2021)	Population(2016)	Change	Land area(km²)	Populationdensity
0	Charlottetown	City	Queens	1855	38809	36094	+7.5%	44.27	2.0
1	Summerside	City	Prince	1877[c]	16001	14839	+7.8%	28.21	2.0
2	Alberton	Town	Prince	1913	1301	1145	+13.6%	4.70	2.0
3	Borden-Carleton	Town	Prince	1995[d]	788	724	+8.8%	12.94	2.0
4	Cornwall	Town	Queens	1995	6574	5348	+22.9%	28.21	2.0

Next, use the same technique to find a list of municipalities in each province.


provinces = ['British Columbia', 'Alberta', 'Saskatchewan', 'Manitoba', 'Ontario', 'Quebec',
            'New Brunswick', 'Newfoundland and Labrador', 'Nova Scotia', 'Prince Edward Island']

p_dfs = {}
def p_url(p):
    return f"https://en.wikipedia.org/wiki/List_of_municipalities_in_{p.replace(' ', '_')}"

for province in provinces:
    print(f"getting municipalities for {province}")
    r = requests.get(p_url(province))
    df_list = pd.read_html(r.text)

    if province == 'Ontario' or province == 'Manitoba': # different html for these two.
        p_dfs[province] = df_list[1]
    else:
        p_dfs[province] = df_list[0]
    print(f"{province} has {len(p_dfs[province])} municipalities")

Clean that data up a bit, and work around the annoyingness of MultiIndex:

canada_dfs = []

def find_series_by_column(df, look_for):
    for x in df.columns:
        if isinstance(x, tuple):
            if look_for in x[1]:
                pop_column = x[0]
                return df[x[0]][x[1]]
                break


for province, df in p_dfs.items():
    prov_series = pd.Series([province for x in range(len(df.index))])
    pop_series = find_series_by_column(df, 'Population')
    name_series = find_series_by_column(df, 'Name')
    canada_dfs.append(pd.DataFrame({"Province": prov_series, "Name": name_series, "Population": pop_series}))
canada_df = pd.concat(canada_dfs)

Gives data like this:

	Province	Name	Population
378	Ontario	Timmins	41788
890	Quebec	Saint-Thomas	3249
145	Saskatchewan	Tisdale	2962.0
827	Quebec	Saint-Patrice-de-Sherrington	1960
45	British Columbia	Terrace	12017
41	Manitoba	Louise	2025
806	Quebec	Saint-Modeste	1162
88	Saskatchewan	Langenburg	1228.0
395	Quebec	Mont-Tremblant	9646
52	Saskatchewan	Davidson	1044.0

Do an anti-join to find the muncipalities without transit:

with_transit_df = canada_df.merge(trans_df, on=['Name', 'Province'], indicator=True, how='left')
without_transit_df = with_transit_df.loc[with_transit_df._merge == 'left_only', :].drop(columns='_merge')

A little data cleanup:

import re
without_transit_df['Population2'] = without_transit_df['Population'].fillna("0").map(lambda x:
        str(re.sub("\[\d+\]", "", str(x)))
     )
without_transit_df['Population2'] = without_transit_df['Population2'].map(
        lambda x: str(x).replace(".0", "").replace("nan", "0").replace(",", "")
    ).replace('', 0).astype('int32')

And sort:

	Province	Name	Population2
47	British Columbia	Vancouver	662248
44	British Columbia	Surrey	568322
1186	Ontario	Markham	328966
1356	Ontario	Vaughan	306233
2	British Columbia	Burnaby	249125
1156	Ontario	Kitchener	233222
41	British Columbia	Richmond	209937
1286	Ontario	Richmond Hill	195022
1251	Ontario	Oshawa	159458
7	British Columbia	Coquitlam	148625
19	British Columbia	Kelowna	144576
69	British Columbia	Langley	132603
1025	Ontario	Cambridge	129920
1371	Ontario	Whitby	128377
972	Ontario	Ajax	119677
85	British Columbia	Saanich	117735
2437	Quebec	Terrebonne	111575
2596	Newfoundland and Labrador	St. John's	108860
11	British Columbia	Delta	108455
1360	Ontario	Waterloo	104986
2872	Nova Scotia	Cape Breton	94285
1047	Ontario	Clarington	92013
49	British Columbia	Victoria	91867
1268	Ontario	Pickering	91771
23	British Columbia	Maple Ridge	90990
78	British Columbia	North Vancouver	88168
1457	Quebec	Brossard	85721
1886	Quebec	Repentigny	84285
1221	Ontario	Newmarket	84224
2496	New Brunswick	Moncton	79470
28	British Columbia	New Westminster	78916
1145	Ontario	Kawartha Lakes	75423
2497	New Brunswick	Saint John	69895
1022	Ontario	Caledon	66502
1226	Ontario	Norfolk	64044
34	British Columbia	Port Coquitlam	61498
1113	Ontario	Halton Hills	61161
29	British Columbia	North Vancouver	58120
1436	Quebec	Blainville	56863
990	Ontario	Aurora	55445
1766	Quebec	Mirabel	50513
1526	Quebec	Dollard-des-Ormeaux	48899
2881	Nova Scotia	Kings[g]	47404
1751	Quebec	Mascouche	46692
21	British Columbia	Langford	46584
2472	Quebec	Victoriaville	46130
1372	Ontario	Whitchurch-Stouffville	45837
1112	Ontario	Haldimand	45608
1097	Ontario	Georgina	45418

This really isn’t a satisfactory answer for me to be honest. Most of the entries on this list were either missed in the join (e.g. Metro Vancouver not matching Vancouver, similar issue for York Region), or the municipalities listed are part of a larger municipality that is served by public transit (e.g. Markham). Fixing those issues would be possible, but a lot of work. It was time for the paretto principle. So I looked through the list. This isn’t definitive but I think that Kawartha Lakes, Ontario, (population ~75000), is the largest municipality in Canada without a municipal bus service. They do have a municipal service with these little half-bus, half-van things though, so maybe that counts.

"Infinite Word Ladder"

Ben Wendt — Wed, 16 Feb 2022 19:00:00 -0500

One of the paper handouts my child used to do at school was the “word ladder.” The exercise is a starting word and a series of instructions, with a hint/definition of the word that will be created.

For example, given the starting word “jump”, a sequence like this could be created:

Replace first letter: get rid of
Replace last letter: bereft of speech.
Replace first vowel, add er at end: wetter.

And then the obvious answer is “damper”.

Given the ongoing waves of home / remote schooling we have as a result of the covid pandemic, parents will often find themselves looking for work- sheets for their ids to do. So I decided to write some code that could make these word ladders.

Firstly, I needed to decide what words would be valid for the puzzles. Clearly we don’t want words that are above grade level that the kid wouldn’t have heard before. So I won’t be giving “clonality” as an answer in the excercise. My first though was to use word frequencies, under the assumption that kids would know the most common words. Unfortunately this assumption did not bear out. There are a lot of commonly used words that I wouldn’t expect children to know.

Thankfully some kind folks already realized this and released a dataset of words kids should know. It’s called the Dale-Chall Word List.

The Dale-Chall Word List contains approximately three thousand familiar words that are known in reading by at least 80 percent of the children in Grade 5. It gives a significant correlation with reading difficulty. It is not intended as a list of the most important words for children or adults. It includes words that are relatively unimportant and excludes some important ones.

So with this dictionary I just needed to write a few lines of code to enact the logic of the word puzzle.

First, start with a data class to hold the “rungs” on the ladder:

from collections import namedtuple
Rung = namedtuple("Rung", "word description instructions")

I needed definitions of the words. This is the worst part of the result, as the definitions are very dictionary-like.

from PyDictionary import PyDictionary
dictionary=PyDictionary()
def dictionary_meaning(word):
    defn = dictionary.meaning(word)
    if defn is None:
        print(f'no definition for {word}')
        return ''
    items = list(defn.items())
    if len(items) == 0:
        print(f'no definition for {word}')
        return ''
    return items[0][1][0]
    return ' '.join(items)

A method of deciding which words are valid:

def okay_word(word, used_words={}):
    return (word in dale_chall) and (word not in used_words) and ("'" not in word)

And now the fun part, the method with the list of available edits that finds new rungs to put in the ladder:

def edit_word(word, used_words={}):
    edits = []
    for letter in string.ascii_lowercase:
        if okay_word(letter + word, used_words):
            edits.append((letter + word, 'add letter at start'))
        # don't want to just pluralize.
        if letter != "s" and okay_word(word + letter, used_words):
            edits.append((word + letter, 'add letter at end'))
        if okay_word(letter + word[1:], used_words):
            edits.append((letter + word[1:], 'change first letter'))
        if okay_word(word[:-1] + letter, used_words):
            edits.append((word[:-1] + letter, 'change last letter'))
    for letter1 in string.ascii_lowercase:
        if letter1 == word[0]:
            continue
        for letter2 in string.ascii_lowercase:
            if letter2 == word[-1]:
                continue
            new_word = letter1 + word[1:-1] + letter2
            if okay_word(new_word):
                edits.append((new_word, 'change first and last letter'))

    for pair in first_twos:
        if pair[0] == word[0] and pair[1] == word[1]:
            continue
        if pair[0] == word[0] and pair[1] != word[1] and okay_word(pair + word[2:]):
            edits.append((pair + word[2:], 'change second letter'))
        elif pair[0] != word[0] and pair[1] == word[1]:
            # already checked first letter change above.
            continue
        elif okay_word(pair + word[2:], used_words):
            edits.append((pair + word[2:], 'change first two letters'))
    for vowel_source in ['a', 'e', 'i', 'o', 'u']:
        for vowel_dest in ['a', 'e', 'i', 'o', 'u']:
            vowel_replaced = word.replace(vowel_source, vowel_dest, 1)
            if okay_word(vowel_replaced, used_words):
                edits.append((vowel_replaced, 'change first vowel'))

    edits = [(w,d) for (w,d) in edits if w != word]
    return edits

I like this and I think it might be fun to try to add new edits.

Then just a couple methods to glue this crap together and make a ladder:

def find_next_rung(rung, used_words={}):
    edits = edit_word(rung.word, used_words)
    if len(edits) == 0:
        print(f"failed to find a suitable word based on {rung.word}. :(")
        return None
    found_word, found_instruction = random.choice(edits)

    return Rung(found_word, dictionary_meaning(found_word), found_instruction)

def get_ladder(word):
    ladder = [Rung(word, '', '')]
    used_words = {ladder[0].word}

    for i in range(100):
        next_rung = find_next_rung(ladder[len(ladder) - 1], used_words)
        if next_rung is None:
            print('breaking')
            break
        used_words.add(next_rung.word)
        ladder.append(next_rung)
    return ladder

And here’s an example output:

[Rung(word='shoe', description='', instructions=''),
 Rung(word='shop', description='a mercantile establishment for the retail sale of goods or services', instructions='change last letter'),
 Rung(word='show', description='the act of publicly exhibiting or entertaining', instructions='change last letter'),
 Rung(word='meow', description='the sound made by a cat (or any sound resembling this', instructions='change first two letters'),
 Rung(word='glow', description='an alert and refreshed state', instructions='change first two letters'),
 Rung(word='know', description='the fact of being aware of information that is known to few people', instructions='change first two letters'),
 Rung(word='grow', description='pass into a condition gradually, take on a specific property or attribute; become', instructions='change first two letters'),
 Rung(word='flow', description='the motion characteristic of fluids (liquids or gases', instructions='change first two letters'),
 Rung(word='snow', description='precipitation falling from clouds in the form of ice crystals', instructions='change first two letters'),
 Rung(word='plow', description='a farm tool having one or more heavy blades to break the soil and cut a furrow prior to sowing', instructions='change first two letters'),
 Rung(word='crow', description='black birds having a raucous call', instructions='change first two letters'),
 Rung(word='crown', description='the Crown (or the reigning monarch', instructions='add letter at end'),
 Rung(word='frown', description='a facial expression of dislike or displeasure', instructions='change first letter'),
 Rung(word='brown', description='an orange of low brightness and saturation', instructions='change first letter'),
 Rung(word='clown', description='a rude or vulgar fool', instructions='change first two letters'),
 Rung(word='drown', description='cover completely or make imperceptible', instructions='change first two letters'),
 Rung(word='known', description='be cognizant or aware of a fact or a specific piece of information; possess knowledge or information about', instructions='change first two letters')]

I Did a Deep Q Learning Course

Ben Wendt — Thu, 10 Feb 2022 19:00:00 -0500

I had access to udemy courses back when I was working at Deloitte Digital. One interesting course I did was the introduction to deep q learning from Pieran Data. Deep Q learning is a way to use neural nets to do reinforcement learning. It was an interesting course and I learned a lot, so I’ll give a recap here.

Classical Q learning is a reinforcement learning algorithm that revolves around making a table of all possible states and actions, and gives an expected reward for each of these combinations. For problems with a continuous state or action space, the states and choices are discretized by bucketing (and bucket distribution is a hyperparameter).

The table is made iteratively. At first it’s filled with zeroes, as no rewards are known, and choices are made at random. Expected reward values are updated using the Bellman Equation. As expected reward values are updated, the randomness of action choices is reduced to fine-tune the reward values (this is called the Epsilon Greedy Strategy).

Classic Q learning is great for a certain class of reinforcement learning problems. Here’s an example we did in the course of classic q learning beating the “hill car challenge.”:

Sorry, your browser doesn’t support embedded videos.

This works great but most realistic problems have too large a space of states and actions, even after bucketing. The key insight of deep Q learning is that similar states will lead to similar actions, so we can model the states and actions using neural networks to make it a tractable problem. It’s been almost a year since I covered this materal , and I feel pretty rusty so I won’t get too deep into it, but here’s an example we did in the course of getting an AI that can play Atari games:

Sorry, your browser doesn’t support embedded videos.

Overall it’s a very cool technique but I doubt I’ll ever get much use out of it.

3D Printed Rocket

Ben Wendt — Mon, 24 Jan 2022 19:00:00 -0500

My old work (TWG) had a 3D printer for employees to play with and use. In the summer of 2019 my son was getting into model rocket launching, but we kept losing them in trees. So I decided to try 3d printing some rockets. It never really worked out, but in my new spirit of blogging failures, I’m posting about it. My goal was to make my own easily produced reusable rocket.

I am not a rocket scienctist or enginer, but I’ve played a fair bit of Kerbal Space Program and watched a lot of Scott Manley videos. I didn’t really incorporate any proper rocket design into my ideas here. I designed the rockets with these over-sized fins to facilitate 3d printing, by having a larger contact area on the printing surface.

In terms of “easily produced” I wanted something I could easily print at work, i.e. I could fire something up in a few minutes and have it finish within a morning or afternoon so I wasn’t wasting a lot of time at work doing this.

Unfortunately the launch wasn’t a success. It would launch, fly hundreds of feet into the air, then blow up:

The issue arises from how a model rocket engine works. After ignition, the engine burns it’s propellant, launching the rocket high into the sky. After this is burned up, a time delaying, non-propelling burn occurs, to allow the rocket to use its forward momentum up before deploying the parachute at apogee. Once the time delay is used up, a smaller charge blows upward into the rocket’s body, pushing off the nose cone and the chute. This is shown in the images below:

Image source

My problem was the my single-walled rocket couldn’t handle the pressure of the chute charge, so the fuselage would rupture. They were all blowing up at step 4 in the launch sequence image above. I think this is slightly compounded by the 3d print ridges adding more friction to the nose cone removal, but the weak body is probably the main issue.

I tried iterating on the design by taping the outside of the fuselage to reinforce it (still not strong enough), and having a wider diameter of fuselage to reduce internal pressure (also not strong enough). The real solution is probably one of: print a thicker fuselage, (but this would have taken too long), or to slide a paper tube into the fuselage to handle the pressure, (but I didn’t want to source those). I shelved the project having run out of desirable alternatives.

The Best And Worst Words In Wordle

Ben Wendt — Mon, 17 Jan 2022 19:00:00 -0500

Wordle is a popular word deduction game. The aim is to guess a word given six guesses. You can only guess valid English words that are in wordle’s dictionary. Each wrong guess will yield hints. A letter highlighted in yellow indicates it matches the target word somewhere. A letter marked in green indicates the correct letter in the correct position. I feel like a branching strategy (preferably choosing from common letters) is the best. But what word should you start with? And what is the worst word to start with?

Using the timeless magic of “inspect element” I grabbed the complete dictionary of 10657 words wordle considers to be valid (compare this to 10230 in my /usr/share/dict/words on mac). Wordle’s word list is pretty liberal with what it considers to be a valid English word, containing things like:

So what is the best and worst word to start? The rules of the game mean that the first guess will be best when it draws from the most common letters in the dictionary. So what are the most common letters in wordle’s set of five letter words? Here’s the distribution:

We can see here that z, j, x, q are the least commonly used letters, but words also need vowels, and y is the least common vowel. Unfortunately, no words use only there letters. Through trial and error of adding more commonly used letters, I found the word xylyl, which draws mostly from uncommon letters, but I wanted to be a bit more sure.

Here’s a couple methods that calculate two words “yellow” and “green” scores:

def yellow_matches(w, x):
    return len(set(w).intersection(x))

def green_matches(w, x):
    return len([z for (y, z) in zip(w, x) if y == z])

Using this we can check each words score against all other words. Sorting by yellows, then greens gives these best starting words:

word	yellow	green
aeros	20698	7505
soare	20698	5610
reais	20402	7482
serai	20402	4800
aesir	20402	3618
aloes	20092	7774
toeas	19992	8021
stoae	19992	4119
lares	19926	9414
rales	19926	9149

Or sorted by greens, then yellows:

word	yellow	green
sores	16277	9982
sanes	16625	9914
sales	16854	9825
sones	15442	9772
soles	15671	9683
bares	18712	9661
cares	18932	9649
pares	18999	9642
sates	16754	9594
tares	19826	9591

So it looks like “aeros” and “sores” are good starting words. I don’t think the actual winning words ever end with ‘s’, so let’s look at those options. Most yellows:

word	yellow	green
soare	20698	5610
serai	20402	4800
aesir	20402	3618
stoae	19992	4119
laser	19926	5371
seral	19926	4851
taser	19826	5548
strae	19826	4405
earst	19826	4044
resat	19826	3747

Most greens:

word	yellow	green
saree	17460	7300
soree	16277	7158
sared	19388	6907
sored	18205	6765
sooey	14819	6711
saned	18553	6697
boree	12211	6695
raree	12142	6624
laree	14608	6590
samey	17572	6554

And now let’s look at the worst starting options:

word	yellow	green
xylyl	4330	1425
fuffy	4376	2751
gyppy	4396	2382
hyphy	4482	2118
cocco	4710	3467

Those look like some terrible options to start with, but today I started with “aeros”, and the correct answer was “proxy”. I think I would have been better off starting with the “xy” from “xylyl” as those are higher value letters.

Using an Artie 3000 as a Plotter

Ben Wendt — Tue, 28 Dec 2021 19:00:00 -0500

When I was a child, they had a plotter at my mom’s office. I don’t remember the model, but it was something like a HP 7550A. I was fascinated by it, and I’m sure I watched it go for many hours. From that, I’ve always loved plotter technology. I’ve always kind of wanted to make my own. Last year, the urge became too great and I tried out the branchiograph project. I didn’t get the best results for various reasons.

This Christmas, we got my son an Artie 3000 robot. It’s a python-codable (more on this) turtle-graphics-like robot that uses a pen to draw on paper. I decided to try to turn it into a plotter.

So my first let-down on this project is that, although python is supported, it’s only supported within the web interface you use to interact with the robot. There’s no public library to pip-install that lets you interact with the robot. So even though the box says “code in python” with a little screenshot showing an import command, it’s not really that easy. I thought about inspecting the web-socket traffic that controls the robot, but I thought I would try generating some python code for the robot that should draw something interesting before going to that effort.

I started by finding a decent image to try to draw. My obvious choice was Mario. I love trying to impress my kids, so I picked Mario. My son is nuts about Mario. I can’t blame him.

My first thought of row to take an rgb image and prepare it for printing in monochrome was just to do a threshold, but I quickly decided that wouldn’t look good at all. Thankfully I was alive in the 90s, and I remembered how people used to try to make 8-bit images look good: dithering! If it works for 8 bits, why not 1? The low-tech magazine uses the same 1-bit dithering technique to reduce their image sizes.

Here’s what Mario looks like after resizing and dithering to 1-bit:

I wrote some fairly trivial code to convert this row-wise into a series of turtle commands. Here is my first result:

I love this cool spiral look, but there’s a clear problem here. You can see that there’s a big spiral. This is because and the end of each row, the robot turns left, moves to the next row, and turns back right. But it has a slight preference to the left, about 4 degrees in my tests. My image was 70 rows, so that means the robot would have turned 210 degrees by the time it finished printing.

I ran some tests to manually correct for the turning bias, and got more results:

This is two prints over one-another. You can make out Mario’s hat, but the line pitch is all wrong.

I got a bit discouraged at this point because it seems pretty clear that the robot has a minimum distance it will go forward after turning. I’m not sure why. I experimented with getting it to move forward too far, and backing up to get the desired line pitch, but I couldn’t get it lined up right. It’s probably not insurmountable but I think I may have reached my personal point where this project ceased being fun. Projects that are given up on mid-way can be noteworthy too, so I thought I would put up a post.

Finding Latent Faces-with Non-negative Matrix Factorization

Ben Wendt — Mon, 01 Nov 2021 20:00:00 -0400

Non-negative matrix factorization is a fast technique for generating embeddings from a dataset. More concretely, given a matrix you can decompose it into two matrices that are approximattely multiplicands of the matrix. I.e. given matrix M, it finds matrices A and B such that M ≈ AB.

A very cool thing about embedding matrices is that the rows and columns that comprise these matrices naturally tend to fill with values that will multiply together to create the values in your data. More concretely again, if your dataset is pictures of faces, and embedding matrix will be composed of face-like images that will be added together in varying amounts to make the faces in the dataset.

So let’s make some faces.

Thankfully sklearn.decomposition has an NMF class. We can use that to do all the tough work here. I’ll be using the ORL images dataset for this. This is a classic greyscale faces dataset that’s been used in computer vision for decades.

Here’s an example from the dataset:

source

Here’s the code to load all the images into a big numpy array:

    images = []
    for file in os.listdir('archive'):
        img = np.array(Image.open('archive/' + file))
        if len(img.shape) == 3: # some images come in rgb.
            img = img[:, :, 0] # not really desaturation.
        images.append(img.flatten())
    images = np.array(images)

Let’s make 20 latent faces:

    k = 20
    nmf_model = NMF(n_components=k, random_state=42)

    nmf_model.fit(images)

Now we can decompose the embeddings to see what the latent faces look like:

    images = []
    i = 0
    for image in nmf_model.components_:
        image = image.flatten()
        image.resize((80, 70))
        images.append(image)
        plt.figure()
        plt.savefig(f'imgs/{i}.png')
        i += 1

And here’s the cool generated face image:

You can see that the faces aren’t entirely novel, and have characteristics of faces from the dataset, such as glasses. Overall it’s a pretty cool technique that shows an interesting aspect of NNMF and embeddings. I tend to work with embeddings a lot more with NLP in my work, so it’s great to have an eye-catching and visual demo of what the technique does.

A "100 emoji" maker

Ben Wendt — Sun, 31 Oct 2021 20:00:00 -0400

The 100 emoji is iconic; it’s a symbol recognized across cultures and it has a semiotic impact globally. In a world where most societies have standardized around a base-10 (or decimal) number system, a doubly-underlined 100 is a shining beacon of success, or total mastery. The forthright and striking symbol of flawless execution, total knowledge, and success lends itself naturally, in this remix culture we live in, to imitations. If 100 can be thus underlined, can we convey the same messages by borrowing the style of the emoji but with different content? Of course we can, that’s what cultures do.

So let’s whip up a 100 emoji generator. I do most of my f-around work in python in a notebook these days, for several reasons:

I do most of my day job in python,
Libraries. As much as I love mucking around in elixir or whatever you can’t beat having numpy and PIL for something like this.
Jupyter: not having to duck in and out of terminal and preview windows tightens up the cognitive loop of development. You arguably end up with shittier code but who really cares for something like this. It’s better to finish a stupid project than get bogged down with irrelevancies and decide it’s not worth the effort.

So here’s a method that makes 100 emoji:

    # font from https://www.dafont.com/captain-redemption.font?text=100+365&back=theme
    # copied some stuff from https://stackoverflow.com/a/63005869/973810
    # reference https://stackoverflow.com/questions/17056209/python-pil-affine-transformation

    from PIL import Image, ImageFont, ImageDraw, ImageFilter
    import numpy as np

    def make_image(number, w=100, h=100):

        fill = (222, 0, 0) # a nice red
        img = Image.new('RGB', (2*w, 2*h), (255, 255, 255))

        # found this free for personal use font. It't more a 300 vibe, but close enough.
        font_filename = 'Captain Redemption.ttf'
        font_size = 0
        number = str(number)

        # find a good font-size to fit the width
        for i in range(1, w):
            font = ImageFont.truetype (font_filename, i)
            x, y = font.getsize(number)
            if x > 0.9 * w:
                font_size = i
                break
        draw = ImageDraw.Draw(img)
        i = 0
        # do the underlines
        for y in range(int(2*0.53*h), int(2*0.58*h)):
            draw.arc([(int(2*-0.10*w), y), (int(2*1.60*w), int(y + 2*0.20*w))], start=230, end=280-i, fill=fill)
            i += 2
        i = 0
        for y in range(int(2*0.62*h), int(2*0.67*h)):
            draw.arc([(int(2*-0.10*w), y), (int(2*1.60*w), int(y + 2*0.20*w))], start=233, end=274-i, fill=fill)
            i += 2
        # Use supersampling to achieve anti-aliased underlines
        img = img.resize((w, h), Image.ANTIALIAS)
        draw = ImageDraw.Draw(img)        
        font = ImageFont.truetype (font_filename, font_size)
        x, y = font.getsize(number)

        draw.text((0, 0), text=number, font=font, fill=fill)

        draw = ImageDraw.Draw(img)
        width = int(w/17.5)

        # warp/sheer the image with more height on the left.
        img = img.transform((w, h),
                        Image.AFFINE, (1, -0.1, -3, 0.1, 0.62, 5),
                        resample=Image.BICUBIC, fillcolor=(255,255,255))
        return img

I like to finish off my blog posts with a list of things I would like to improve, so here goes:

The first character should be taller, but not rendered with a larger font as that would make it wider. This goes beyond the capabilities of true type fonts.
It doesn’t work for longer words.
Related to point 1, a lot of the “math” in the method was trial-and-error. It would be a lot better to have a system of drawing different elements in their own areas. In the 100 emoji these are polygonal, which would get quite complex so I didn’t bother for a joke project.

Fibonacci Spheres in Blender

Ben Wendt — Mon, 31 Aug 2020 20:00:00 -0400

I’ve been messing about with blender a bit since the pandemic started. I originally thought I could use it for visualizing data, which is certainly true, but I’ve found more that I’ve been getting in blender-world. I was watching a “coding adevntures” video over the weekend where the author described using fibonacci spheres as a way of evenly distibuting points on a sphere. But I couldn’t find an easy way to do this in blender.

So I made an extension. It’s mostly cobbled together from code snippets I found on-line. You can check it out on my git hub.

Archiving photos from iOS to a network drive

Ben Wendt — Fri, 31 Jul 2020 20:00:00 -0400

I wrote a while back about a script I wrote to run on my android phone that would sort and archive any photos I had on the device into dated folders I have on a network drive. Privacy and de-googling concerns led me to get an iphone when the time came to upgrade (essential, my old phone’s manufacturer shut down, and faced with an aging battery and a looming lack of support I decided to upgrade). This left me with a problem, how would I sort my images into my archive?

I’ve found it a bit harder to do fun hacks like the photo sorting hack on iOs as compared to android. My new work flow is:

import (and delete) everything into i photo.
export everything from there to a folder on my laptop.
run this script:

    require 'set'

    class Copier

        def initialize(path)
            @path = path
            @created_folders = [].to_set
            @path = ''
            @userhost = 'network-host'
            @destpath = '/mnt/big-disk/photos'
        end

        def get_date(file)
            # some say Modified, other Modification.
            mod_date = `exiftool "#{file}" | grep 'Modif'`
            mod_date = mod_date.split("\n")[0] # first line
                            .split(" : ")[1] # the date time part
                            .split(" ")[0] # the date part
            mod_date.split(':')
        end

        def create_folder_if_need_be(date)
            year, month, day = date
            unless @created_folders.include?(date)
                puts "creating folder for #{date.join('-')}"
                `ssh #{@userhost} "mkdir #{@destpath}/#{year}" 2> /dev/null`
                `ssh #{@userhost} "mkdir #{@destpath}/#{year}/#{month}" 2> /dev/null`
                `ssh #{@userhost} "mkdir #{@destpath}/#{year}/#{month}/#{day}" 2> /dev/null`
                @created_folders << date
            end
        end

        def transfer_file_to_dated_folder(file, date)
            year, month, day = date
            `scp "#{file}" #{@userhost}:#{@destpath}/#{year}/#{month}/#{day}`
            puts "transferred #{file}"
        end

        def delete_local_file(file)
            `rm "#{file}"`
        end
    end

    path = "./photos/"
    files = Dir["#{path}*"].sort_by(&:downcase)
    c = Copier.new(path)
    files.each do |file|
        date = c.get_date(file)
        c.create_folder_if_need_be(date)
        c.transfer_file_to_dated_folder(file, date)
        c.delete_local_file(file)
    end

Looking at this now, it would be great to move the core of the files.each block into the class, and read the path and host from argv but it works so I’m happy with it.

Uploading and sorting photos from android over sftp using termux

Ben Wendt — Mon, 30 Sep 2019 20:00:00 -0400

I was super excited to find that I can install bash on my phone, then scp and ruby. It opens up a lot of possibility of what a phone should really be able to do. Too often I feel constrained by limited options in apps.

An example of this is taking old photos and putting them into my long term storage on my home computer. I backup to google photos, but I like to keep a local copy too. My old workflow for organizing files was to either use usb or “andftp” app on my android to get the files onto my machine, then sort them roughly by date into folders for ease of retrieval. Both of these methods are time consuming and don’t give the best result.

Here is my new work flow…

    userhost = 'someguy@192.168.0.100'
    destpath = '/mnt/big/camera'

    ['PIC', 'VID'].each do |type|
      transer_files(type)
    end

    def transfer_files(type)
      fs = ` ls storage/dcim/camera/#{t}* `.split("\n")

      prefixess = fs.map {|f| f.gsub(/.+#{t}_/, '').gsub(/_.+/,'')}.uniq

      years = prefixes.map{|p| p.slice(0,4)}.uniq

      prefixes.each do |p|
        handle_date_prefix(p)
      end
    end

    def handle_date_prefix(p)
      year = p.slice(0,4)

      mth = p.slice(4,2)
      day = p.slice(6,2)


      prefs.each do |p|
        transfer(year, mth, day)
      end
    end

    def transfer(year, mth, day)
     `ssh #{userhost} "mkdir #{destpath}/#{year}" `
     `ssh #{userhost} "mkdir #{destpath}/#{year}/#{mth}" `
     `ssh #{userhost} "mkdir #{destpath}/#{year}/#{mth}/#{day}" `

      c= `scp storage/dcim/camera/#{t}_#{p}* #{user_host}:#{destpath}m/#{year}/#{mth}/#{day} && rm storage/dcim/camera/#{t}_#{p}*`
      puts c
    end

This nicely organizes all my files by date and clears out the space on my phone.

Fizzbuzz in Different Paradigms / Techniques

Ben Wendt — Mon, 31 Dec 2018 19:00:00 -0500

I was reading an article on Fizzbuzz. Across the various languages used, the article has two recurring techniques. For each technique listed here, I will give it a rating on a scale of my choosing. The two techniques listed in the article are:

1. conditional logic

This is the standard technique that everyone, regardless of experience, would probably think of first. The following example will run in every c inspired language known to humankind (actually, unless otherwise noted, all code is js).

    for (i = 0 ; i < 33 ; i++) {
      if (i % 15 == 0) {
        console.log('fizzbuzz')
      } else if (i % 5 == 0) {
        console.log('fizz')
      } else if (i % 3 == 0) {
        console.log('buzz')
      } else {
        console.log(i)
      } 
    }

understandableness factor: ⭐⭐⭐

The cool-ness here is entirely due to being immediately understandable by anyone who has had an intro to programming.

2. pattern match

Fancy (i.e. “functional”) languages do a fancy thing called pattern matching, which is a lot like coding a conditional, but you let the compiler / runtime / interpreter determine when a variable matches a certain pattern rather than checking whether a certain condition is true. So instead if saying, if x is like this, do that, you would say when x matches this, do that.

Here’s an example in elixir because I am not smart enough for haskell:

    defmodule FizzBuzz do
        def run(x) when rem(x, 15) == 0 do
            IO.puts("fizzbuzz")
        end
        def run(x) when rem(x, 5) == 0 do
            IO.puts("fizz")
        end
        def run(x) when rem(x, 3) == 0 do
            IO.puts("buzz")
        end
        def run(x) do
            IO.puts(x)
        end
    end
    Enum.each(0..31, fn(x)-> 
        FizzBuzz.run(x)
    end)

cool factor: ⭐⭐⭐⭐

This is a built in language feature, but pattern matches are definitely more interesting than condtionals.

Feeling a little let down.

Given that there are ten languages in the article and only two techniques, I was disappointed. I’ve done a lot of “write x-in-y” posts before, but I feel like there isn’t much point in writing the same code in ten different syntaxes. I thought it would be cool to see what other techniques could be devised.

0. array programming

I googled how someone would do this in an array programming language, and found a terrific article explaining all about how to do so in j. The idea is to filter the list by which numbers match which modulus, and then do a lookup into an array to get the correct answer.

cool factor: ⭐⭐⭐⭐

Anything in j is super cool.

1. exception handling

They tell you not to use exception handling for control flow. But everyone keeps doing it, so it must be good. There’s one if statement in there that I would love to get rid of, but it would be difficult without jumping through more hoops than this jokey article deserves.

    fbs = [[15, 'fizzbuzz'], [5, 'fizz'], [3, 'buzz']]
    for (i = 0 ; i < 16; i++) {
        fails = 0
      try {
        for (e of fbs) {

          x = e[0]
          msg = e[1]
          a = [() => console.log(msg)]
          try {
            a[i % x]()
            break
          } catch (e) {
            fails++
            if (fails > 2) {
              throw e
            }
          }
        }
      } catch (e) {
        console.log(i);
      }
    }

kludge factor: 🦃🦃🦃🦃🦃

I give this one five kludge turkeys. Two for using the exception control flow thing, and three for calling the function in an array accessed by modulus that will raise and error when the modulus isn’t zero. I wanted to just divide by zero, but javascript is wrong and thinks that division by zero returns infinity, so I had to kludge it up a notch.

2. boolean operators (`if`s in disguise)

This takes advantage of the short circuiting nature of && and || to simulate the conditional logic.

     for (i = 0 ; i < 17 ; i++) {
        ((i % 15 == 0) &&
          (console.log('fizzbuzz') || true))
        || ((i % 5 == 0) &&
          (console.log('fizz') || true))
        || ((i % 3 == 0) &&
          (console.log('buzz') || true))
        || console.log(i)
      }

hack factor: 💽💽💽💽

I give this one four cd-roms. You will honestly see code that uses the short circuiting of boolean operators for control flow, and in fact some languages like php and ruby have special low-priority and and or operators to supplement && and || that are specifically meant for control flow. So you’re definitely a hacker if you use this.

3. multiplication (`if`s in a better disguise)

Some languages let you multiply strings by integers to make repeated strings, e.g. 2 * 'oy' == 'oyoy'. Ruby does this.

We can exploit this to make fizz buzz by doing slightly convoluted things to multiply fizz, buzz, and fizzbuzz by either 1 or 0 times to make the desired strings. This effectively is re-implementing the if conditional by doing something 0 or 1 times:

    def fizzbuzz_mult(n)
      s = ('fizzbuzz'*(1- [1,n% 15].min)) 
      s << 'fizz    ' * (1 - s.length / 8) * (1- [1,n% 5].min)
      s << 'buzz    ' * (1 - s.length / 8) * (1- [1,n% 3].min)
      s << n.to_s * (1 - s.length / 8)
      s.strip
    end

    1.upto(16).each {|n| puts fizzbuzz_mult(n)}

hack factor: 💽💽💽💽💽

This is essentially a hack on top of the boolean method, so it gets one extra hacker cd-rom.

4. done declaratively in sql

I’m so happy about this one. Every line has a little treat:

    select distinct on (range.num)
      coalesce(msg, cast(range.num as text))
    from (select * from generate_series(1, 16) num) as range
      left join (
        values ('fizzbuzz', 15),
          ('fizz', 5),
          ('buzz', 3)
      ) as lookup (msg, num)
      on range.num % lookup.num = 0;

awesome factor: 🎆🎆🎆🎆🎆

This one gets full awesome-ness score. With my undying infatuation with SQL, I can’t help but fawn over joining the series generation to a SELECT VALUES, then using DISTINCT ON to only return the first joined value, and coalescing in the default case from the nulls. It’s really lovely. Rock on SQL!

5. list reduction

This is in ruby again. The idea is to evaluate all of the if options until one is found to pass, then short circuit out of the rest of the loop. I really like the 1 modulus, n message part of this that keeps the special default case in the same reduction. The fizzbuzz problem is basic, but in it’s spirit it is really about handling a list of preferred possibilties. In that lens, this is the best solution.

    def fizzbuzz(n)
      [[15, :fizzbuzz], [5, :fizz], [3, :buzz], [1, n]].reduce(true) do |a, (m, msg)|
        a && if n % m == 0
          puts msg
          false
        else
          true
        end
      end
    end

    0.upto(16) do |i|
      fizzbuzz(i)
    end

Loveability: 💞💞💞💞

I really like how this hides the conditional logic behind a list reduction. This is actually a really useful technique when you have to evaluate an arbitary number of booleans.

6. oop

This is, of course, a bunch of java boilerplate wrapped around the original conditional technique. I prefer this in a way though. We’re only ever doing the conditional check once in any of these implementations, but in the real world this technique is better because we isolate that logic in the factory, and the let our polymorphism handle any future differences in behaviour.

    interface FizzBuzzable {
      void say();
    }

    abstract class FizzBuzzer implements FizzBuzzable {
    }

    class FizzBuzz extends FizzBuzzer {
      void say() {
        println("fizzbuzz"); 
      }
    }

    class Fizz extends FizzBuzzer {
      void say() {
        println("fizz"); 
      }
    }

    class Buzz extends FizzBuzzer {
      void say() {
        println("buzz");
      }
    }

    class Other extends FizzBuzzer {
      int n;
      Other(int m) {
       n = m; 
      }
      void say() {
        println(n);
      }
    } 

    class FizzBuzzFactory {
      FizzBuzzer getFizzBuzzer(int n) {
        if (n % 15 == 0) {
          return new FizzBuzz(); 
        } else if (n % 5 == 0) {
          return new Fizz(); 
        } else if (n % 3 == 0) {
          return new Buzz(); 
        } 
        return new Other(n);
      }
    }

    FizzBuzzFactory factory = new FizzBuzzFactory();
    for (int i = 0; i < 17; i++) {
      (factory.getFizzBuzzer(i)).say();
    }

Enterprisey-ness score: 🏦🏦🏦

This is enterprisey, but not that enterprisey. It needs more lines of code.

Usefulness score: 🛠🛠🛠🛠

If you were actually coding something with more complex logic than writing out a string given a certain condtion, polymorphism would be a useful tool to reach for.

7. functions

This one makes use of functions and currying. The checks will only happen if the previous failed.

    modder = (m) => (x) => x % m == 0
    mod15 = modder(15)
    mod5 = modder(5)
    mod3 = modder(3)

    fizzbuzzCurry = (fn, msg) => (n, a) => fn(n)? console.log(msg) : a[0](n, a.slice(1))

    sayFizzbuzz = fizzbuzzCurry(mod15, 'fizzbuzz')
    sayFizz = fizzbuzzCurry(mod5, 'fizz')
    sayBuzz = fizzbuzzCurry(mod3, 'buzz')
    sayDefault = n => console.log(n) // so it does not also output empty array

    fizzbuzz = (n) => sayFizzbuzz(n, [sayFizz, sayBuzz, sayDefault])

    for (i = 0; i < 17; i++) {
      fizzbuzz(i)
    }

functional score: λλλλ

I really enjoy how this one uses some functional programming concepts to achieve the desired result.

8. code generation

This one is mostly done to be silly. No one would ever do this, but I suppose that if you did it would make sense to only generate this ridiculous code once, to a file, then use that from then on.

    <?php

    define('BIGGEST_NUMBER', 32767);

    $str = 'function fizzbuzz($n) {' . "\n";

    for ($i = 0; $i < BIGGEST_NUMBER; $i++) {
      $msg = '';
      if ($i % 15 == 0) {
        $msg = 'fizzbuzz';
      } else if ($i % 5 == 0) {
        $msg = 'fizz';
      } else if ($i % 3 == 0) {
        $msg = 'buzz';
      } else {
        $msg = $i;
      }
      $str .= 'if($n == ' . $i . '){echo "' . $msg  . "\\n\"; return;}\n";
    }
    $str .= 'echo("error: this function only works with positive 16 bit signed integers.\n");';
    $str .= "}\n";

    eval($str);

    for ($i = 0; $i < 17; $i++) {
      fizzbuzz($i);
    }

The code that this generates is 32000 lines of:

    function fizzbuzz($n) {
    if($n == 0){echo "fizzbuzz\n"; return;}
    if($n == 1){echo "1\n"; return;}
    if($n == 2){echo "2\n"; return;}
    if($n == 3){echo "buzz\n"; return;}
    if($n == 4){echo "4\n"; return;}
    if($n == 5){echo "fizz\n"; return;}
    if($n == 6){echo "buzz\n"; return;}
    if($n == 7){echo "7\n"; return;}
    if($n == 8){echo "8\n"; return;}
    if($n == 9){echo "buzz\n"; return;}
    if($n == 10){echo "fizz\n"; return;}
    if($n == 11){echo "11\n"; return;}
    if($n == 12){echo "buzz\n"; return;}
    if($n == 13){echo "13\n"; return;}
    if($n == 14){echo "14\n"; return;}
    if($n == 15){echo "fizzbuzz\n"; return;}
    ...

kludgey-ness score: 💩💩💩💩💩

I only filled this in because it’s another unconventional technique that would solve this question.

9. declaratively in css

This one uses css’s nth-child formulae, and content to fill in the values. I kind of hate this one, because I don’t think you can refer to which child is matched in the content attribute, so you have to set up a separate css rule for each default case, making them totally not default and going against the spirit of the.

    <style>
    .fizzbuzz :nth-child(15n+1):after {
     content: '1'
    }
    .fizzbuzz :nth-child(15n+2):after {
     content: '2'
    }
    .fizzbuzz :nth-child(15n+3):after {
     content: 'buzz'WSh-child(15n+9):after {
     content: '9'
    }
    .fizzbuzz :nth-child(15n+10):after {
     content: '10'
    }
    .fizzbuzz :nth-child(15n+11):after {
     content: '11'
    }
    .fizzbuzz :nth-child(15n+12):after {
     content: '12'
    }
    .fizzbuzz :nth-child(15n+13):after {
     content: '13'
    }
    .fizzbuzz :nth-child(15n+14):after {
     content: '14'
    }
    body .fizzbuzz :nth-child(15n+0):after {
     content: 'fizzbuzz'
    }
    </style>
    <div class='fizzbuzz'>
      <div></div>
      <div></div>
      <div></div>
      <div></div>
      <div></div>
      <div></div>
      <div></div>
      <div></div>
      <div></div>
      as many of these as desired.
    </div>

kludge score: 💩💩

I give this one two poops of kludge.

Conclusion

I set out to solve this issue in ten different ways, instead of in ten different languages. I ended up stealing one of the techniques, but ended up using 7 languages in this article. It has been a really fun exercise. In the 1992 Remix of Scenario from A Tribe Called Quest Cut Monitor Milo asks “What does it take to check a technique?”, the answer: “Many styles, many styles!” I have enjoying checking my technique and working on my many styles while making this. If I ever have to apply for a job again, and they ask me about this chestnut, I may surprise them with 32000 lines of PHP.

Deep Learning Fundamentals

Ben Wendt — Fri, 31 Aug 2018 20:00:00 -0400

Here’s a talk I gave at work that gives a run-down of some fundamental concepts in deep learning, and I finished the talk with an example of a convolutional neural network in keras.

A Ruby Pipe

Ben Wendt — Wed, 31 Jan 2018 19:00:00 -0500

Javascript is going to get a pipeline operator any day now, and I love working with it in elixir, so I thought I’d try to make one in ruby.

So here’s an implementation

    class Object
      def pipe arg
        arg.(self)
      end
    end

And it works like this:

    plus1 = -> (x) {x+1}
    1.pipe(plus1) # 2
    1.pipe(plus1).pipe(plus1) # 3

And you can do things like add a logging aspect:

    putit = ->(x) {puts x; x}
    1.pipe(plus1).pipe(putit).pipe(plus1) # outputs 2 and returns 3

It’s pretty neat, but it only works with lambdas and procs for now. And it’s obviously not standard ruby so you’d be swimming against the tide if you tried to do anything useful with this.

Introduction to J

Ben Wendt — Sun, 31 Dec 2017 19:00:00 -0500

I tried using j for some of my advent of code puzzles. I found it wasn’t totally suitable for a lot of the challenges, but I had fun learning about J nonetheless.

J is a declarative and functional array programming language. I like this quote from J’s new user guide:

J isn’t just another way to declare variables and write loops. J is a way of thinking big: describing an algorithm by looking at it as a whole and breaking it into its natural parts. You’re going to have to spend some time learning what those natural parts are. Your skill as a program designer will help, but it will be fighting against your learned tendency to think small.

But I won’t be giving an example of that. I’ll just show how some of J’s vocabulary works.

Here’s how to multiply two numbers in J:

       4 * 4
    16

But let’s look at a silly way of doing the same multiplication:

       NB. note that comments in J start with NB., for nota bene
       1
    1
       NB. commands read right to left.

       NB. $ is the reshape operator it will take the input and
       NB. change it into an array with the size on the left.
       4 $ 1
    1 1 1 1
       NB. you can reshape to multiple dimensions
       4 4 $ 1
    1 1 1 1
    1 1 1 1
    1 1 1 1
    1 1 1 1
       NB. now we have 4 * 4 = 16 ones above, we need to add them up.

       NB. / is the insert operator. It adds it's left operand between
       NB. everything in the right operand.
       NB. So this shows the sum of all of the columns:
       + / 4 4 $ 1
    4 4 4 4
       NB. So, to get the result of 4 * 4, we need to apply the + insert twice:
       + / + / 4 4 $ 1
    16

Now, let’s imagine Guass knew J when he was in school, he could have done this:

       +/ 1 + i.100
    5050

Which probably isn’t as fun as the original story.

And for day of of 2017 advent of code, the question was very well suited for J. It gives you a 16 x 16 array of numbers, and asks for the sum of the max of each column minus the minus of each column. So in J, that was:

    e =: 790 ... NB. tonnes of numbers
    NB. reshape the input.
    f =: 16 16 $ e

    NB. insert the minimum into the list
    max =: >./
    NB. insert the maximum into the list
    min =: <./

    NB. sum of maxes minus mins. (|: is transpose)
    +/ (max |: f - min |: f) NB. answer

And that’s it. I really love J but I doubt I’ll get any chance to use it professionally.

Popping errors out from ruby threads using a Queue

Ben Wendt — Tue, 09 May 2017 20:00:00 -0400

I had an issue at work where some database writes were throwing errors within a thread, and the error was being silently eaten when the thread died.

The solution is to use a queue for inter-thread communication, like this:

q = Queue.new

Thread.new do
    i = 1
    loop do
    begin
        sleep i
        i += 1
        raise 'whatever'

    rescue StandardError => e
        q << e
    end
    end
end

loop do
    puts q.size
end

Intro to Bayesian Classifiers

Ben Wendt — Sun, 30 Apr 2017 20:00:00 -0400

Bayes’ theorem is a way of determining the likelihood of an event A given that another event B has occurred. It’s a way of making an educated guess without much information to go on.

It’s a way of going from knowing the likelihood of evidence, given a known outcome to knowing the likelihood of a known outcome given existing evidence. E.g. What’s the likelihood tacos are for dinner? If we know the likelihood of tacos, the likelihood of a meal being dinner, and the likelihood of tacos, given that it’s dinner, we can find this out.

Bayes’ Theorem gives you an idea of how much you should trust the evidence you have. If your evidence is weak, you won’t end up trusting it. For example, I’m sure my neighbours don’t think that our house is on fire every time they hear my fire alarm going off at supper time. (But it is pretty good evidence that I’m a lousy cook.) They’ve lost their trust in the evidence that my fire alarm gives that our house is on fire.

Say you are looking for P(A|B), the probability of event A occurring given the fact that we have the evidence B. For this to work, you need to know:

P(B|A): The probability that an event B is an A
P(A): The probability of A occurring. (This is often called the prior probability of A.)
P(B): The probability of B occurring.

Given these data, you can put the numbers into the following equation:

P(A|B) = P(B|A) * P(A) / P(B)

You might ask: “How do we get this equation?” We can start with the axiom of probability: P(A and B) = P(A given B)P(B). This axiom is saying that the probability of two things happening is the same as the probability that one thing happened, given that the other already happened.

This gets us 90% of the way to Bayes’ Theorem. From the axiom, you see that P(A given B) = P(A and B)/P(B). Now use P(B and A) = P(A and B), and reverse the terms in the axiom to get P(A given B) = P(B given A)P(B)/P(A).

A Simple Example

Suppose you have two bags containing marbles. Bag A contains 2 red balls and 1 yellow ball. Bag B contains 1 red ball and two yellow balls.

You don’t know which bag is which.

If you pull a yellow ball of a bag, what is the probability that it is bag A? We have an intuitive feeling that it’s most likely Bag B, but Bayes’ can give us an exact answer. This will be given by:

P(Bag A given a yellow ball being pulled) = P(Yellow ball given bag A) * P(bag A) / P(yellow ball)

P(yellow ball given bag A)``: This is the number of yellow balls (1) in bag A divided by the total number of balls in bag A (3), so1/3`.
P(bag A): This is the number of bags that are bag B (1) divided by the number of bags, so ‘1/2’.
P(yellow ball): This is the probability of drawing a yellow ball from any bag. Thus it’s the total number of yellow balls across all bags (3), divided by the total number of balls in all bags (6), so ‘3/6’

Plugging these values in, we get the following:

P(Bag A|yellow) = (1/3) * (1/2) / (3/6)

The result is 1/3. So we know that if we reach our hand into one of bag A and bag B, and draw a yellow ball, that the probability that we reached into bag A was 1/3. This matches our intuition, but Bayes’ Theorem is most well known for it’s intuition defying examples.

A mind-expanding example

You’re walking down the hall at work. You see someone walk out of the software development wing who is odd. You wonder to yourself: is this person a Fortan developer? Your sneaking suspicion is that this odd person is probably a Fortran developer because you have a feeling like Fortran developers are mostly odd. Let’s work through an example.

P(fortran developer, given odd) = P(odd, given Fortran developer) * P(fortran developer) / P(odd)

Let’s substitute in some estimates for these values.

P(odd, given fortran developer): Let’s say 85% of fortran developers are odd.
P(fortran developer): Most developers are average and use react. Only 1% of all developers use Fortran.
P(odd): Across the whole company, you estimate that 10% of developers are odd.

Let’s plug the numbers into Bayes’ theorem and find out whether the odd person we have observed is probably a Fortran developer:

P(fortran developer, given odd) = (0.85) * (0.01) / (0.1)

And the answer is 0.085, or 8.5%. This runs counter to the original intuition that the developer must be a fortran developer because they are odd. While it’s true that most fortran developers are odd, there are so few fortran developers total, meaning that most of the time an odd developer is not a fortran dev.

The odd Fortran developers are outnumbered by the comparatively less common odd react developers, but that the total number of odd react developers is still higher. So just because someone is odd, it doesn’t make them a Fortran developer.

Learning Through Bayes’ Theorem

This is all super interesting and cool, but there’s an even more cooler application of this theory we can look at. Notice how the prior probabilities we worked with in the examples before were static. We can continually update these to get a better model of our simulation as we encounter new data. Assuming that the model behaves consistently, we will get continually better results.

A historical example of this line of reasoning comes from a thought experiment performed by Thomas Bayes. The experimenter asks an assistant to throw a ball onto a table (for the sake of the experiment, it doesn’t bounce or roll). The goal of the experiment is to determine the location of this ball without being able to see its location. The assistant would then throw ball after ball onto the table and tell the experimenter whether each successive ball landed to the left or right, and behind or in front of the first ball. By running successive trials, the experimenter can update their view of where the ball might be. For example, if most subsequent balls are reported as being to the left of the first ball, it’s safe to guess it’s near the right side of the table. Bayes’ Theorem lets us quantify this measurement.

A proper learning example

Now we’ve seen how we can apply a learning and Bayes’ to improve our predictions using probability. Let’s apply the same to classifying something a bit more complex. Here we will be tracking multiple probabilities and combining those to predict a classification of a document.

This technique assumes that we have a training set of data where a human has manually gone through and confirmed the classifications of a group of documents.

Example of headlines and news category:

text

classification

Local Team Loses Game

sports

Player Loses Game

sports

Movie breaks box office record

entertainment

English team scores goal

sports

gravity waves recorded using machine

science

New Comedy Is A Good Movie

entertainment

New Global Warming Movie

science

…

Using this data we can train a classifier… Let’s break down the results by word and see which words tend to be in certain categories:

word

P(sports)

P(entertainment)

P(science)

Local

1 / 1

Team

2 / 2

Loses

2 / 2

Game

2 / 2

Player

1 / 1

Movie

2 / 3

1 / 3

breaks

1 / 1

box

1 / 1

office

1 / 1

record

1 / 2

English

1 / 1

Scores

1 / 1

Goal

1 / 1

Gravity

1 / 1

Waves

1 / 1

Using

1 / 1

Machine

1 / 1

New

1 / 2

Comedy

1 / 1

Good

1 / 1

(Note that this data will work for an example, but for good results you would want a much larger training set than this.)

Here we can see the prior probabilities of a given word in a title affecting the classification of the document. Now let’s assume that we fed millions of headlines into this system and ended up with something like this:

word

P(sports)

P(entertainment)

P(science)

Local

33000

Team

35578

Loses

29666

200

…

And so on. This gives an example of how sports-centric terms will train to give high probabilities of related documents landing in the sports category. The prior evidence here is strong.

When we want to classify a new document, how do we compute the probability of it being in a given class, now that it has multiple terms? First, let’s think about what it would mean if there were only two terms to worry about.

P(sports, given win AND team) = P(win, given sports AND team) * P(team given sports) * P(sports) / (P(win AND team) * P(team))

This is already complex for only two terms, but let’s look at a simpler way to look at this. We can assume independence of the terms. This may not be the most accurate model but it will give us results that we can work with and evaluate. This assumption that the terms are independent makes calculation simple, but it’s also not entirely accurate, hence why this technique is called ‘naive’. This is the ‘bag of words’ technique commonly used in NLP. For example, headlines that contain ‘delicious’ would tend to occur more often with ‘cake’ than ‘manure’, meaning ‘delicious’ and ‘cake’ have a dependence. Naive Bayes’ classifiers can compete very well against other classifiers in terms of accuracy, but we need to be mindful of this limitation that we have imposed on ourselves.

If we assume that instances of ‘win’ and ‘team’ in a sentence are independent, (which they aren’t entirely), we get an easier formula where we already have the answers in our table.

P(sports, given win AND team) = P(win given sports) * P(team given sports) * P(sports) / (P(win) * P(team))

This will be a lot easier to calculate. And we can generalize this formula:

P(sports, given words) = (product of each P(word|sports)) * P(sports) / (product of each P(word))

With this formula in hand, we just have to iterate through each classification to determine if a document is in a given class, then we can report the most likely classification along with it’s probability. If a user confirms the classification, then we can add the new information to the prior probability table.

So how would you implement this?

Generally as a developer it’s best to reach for a pre-built tool rather than making your own. You don’t usually make yourself a hammer every time you see a loose nail. Similarly it’s best to reach for a pre-built library that can handle this kind of classification for you.

But there is real value in understanding how algorithms work. So let’s take a look at how one would implement a naive Bayesian classifier.

What would the psuedo-code look like?

classify classes = [], document:
  max_class, max_score = null, 0
  for (class : classes) {
    score = get_score(class, d)
    if score > max_score
      max_class = class
      max_score = score
  }
  return max_class;

get_score class, document:
  score = log(get_prior(class))
  features = tokenize(doc)
  for(feature : features) {
    score += log(getProbability(feature))
  }
  return score;

Here we have two methods. classify is pretty self explanatory. It’s just finding the maximum scored class for the given document. get_score is more interesting. Here is where we apply naive Bayes to calculate the scores for each class. Notice two things:

We’re using log and addition instead of multiplication. This is typical in NB implementations. This is done to avoid issues with underflow in floating point numbers. But the result is the same; adding logs is equivalent to multiplication (just ask your slide rule). But this log arithmetic means that the number we’re getting here isn’t a probability. That’s fine because we just want to find with class gives the highest number.
Notice how we don’t divide by the denominator P(B), this is because it’s a constant across all of our documents so there is no need to repeatedly do this calculation since we’re just looking for a the highest number, not a probability.

An important part is missing here, we didn’t compute the tables of prior probabilities. As mentioned before, we can do that by working through a collection of pre-classified training documents and building up counts of which features lead to which classes. Let’s look at how that might look:

train class, features:
  increment_class(class)
  for (feature : features) {
    increment_feature(feature, class)
  }

The training will keep track of the occurrence of the different class and the occurrence of different features within each class.

Here is a working example in old-fashioned javascript that does the same thing:

function Bayes() {
  this.features = {}
  this.categories = {}
}

Bayes.prototype.getScore = function(category, features) {
  score = Math.log(this.getPrior(category))

  for(var feature of features) {
    score += Math.log(this.getProbability(feature, category))
  }

  return score
}

Bayes.prototype.getPrior = function(category) {
  return (this.categories[category] || 0.1) / this.totalDocuments
}

Bayes.prototype.getProbability = function(feature, category) {
  return (this.features[category][feature] || 0.1) / this.categories[category]
}

Bayes.prototype.classify = function(features) {
  let maxClass = null
  let maxScore = -Infinity
  for (var category in this.categories) {
    const score = this.getScore(category, features)
    if (score > maxScore) {
      maxClass = category
      maxScore = score
    }
  }
  return maxClass
}

Bayes.prototype.incrementTotal = function(category) {
  this.totalDocuments = (this.totalDocuments || 0) + 1
}

Bayes.prototype.incrementCategory = function(category) {
  this.categories[category] = (this.categories[category] || 0) + 1
}

Bayes.prototype.incrementFeature = function(feature, category) {
  if (!this.features[category]) {
    this.features[category] = {}
  }
  this.features[category][feature] = (this.features[category][feature] || 0) + 1
}

Bayes.prototype.train = function(category, features) {
  this.incrementTotal()
  this.incrementCategory(category)
  for (var feature of features) {
    this.incrementFeature(feature, category)
  }
}

b = new Bayes()

b.train('city', ['subway', 'tower', 'hotel'])
b.train('city', ['opera', 'tower', 'hotel'])
b.train('city', ['opera', 'subway', 'hotel'])

b.train('country', ['cow', 'field', 'hill'])
b.train('country', ['cow', 'field', 'farm'])

// city
console.log(b.classify(['subway', 'hill', 'tower']))

// country
console.log(b.classify(['farm', 'hill', 'sheep']))

And that is it. We’ve successfully used Bayes’ Theorem to classify documents.

references

Adding Database Constraints Using the `rein` Gem

Ben Wendt — Mon, 27 Mar 2017 20:00:00 -0400

rein is a gem for adding database constraints in rails migrations. It’s always been possible to set these up using execute calls in the migration, but rein makes it look rails-y.

Letting your database manage constraints is often a great idea. I’m not a huge fan of the rails-way of letting the application layer manage all relationships between data. My main concerns with this are:

A developer cannot look at the database and understand the meanings of all the data it contains. A prime example of this is using rails enumerations to store integers that have a meaning to the application.
Letting the application manage this for you locks you in to rails, but you really want flexibility in terms of what technology you want to implement the application in. If you were to need to add another service with access to the database, it would need to duplicate the management of any information it needs access to. The could be a real pain point when using any other technology than rails (and even if the other app was in rails). Adding constraints in the database eases this issue, as regardless of what technology is used on the application layer, you will have guarantees of your data’s consistency from the database.

So let’s take a look at a sample migration using rein, and what limitations it puts on the application layer.

class CreateBooks < ActiveRecord::Migration
  def up
    create_enum_type :binding, ['hardcover', 'softcover']
    create_table :books
    add_column :books, :name, :string
    add_column :books, :description, :string
    add_column :books, :binding, :binding, :default => 'hardcover'
    add_column :books, :publication_month, :int

    add_numericality_constraint :books, :publication_month,
      greater_than_or_equal_to: 1
  end

  def down
    drop_enum_type :binding
    drop_table :books
  end
end

This will set up a new type called binding, and a books table with a numericality constraint on its publication month field. Let’s see what happens when we try to save some data that the database doesn’t like:

irb(main):001:0> b = Book.new
=> #<Book id: nil, name: nil, description: nil, binding: "hardcover", publication_month: nil>
irb(main):002:0> b.publication_month = -1; b.save
ActiveRecord::StatementInvalid: PG::CheckViolation: ERROR:  new row for relation "books" violates check constraint "books_publication_month"

And with the enum type, you see something similar:

b.binding = 'none'; b.save!
ActiveRecord::StatementInvalid: PG::InvalidTextRepresentation: ERROR:  invalid input value for enum binding: "none"

This is great! We can’t save this data to the database. Now, ideally we would set up validations that mirror these constraints so that we can handle this data gracefully. Further than this, we can use a rails enum in the model:

enum :binding, {
  'hardcover' => 'hardcover',
  'paperback' => 'paperback'
}

This will give us methods like book.hardcover?, book.paperback!, scopes like Book.hardcover and do all of our validations. It’s a very effective pairing.

My first slack bot

Ben Wendt — Wed, 14 Sep 2016 20:00:00 -0400

I am still working on picking up some elixir, so when a coworker mentioned that writing a Slack bot is cool, I decided to give that a try. Slack is a bit easier to work with than IRC because the communication is done in JSON so you can skip a step in message parsing.

The login process for a slack bot is:

Authenticate over HTTP using your bot’s token.
Open a websocket connection using the socket url returned in 1.
Communicate using JSON over the socket.

Each of these steps required importing a library (:httpotion, "meh/elixir-socket", and :poison). This is a bit of a departure from go where the standard library has HTTP and JSON support, and there’s an x-package for websockets. But everything worked well. It’s three external dependencies instead of two.

Elixir does have a package repository, so it’s way ahead of go in that regard.

You can take a look at my slack bot here.

The World's Simplest IRC Bot - Again

Ben Wendt — Wed, 14 Sep 2016 20:00:00 -0400

I was talking to a colleague who was expounding about the coolness of elixir. So I thought I should give it a try. In the spirit of doing something non-trivial to learn a new tool, I decided to port over the IRC bot I’ve been working on in go and ruby (which passes my bar for non-triviality).

You can check it out here. It does the same super-simple (but easily extensible) bot behaviour of replying to a specific salutation.

Elixir seems pretty nice. It’s syntax is clearly influenced by ruby, which is great. It is familiar enough that you want to be able to pick it up. But once you dig in a little bit you start finding that nothing is quite what you’d expect. I ended up feeling a bit like I was coding a hybrid of go and ruby. That was great because those are what I’ve been working on exclusively for the past eighteen months (ignoring inevitable forays into JS).

The error messages are pretty helpful, definitely less onerous than those in go. And I didn’t hit any head-wall situations. So far it has been very developer friendly. The online documentation is good, but there doesn’t seem to be as much on google or stack overflow as I’m used to seeing in most other languages. I’d love to keep working in elixir if I ever get the chance.

The World's Simplest IRC Bot

Ben Wendt — Wed, 31 Aug 2016 20:00:00 -0400

I saw an article on twitter about writing an IRC bot in ruby. It piqued my interest because of nostalgia and because I’m a slack curmudgeon. I believe that IRC has value. I never actually got around to trying the cinch gem that the article recommends, although if I ever need to do anything serious in this realm it wil be the first tool I turn to.

Instead I found an article about writing a really simple ruby class for a bot. It never really occurred to me how simple the actual connection part of IRC is, so after playing around with this code for an hour or two I decided to port it to Go.

What I came up with is an extremely simple connection struct that wraps a tcp socket with some metadata, and allows for injection of a message handler, while maintaining the PING / PONG keep-alives on its own.

I spent at least twenty minutes trying to figure out why this wasn’t working before realizing fmt.Println puts in a line break, while fmt.Fprintf does not. Don’t trust your debugging output too much.

func (c *IrcConnection) Say(msg string) {
    fmt.Println(msg)
    fmt.Fprintf(c.conn, fmt.Sprintf("%s\n", msg))
}

For an afternoon project, I like how neat and compartmentalized it is, excusing the natural verbosity of Go. If I ever put more time into this, I’d like to:

Add tests. I wavered on this because at the end of the day it has to connect to a real IRC server so I did all my development against that. Sorry freenode. But it would be pretty easy to make an IrcConnection interface, craft a dummy version with a buffer I can control, and use that to test that things like PONG work correctly.
Factor code more. For example, I could definitely use a separate method for all the handshake messages done during connection.
Change Run to work in a goroutine, and make a separate message handler that allows user input. That would turn this into a single channel IRC client application, which would not be the least cool thing in the world.

That’s it. Just another fun little side project I thought I would share.

Designing Models for 3D Printing in openSCAD

Ben Wendt — Thu, 07 Apr 2016 20:00:00 -0400

My work (TWG) has a 3D printer for employees to play with and use. Sometime last summer I downloaded some train tracks from thingiverse and printed them for my son to play with. He loves the custom tracks dearly. He’s always excited to visit my work to see what a coworker called “the toy factory”. And often he will ask me to print him something while I’m at work. So I get a lot of emotional reward from using this machine.

Thomas and Friends (based on the books by Rev. Audrey) has a line of toys called “wooden railway”. The tracks are an ad hoc standard, and work with many different brands including Brio (my favourite), imaginarium, Melissa and Doug, and (for the most part) Ikea. You can find lots of designs for brio compatible tracks on thingiverse. Seemingly all kids love these.

Unfortunately for parents, there are also Thomas “Take ‘n Play” trains which are about 70% of the size of the wooden railway models. The engines and rolling stock don’t fit on the same track, and the track connectors are incompatible. I accidentally bought my boy a Take ‘n Play engine once, and my son received a track set as a gift from a grandparent. This was quite a limited set that didn’t offer much room for creativity.

My son enjoyed it, but he asked me to print tracks that would work for this system. I checked thingiverse and google and could find none. This provided the challenge I needed to design and make my own 3D models.

I chose to use openSCAD for modelling. I was reading a comparison of different modelling software and it said that openSCAD has for loops, and I was sold. I love a good for loop.

openSCAD has three primitives and extrudable polygon support, as well as several object transformations like skewing and resizing. It also supports module which works similar to a function call and allows you to repeat groupings of commands to repeat code and make it more readable. With these tools you can build up more complex designs.

Designing the object was an interesting process. At first I built a simple version of the object with primitives and hard-coded dimension values, which I measured with a micrometer. My next step was to factor common variables and move objects and transformations out into modules.

Because I generally have at least a few lines of code in my blog posts, here’s a simple module that defines the center rails for my track design:

module center_rails(offset, rail_width, length, depth) {
    translate([offset,0,1]) {
        cube([rail_width, length, depth], center=true);
    }

    translate([-offset,0,1]) {
        cube([rail_width, length, depth], center=true);
    }
}

You can see the whole code in a github repo, or get this object on thingiverse.

Through this process I learned that what you measure with a ruler isn’t necessarily what you need to print to have a working model. I created about eight prototype pieces before I had compatible male and female connectors. In my first attempt, I used the same measurements for each, but this didn’t work because the connectors need a little bit of space to slide past one another. It was a really fun process and I’m very much looking forward to showing these to my son tomorrow.

Playing with Vue.js

Ben Wendt — Sun, 03 Apr 2016 20:00:00 -0400

A coworker recently filled my ear with praise for vue.js. I’m always eager to try new frameworks, and the benefits over angular (namely performance and simplicity) sound really promising. I haven’t had a chance to develop anything serious with this, but my initial attempt was exhilirating.

I saw a link to the xkcd api in a tweet the other day, and decided to try it out with vue.js. I don’t really enjoy the comic strip, but it’s a publicly availably API that isn’t totally boring. So… what the hell. Why not?

Vue.js template code looks a lot like angular. So here’s my simple template:

    <div id="app">
      <div>
        <button v-on:click="previous">&lt;</button>
        <button v-on:click="next">&gt;</button>
      </div>
      <template v-if="data.img">
        <img src="{{ data.img }}">
        <h1> {{ data.safe_title }} </h1>
        <p>
          {{ data.transcript }}
        </p>
        <p> {{ number }} / 1600 </p>
        <p> {{ error }}
      </template>
    </div>

I’m impressed by how little code gets this running. A good chunk of my code is native run-of-the-mill XMLHttpRequest boilerplate:

      vm = new Vue({
        el: '#app',
        data: {
          number: 400,
          data: {}
        },
        methods: {
          previous: function() {
            this.number --
            this.fetchData()
          },
          next: function() {
            this.number ++
            this.fetchData()
          },
          fetchData: function() {
            var xhr = new XMLHttpRequest()
            var self = this
            xhr.open('GET', 'xkcdpass.php?n=' + self.number)
            xhr.onload = function() {
              self.data = JSON.parse(xhr.responseText)
            }
            xhr.send()
          }
        }
      })
      vm.fetchData()

You can try it out here.

Relevance Graphing

Ben Wendt — Sat, 26 Mar 2016 20:00:00 -0400

Last month a workmate and I flew down to San Francisco to go to elasticon.

I attended an interesting talk given two employees of Giant Oak. Giant Oak does contracting for government agencies to try to solve social problems. I am not sure if this is their motto but one of the speakers say that they “see the people behind the data,” which sounds really cool.

Their work involves things like trying to answer questions about war insurgency, or terrorism. They don’t work on computer science questions, but social science questions. Some of their analytical tools for social scientists are built on ElasticSearch.

One of the tools used to reveal data poaching data was a graph representation of their data.

The graph databases I’ve used, like neo4j, allow a user to define entities and create a graph by defining relationships between those entities. But elasticsearch has a new feature coming out in version 5 for auto generating edges based on relevance. A user will not have to manually set up these relations when creating data. One of the developers Giant Oak made an elasticsearch reporting tool, similar to kibana, called unicorn. It is built for revealing relationships between records in the database, and uses existing elasticsearch technology.

As opposed to standard graph databases where the relationships on the edges of graphs are explicitly defined, this tool generates entity relationships based on relevance. So when different documents share attributes, they are linked.

This work has led to over 100 arrests of human traffickers and poachers. Revealing the stories hidden the data, can be fun or informative, or even save people’s lives and make the world a better place.

Fun with `fork`

Ben Wendt — Mon, 01 Feb 2016 19:00:00 -0500

In irb,

fork

Now typing exit will not work because your keystrokes might go to either the parent process or the forked child process. I found it to be impossible to get the characters exit to coherently all go to the same process.

Getting a sum of durations in rails

Ben Wendt — Sun, 31 Jan 2016 19:00:00 -0500

For my “I had trouble finding this on google” series, here’s a solution I found to getting a total duration from a collection of objects with durations stored as Time objects.

total_duration = durations.reduce(Time.gm(2000)) do |total, time|
  total += time.to_i
end

There are two little tricks happening here. The first is that durations are stored as a time of day on January 1st, 2000, so we use that as the reduce seed. The second thing is that the Time class defines addition for integer objects, but not for other times, so you have to convert your duration to an integer first.

No rocket science here, but a couple little quirks that made it a little harder to add two times together than I expected.

Brainfuck Interpreter

Ben Wendt — Fri, 22 Jan 2016 19:00:00 -0500

Every programmer has some level of interest in knowing how to write a language. I’ve always wanted to tinker with this level of software development, but even after years of development experience you never really get exposed to this stuff because what’s already there generally just works. On top of that, it’s a somewhat complex process involving several steps with intimidating terminology.

But the best defense against not knowing how to do something that you want to do is always jumping in and doing it. So, with help from an excellent article by Ben Johnson about writing your own parser in go, I dived in. This article has easily digestable definitions and example of the different steps of the parsing and lexing process.

lexing

Lexing is the process of converting a steam of structured text, like the programming language code you want to interpret, into a list of pre-defined symbols. So you could take a string like "while" and convert it into something your compiler / interpreter undestands as a while symbol.

parsing

Parsing is taking a bunch of symbols generated by the lexer and turning them into a structure that your compiler / interpreter will be able to work with. So it takes something like [IF, CONDITION(true), START_BLOCK, PRINT, STRING(hello), END_BLOCK] and changes it into something more like:

if condition(true)
  print 'hello'

choosing brainfuck

brainfuck is an esoteric programming language, meaning no one uses it for any serious work. I chose to write an interpreter for it because it has only eight commands, and it is very conceptually simple. Most compilers and interpreters will parse lexed tokens into something called an “abstract syntax tree”. Since brainfuck is so simple, I was able to skip this step and make a struct with methods that act on the structs state in place. Brainfuck also has lots of example code online that was very useful for writing my tests (more on that later). Brainfuck is also a fairly minimal language that is still Turing-complete, which is a desirable quality to have in a language for which you are writing an interpreter.

Brainfuck is also really nice because it doesn’t have a context aware grammar, meaning that you don’t have to keep track of when you’re in string. There are no variables or functions, so you don’t need a symbol table. In all, it’s a simple featureless language perfect for this kind of task.

What I came up with probably wouldn’t get a great grade if it was what I handed in for what I assume might be the first assignment in a compilers course, but I learned a lot that was worthwhile, so I’ll discuss those findings here.

implementation

Implementation was actually a bit easier than I expected. With a bit of help writing the lexer from the aforementioned blog post, I was able to create a Machine struct with a series of operations. Each of these operates on the state of the struct, which contains operations, current operation, a sequence of bytes that can be manipulated, a reference to the current byte being processed, input, and output:

type Machine struct {
    Position int
    Operation int
    State []byte
    Output []byte
    Input *bytes.Reader
    Operations []Token
}

The operation array is filled with a collection of tokens, and as the machine moves through the operations, each token is mapped to a method that operates on the object. Most of these alter the State slice or position in some way, except for the control flow methods [ and ], which read the current position and jump to the corresponding bracket if the value at the current position is zero or non-zero respectively.

Because there is only one control flow operation in brainfuck, and with it being so simple, I saw this as an opportunity to “get my feet wet” writing an interpreter that doesn’t use an abstract syntax tree. It’s a barrier to learning about this technology that I didn’t have to hurdle to get this working. My implementation just does the control flow manually in place on the Machine struct by moving the Position value.

tests

The last time I was playing with brainfuck, I wrote a string to brainfuck “compiler” that gave me some simple test cases for outputting text. It’s also pretty easy to test the input case: just pre-populate your input, then output it, and assert equality.

I was able to find many “Hello world” examples for brainfuck. The first time I put one in it dropped my interpreter into an infinite loop. Resolving this was definitely that hardest part of this process. I ended up just mentally thinking through what would be wrong until I figured it out: it was an off by one error. Once that worked, some of my “hello world” tests almost worked! I was very excited. But they had an unexpected character with byte code 10 at the end. I was pulling my hair out. A quick check of my ascii table answered the question. Whoever coded these put a new line at the end of the output.

(Thinking back I remember seeing CHAR(10) or similar in an old SQLServer database eons ago, but I must have erased that experience from memory.)

After that I put in a program on esolangs.org listed as “often triggers interpreter bugs.” It worked on the first go and I was happy.

conclusion

It’s an interpreter no one will ever use for a language that no one ever uses, but I’m happy with it. I learned enough and had a pleasant experience, with only one ruined sleep. I think esolangs offer a great opportunity for writing interpreters because the languages generally don’t have many commands. Maybe I will try writing one for Piet next.

You can check out my brainfuck interpreter on github.

Republished

Ben Wendt — Fri, 27 Nov 2015 19:00:00 -0500

My old “attach a camera to your RC car” tutorial has been republished into a “1-2-3 Projects” anthology by maker media.

Restricting post-deployment tasks to certain roles using capistrano

Ben Wendt — Mon, 09 Nov 2015 19:00:00 -0500

If you need to only run some tasks on a certain subset of servers in your inventory, first add a role to your inventory file deploy/whatever-environment.rb:

server '1.2.3.4', roles: %w{my_role}

Now you can set up your command to run in one of your tasks in deploy.rb:

on roles(:my_role) do
  within release_path do
    with rails_env: fetch(:rails_env) do
      execute :bundle, :exec, :'script/bend-girders.py', args, :all
    end
  end
end

Converting strings to bf code

Ben Wendt — Sat, 10 Oct 2015 20:00:00 -0400

I was playing around with the esoteric programming language brainfuck. I found that it was pretty time consuming writing out strings of text using only increments and decrements of the data pointer.

So I decided to write a bf string maker in go. Code generation is pretty important in go, so it is fun to generate code for another language using go.

package main

import (
    "os"
    "fmt"
    "strings"
)

func main() {
    buffer := 0;
    if len(os.Args) > 1 {
        in := strings.Join(os.Args[1:], " ")
        runes := []rune(in)
        for _, rune := range runes {
            current := int(rune)
            symbols := ""
            if current > buffer {
                symbols = strings.Repeat("+", current - buffer)
            } else {
                symbols = strings.Repeat("-", buffer - current)
            }
            fmt.Println(symbols + ".")
            buffer = current
        }
    } else {
        fmt.Println("no params")
    }
}

The idea is that any acscii string (a collection of runes in go) can be translated to bf by playing with the bf data pointer. This pointer starts at zero, and we just walk through the string moving the pointer: increments are + commands and decrements are - commands. The . operator outputs the current data pointer.

Search Index Internals

Ben Wendt — Sat, 26 Sep 2015 20:00:00 -0400

Watch me discuss how a search engine works.

A Random Password Generator

Ben Wendt — Thu, 03 Sep 2015 20:00:00 -0400

I’ve posted a random password generator I’ve written to github. The impetus for this was me accidentally posting one of my passwords in an open chat channel at work. I decided it was time to change my password.

I found a python library that generated xkcd style passwords a while back, which I can no longer find. So, as part of my ongoing quest to have fun and program a lot I decided to whip one up myself.

Two glaring improvements are:

The dictionary path is currently hard-coded for ubuntu, which probably north of 90% of people are using by now, but it would be great to make it configurable for people stuck on older machines.

I’d like a “text-munging” option that would randomly replace the occasional x with a %, and the odd s with a $, and so on.

Enjoy!

NLP Essentials

Ben Wendt — Fri, 17 Jul 2015 20:00:00 -0400

Here’s a talk I gave at work.

A Comparison of Stemmers

Ben Wendt — Tue, 07 Jul 2015 20:00:00 -0400

The goal of stemming is to reduce derived forms of a word to something that could be a root form. For example a possible stem of “lighting” or “lighted” is “light”. This is generally done by applying a list of fairly simple rules to a word, possibly recursively, until the algorithm is done and a root form is returned.

There are several reasons for stemming a word. The primary concern is to increase the recall of a search index. The most basic case here is the handling of plurals; a user generally expects the same result when searching for “nachos dip” as when they search for “nacho dip.” But recall can be improved in other cases as well, for example a user searching for “democracy” is probably also interested in results for “democratic”. Another concern is reducing index size; this can increase search speed and reduce storage usage (both of these are good for reducing costs).

But stemming is an inexact science. Most stemming algorithms don’t attempt to extract morphological information about the word to determine the stem, so unrelated terms can be conflated. While working at an old employer, one of my colleagues found an interesting issue caused by imprecise stemming. A client was complaining that a search for “rug” was matching most of the items in their catalog, even though most of them didn’t have anything to do with rugs. It turns out that, as an outdoor gear supplier, the client had the term “rugged” in most of the product descriptions on their site. The solution to this issue was to tweak the stemmer used for this index.

This example shows that, while stemming can improve recall, it reduces precision. As software developers, we are used to making these trade offs. The effectiveness of a given stemming method will vary depending on your corpus. So it’s important to learn the ins and outs of your stemmers, and learn when to tweak them.

I found an interesting comparison of stemmer results in an elastic search user forum.

Original	porter	kstem	minStem
country	countri	country	country
run	run	run	run
runs	run	runs	run
running	run	running	running
read	read	read	read
reading	read	reading	reading
reader	reader	reader	reader
association	associ	association	association
associate	associ	associate	associate
listing	list	list	listing
water	water	water	water
watered	water	water	watered
sure	sure	sure	sure
surely	sure	surely	surely
fred’s	fred’	fred’s	fred’
roses	rose	rose	rose

You can see here that Porter earns its aggressive reputation. Aggressiveness leads to more matches, but also more false matches.

Setting a custom response header in a Gin response

Ben Wendt — Wed, 03 Jun 2015 20:00:00 -0400

I’m using Gin Gonic on a project I’m working on at work these days. I was in a bit of a pickle where I had to specify a response body that I already had as a string, but I needed to put the correct content type header on the response (meaning I couldn’t use the built in .String or .JSON methods.

I had to dig around a bit in the source code a bit to figure this one out. So, If you want to set a custom response header on a gin response, do it like this

imports (
    "github.com/gin-gonic/gin"
    "github.com/gin-gonic/gin/render"
)
func whatever() {
    ...
    c.Render(
        http.StatusOK, render.Data{
            ContentType: "application/json",
            Data:        []byte(response),
        })
    ...
}

An interesting example of using a closure for memoization

Ben Wendt — Thu, 14 May 2015 20:00:00 -0400

Long ago I wrote about the benefits of memoization. It’s a simple idea: a time vs. space trade off. Trade time in CPU for space in memory. A pretty classic example is the massive speed benefit you can get while calculating the Fibonacci sequence by saving values you have already found. (The naive recursive approach recalculates values an exponential number of times.) It’s a toy example but it definitely exhibits the power of the technique.

I have to learn Go for an upcoming project at work. I’m excited about it, and I’ve been starting out by working through the Go Tour lesson series. I was pretty interested to see slide 22 in the more types lesson, an exercise instructing the reader to write a function that calculates Fibonacci numbers using a closure. The way this is set up gently nudges the reader toward writing an answer that uses memoization through a closure.

Here’s what I came up with:

package main

import "fmt"

// fibonacci is a function that returns
// a function that returns an int.
func fibonacci() func() int {
    a := [10]int{1, 1}
    b := 0
    f := func() int {
        val := a[b]
        if val == 0 {
            val = a[b - 1] + a[b - 2]
            a[b] = val
        }
        b ++
        return val
    }
    return f
}

func main() {
    f := fibonacci()
    for i := 0; i &lt; 10; i++ {
        fmt.Println(f())
    }
}

I really liked the way that the solution uses a built in array and a closure to accomplish memoization. It’s very clean. Coming from doing a lot of PHP it’s nice to see because a fairly standard way of implementing this technique in that language is to use the static keyword which always felt a bit hacky (on a side note, you could do this in PHP, but you’d need the use feature on the closure, which I am also not a fan of). The same technique would definitely work in javascript as well. It’s a welcome addition to my toolset.

Using a motion detector with the espruino pico

Ben Wendt — Wed, 06 May 2015 20:00:00 -0400

This is a simple modification of the espruino motion detector tutorial to work with the espruino pico. The main difference is which pins things are hooked up to, and my not using the LED strips, which I don’t own (hint hint, Santa).

First off, using your espruino pico, set it up at the standard left-most position on the bread board.

Then wire up as follows:

HC-SR501	Espruino
VCC	VBAT (5v)
OUT	A7
GND	GND

The main difference here is that we use A7 instead of A1 because on the pico, A1 doesn’t come with a pin soldered on, so you can’t just plug a jumper into the bread board.

The code is basically the same as the tutorial, with the updates of removing the LED strip code and updating A1 to A7

var timeout;

function lightsOn() {
  digitalWrite(LED1, 1);
  console.log('light on ' + (new Date().toString()));
}

function lightsOff() {
  digitalWrite(LED1,0);
}

setWatch(function(e) {
  if (timeout!==undefined)
    clearTimeout(timeout);
  else {
    lightsOn();
  }
  timeout = setTimeout(function() {
    timeout = undefined;
    lightsOff();
  }, 1500);
}, A7, { repeat:true, edge: "rising" });

This thing is so much fun to toy around with. I had fun putting together the pico piano project too.

Playing with my espruino pico

Ben Wendt — Sat, 02 May 2015 20:00:00 -0400

About a year ago I had a little fun using an espruino. I recently “kick-started” their more recent product the espruino pico. This one did really well on kickstarter and I got some great “boost” rewards like an LCD screen and some relays. It should be fun to see what this thing can do.

Below is a simple program that makes a spot spin around on the LED display that came with the kit. Variables like A5 or B10 represent pins on the board, which can be seen in the pico schematic.

This makes use of the PCD8544 driver, which works with the espruino graphics library. This is loosely based on some of the examples on the pico site.

A5.write(0); // GND
A7.write(1); // VCC

// http://www.espruino.com/SPI
var spi = new SPI();
spi.setup({ sck:B1, mosi:B10 });

var height = 48; // these are the resolution of the Nokia 5110.
var width = 88;

var rate = 140;
var theta = 0;
var cx = width / 2;
var cy = height / 2 - 1;
var r = cy - 1;
var rx = r * 1.6;
var dTheta = 0.1;

var g = require("PCD8544").connect(spi,B13,B14,B15, function() {
  setInterval(function() {
    g.clear();
    theta += dTheta;
    var x = cx + rx * Math.cos(theta);
    var y = cy + r * Math.sin(2 * theta);
    g.drawLine(x + 1, y, x + 1, y + 1);
    g.drawLine(x, y + 1, x + 2, y + 1);
    g.flip();
  }, rate);
});

Using Angular-UI and Restangular To Interact With WordPress API

Ben Wendt — Fri, 24 Apr 2015 20:00:00 -0400

At my new job I’ve been working a fair bit with ui router. It’s a fun library and I thought I would brush up a bit and have some fun practice interfacing it with the wordpress api plugin through restangular.

So let’s take a quick look at how this very simple process works.

First, I installed all the dependencies using bower.

Next, set up a simple index.html file:

<!doctype html>
<html ng-app="blog">
<head>
    <script src="bower_components/angular/angular.js"></script>
    <script src="bower_components/angular-ui-router/release/angular-ui-router.js"></script>
    <script src="bower_components/lodash/lodash.min.js"></script>
    <script src="bower_components/restangular/dist/restangular.js"></script>
    <script src="app.js"></script>
</head>
<body>
<div>
<div ui-view></div>
</div>
</body>
</html>

And here’s app.js:

var myApp = angular.module('blog', ['ui.router', 'restangular']);
myApp.config(function($stateProvider, $urlRouterProvider, RestangularProvider) {

RestangularProvider.setBaseUrl('http://benwendt.ca/blog/wp-json/')

$urlRouterProvider.otherwise('/');
$stateProvider
.state('home', {
  url: '/',
  templateUrl: 'posts.html',
  resolve: {
    posts:function(Restangular) {
      return Restangular.all('posts').getList()
     }
  },
  controller($scope, $sce, $filter, posts) {
    $scope.posts = posts
  }
})
}).filter('unsafe', function($sce) { return $sce.trustAsHtml })

The two main takeaways here are:

Set up the base api url using RestangularProvider.setBaseUrl().
Use ui-router’s $stateProvider to set up which template, api requests, and controller to use. Also I’m using a nifty filter I found for running <A href="http://stackoverflow.com/a/19705096/973810">$sce.trustAsHtml</a>.

And the template is super simple, but here it is:

<div ng-repeat="post in posts">
  <div class="post">
    <h1 ng-bind-html="post.title | unsafe"></h1>
    <div ng-bind-html="post.content | unsafe"></div>
  </div>
</div>

So, in conclusion, these technologies make it really easy to pull data off a json api and throw it into the browser. It’s wonderful.

Finding Semordnilaps

Ben Wendt — Fri, 17 Apr 2015 20:00:00 -0400

My wife recently developed an interest in semordnilaps, so I thought I would take a stab at writing a script that will find some. What I came up with finds a subset of all two word to two word semordnilaps. Generally you don’t consider whitespace and punctuation in palindromes and semordnilaps, so this code doesn’t either.

It turns out that filtering out proper names vastly reduces the number of semordnilaps found, and I didn’t really appreciate the ones which contained proper names anyway, so I took those out.

words = {}
File.open("/usr/share/dict/words") do |file|
  file.each do |line|
    if line.length > 4 # don't accept one or two letter words.
      word = line.strip
      next if word.downcase != word # proper names have capitals, exclude those.
      words[word.downcase.gsub(/[^a-z]/, '')] = true
    end
  end
end

words.each do |word1, k|  
  p_words = {}
  word_finds = 0

  word1 = word1.reverse
  first_words = []
  # skip words that when reversed don't match the ending of another word.
  (1 .. (word1.length - 1)).each do |i|
    if words[word1[0..i]]
      first_words << word1[0..i]
    end
  end
  if first_words.count == 0
    next
  end

  words.each do |word2, k|  
    two_words = word1 + word2.reverse    
    (2 .. two_words.length).each do |i|
       if words[two_words[0..(i - 1)]] && words[two_words[i..(two_words.length-1)]]
         p_words[two_words[0..(i - 1)] + ' ' + two_words[i..(two_words.length-1)]] = word2 + ' ' + word1.reverse
         word_finds += 1
       end
    end
  end
  if word_finds > 0
    puts p_words
  end
end

And this gives you some great semordnilaps like:

“cite catnip”=>”pint acetic”
“stub aloof”=>”fool abuts”
“tubas gals”=>”slags abut”
“sane railed”=>”deli arenas”
“reis trailed”=>”deli artsier”
“diva lived”=>”devil avid”
“stabs faced”=>”decafs bats”

All told, I got a 2.7MB text file. I am sure that there are better ones in there.

PHP’s traits vs. ruby’s modules: Battle of the mix-ins

Ben Wendt — Thu, 09 Apr 2015 20:00:00 -0400

A mixin is a class-like language construct meant to add functionality to another class. They are not meant to stand on their own, and generally speaking they cannot. Mixins can be used to give different classes the same interface. Mixins can be compared to multiple inheritence in what they let you accomplish, but they don’t work the same way. Rather than inheriting from multiple classes, you mix them in. (Hence the obvious name.)

PHP, always a late-comer feature-wise in terms of object oriented goodness, got traits in version 5.4.0. Traits allow the developer to add methods to a class with the use statement, like this:

interface Vehicle {
    public function move();
}

trait Wheel {
    public function move() {
        // roll.
    }
}

class Bicycle implements Vehicle {
    use Wheel;
}

$bicycle = new Bicycle;

$bicycle->move();

At this point, Bicycle will implement Vehicle. Even though you a move method is not explicitly specified in the class, the call on the object will work. Traits can only be assigned to a class in the class definition.

Ruby supports mixins using modules. That can be done like this:

module Wheel
  def move
    # roll.
  end
end

class Bicycle
  include Wheel
end

bicycle = Bicycle.new

bicycle.move

At this point, an instance of Bicycle will have a roll method, but the class Bicycle will not. This is because mixins get attached to the instance, not the class. This is the opposite of what happens in PHP.

In this way, PHP’s trait design differs fairly substantially from ruby’s mixin. In my mind, this is most likely because in PHP ideally all of your classes implement an interface. This is a nice feature because it helps a lot with static analysis. Ruby, on the other hand, doesn’t have interfaces as a language construct; classes do, of course, have interfaces but no part of the interpreter is enforcing signatures like the implements keyword makes PHP enforce method signatures. The standard argument against enforcing interfaces is that two unrelated classes can share a signature, so the signature doesn’t really tell you anything about the class.

In the end, neither approach is the ideal case for adding functionality to classes. I can’t think of an object oriented language that doesn’t support dependency injection. As long as you can set properties on an object and call methods, you should be able to do dependency injection. Developers should prefer composition over inheritance. It prevents code coupling: better to couple to a class in one decision than have coupled classes throughout your code base. Inheritance should be a fall-back, but you should always try to implement composition first.

A Whitespace + Punctuation Tokenizer

Ben Wendt — Fri, 13 Feb 2015 19:00:00 -0500

In my previous post, I discussed some tokenization techniques and mentioned that a whitespace-only tokenizer will make tokens that are sub-optimal for indexing. I also mentioned that a simple solution to this is created a whitespace + punctuation tokenizer.

So let’s take a look at how that might work.

import re

def whitespace_punctuation_tokenize(str, punctuation = "[\.,\"']"):
    tokens = re.split(punctuation + "*\s+" + punctuation + "*", str)
    if (tokens[0]):
        tokens[0] = re.sub("^" + punctuation + "+", "", tokens[0])
    if (tokens[-1]):
        tokens[-1] = re.sub(punctuation + "+$", "", tokens[-1])
    return tokens

You would run that code with something like this:

str = """'abc-123' is a cool one. It's far and away the
    ring-tossingest toy this year."""

print whitespace_punctuation_tokenize(str)
print whitespace_punctuation_tokenize(str, "[\.,\"]")

From this, you would see output like the following:

[‘abc-123′, ‘is’, ‘a’, ‘cool’, ‘one’, “It’s”, ‘far’, ‘and’, ‘away’, ‘the’, ‘ring-tossingest’, ‘toy’, ‘this’, ‘year’]

[“‘abc-123′”, ‘is’, ‘a’, ‘cool’, ‘one’, “It’s”, ‘far’, ‘and’, ‘away’, ‘the’, ‘ring-tossingest’, ‘toy’, ‘this’, ‘year’]

Here we see a function that does a regular expression split on an input string and accepts a configurable parameter of which characters to consider as punctuation. In this way, we can specify how want tokens to be delimited with a bit more granular control.

The function makes this split, then corrects and leading punctuation on the first element and any trailing punctuation on the last element. At this point you would have higher quality tokens to pass into your analysis chain.

Android Game Development

Fri, 13 Feb 2015 19:00:00 -0500

Wanting to expand my horizons a little, I decided to give android game development a try. I quickly prototyped a game in canvas + js called protect your thing to give myself an idea of how it would work on the android platform. This was a natural prototyping choice for me because I’ve been working with JS code daily for at least a decade now, and I love playing with canvas. (I frequently will whip up a little graphic in canvas reminiscent of my early QBasic coding days, but flavoured by the years of studying mathematics that followed).

With a prototype of a slightly enjoyable game complete, I set out on the path of android game development. Sereptitiously, the android graphics library has a canvas class that enabled all the stuff I’m used to being able to do in js. Beyond that, I looked up how to make an event loop and started hacking. The result is half-decent. It’s not a AAA game, and it has a few warts, but it was an excellent learning experience and I’m happy and proud to have made it. Check it out:

Protect Your Thing!

Basic Concepts of a Search Index

Ben Wendt — Fri, 30 Jan 2015 19:00:00 -0500

Introduction

A text index is a way to store information about textual data to improve ease of retrieval. The basic idea is to store the information in such a way that it is possible to retrieve what you are looking for without doing a full scan of the data. A text index will do this by storing information about each indexed word separately. So where in the raw text you might have a text field with the contents “this is a description”, a full text index would store information about each individual word separately, so it would store a form of “this”, “is”, “a”, and “description”.

There are two motivations to using an indexing engine: speed and quality of results. Generally speaking, sifting through a massive amount of text is a slow process. (We discussed that earlier on this blog when we looked at the Knuth-Morris-Pratt algorithm). There are also numerous intricacies of language that can trip up a naive search that a search index can work around.

So let’s dive in, shall we?

Tokenizing

The first stage of generating an index is to split the source text up into individual words or phrases. This stage is called “tokenizing.” A simple way of tokenizing text is the “white space tokenizing” technique, which basically splits text up by whitespace, as I did with “this is a description” earlier. But punctuation is a pitfall here. Without considering punctuation, tokenizing a phrase like “this is a pizza, and the pizza is good” would create separate and distinct tokens for each “pizza,” and “pizza”. This is generally not a desired outcome, so most tokenization algorithms will allow for a whitespace tokenizer that also accepts a list of punctuation characters.

There is a lot more room for nuance in tokenization: for example what if you want to use a whitespace+punctuation tokenizer, but you have IP addresses that you want to be searchable in your data? Then tokenizing something like “The IP was 127.0.0.1″ would yield separate tokens for “127”, “0” and “1”.

Another possible issue is that you may want multiple terms to be treated like single terms. For example, records that contain the phrase “magnetic resonance imaging” should be treated as more relevant in searches when the terms all appear together than when they are apart. There are different ways to approach this problem, but one is “NGram” tokenizing (this is a fancy way of saying that a given number N of adjacent terms will each be treated as one token).

The myriad nuances of tokenization are why industry leading search indices allow a developer to define their own tokenizing routine.

Analysis

Fully featured search indices like ElasticSearch or Solr will allow a software developer to define an analysis chain on their indexed data. The analysis chain is a way of normalizing the language that will be indexed so that related terms will be able to match after the index has been created.

Once a collection of tokens is generated, further processing is generally desired before storing indexed terms. If a user searches for “skeet shooting”, records with “skeet shoot” should probably match. Generally this issue is tackled by storing “stemmed” words in the index, where the “stem” of a word is generally a form of the word with any suffixes removed. A popular stemmer is the “Lovins Snowball stemmer.” (I recommend reading this link; it’s well-written and highly fascinating.)

There are two important points to consider here:

Stemming is language specific. Suffixes that denote different meaning vary from language to language. And pictogram based languages are (to my knowledge) practically immune to this approach to stemming.
Simple suffix removal is not a foolproof method of word stemming, for example “flammable” and “inflammable” have precisely the same meaning, but removing their suffixes leaves different stems. You wouldn’t want someone using your index to search for whether a product is flammable, and find no results because it is inflammable.

The solution to the first issue is to use different stemming algorithms for different languages. The solution to the second issue is to support synonyms in the search index; if the system is aware that flammable and inflammable represent the same thing, your users will find the safety information they need even if they use the term that isn’t in your data.

Proper analysis is where an indexing engine will beat out a text search in terms of quality of results.

Storing

Storing indexed data can be done in any number of ways. The important thing is that lookups should be fast. Index sizes and data concerns will determine which data structure is best. For my demo code below I am using PHP so that basically limits me to using PHP’s built-in array data structure as a hash map.

An example index

Here is a couple functions that create a simple index:

file_put_contents('index.dat', serialize(createIndex($records));

/**
* @param $record array collection of records
* @return array record collection combined with indices
*/
function createIndex($record) {
    $output = ['records' => $records, 'index' => []];
    foreach($records as $recordKey => $record) {
        $record = str_replace(array('.', ','), ' ', $record);
        $record = strtolower($record);
        $tokens = whitespaceTokenize($record);

        foreach($tokens as $token) {
            if (!isset($output['index'][$token])) {
                $output['index'][$token] = [];
            }
            $output['index'][$token][$recordKey] = 1;
        }
    }
    return $output;
}

/**
* @param $records array|string record or records to be split by white space.
* @return array tokenized collection
*/
function whitespaceTokenize($records) {
    $out = [];
    if (is_string($records)) {
        $records = [$records];
    }
    foreach($records as $record) {

        $out = array_merge($out, preg_split('/\s+/', $record));
    }
    return $out;
}

Note how the indexer splits out terms then stores them in a hash that maps back to which record the term can be found in. This is the basic idea of a search index.

And here is a quick way to read matches from the index:

$index = unserialize(file_get_contents('index.dat'));

$keywords = strtolower($argv[1]);

$matches = array_keys($index['index'][$keywords]);
echo count($matches) . ' matches' . "\n";
foreach($matches as $matchKey => $match) {
    echo "$match => " . $index['records'][$match] . "\n";
}

You will note here that I did not create a configurable punctuation + whitespace tokenizer. And in fact, none of the more advanced features I discussed earlier are implemented. This code is only meant to illustrate how an index works, not be fully featured. The basic concept here could be cleaned up, coded to a generalized storage interface, and then the more advanced features could be added as needed to make a functioning search index written in pure PHP.

Conclusion

We have seen a mile-high overview of how a search index works, discussed the tokenization and analysis processes, briefly discussed storage, and seen just about the most basic working example of a search index. Search indices are a fascinating technology that empower users through services like Bing, Million Short, and Duck Duck Go to find information on the internet, but we have also identified some issues in the technology and hinted at some methods of dealing with these problems. We may take a more in-depth look at some of those solutions in a future post.

Installing VLC on an ADT-1

Ben Wendt — Tue, 21 Oct 2014 20:00:00 -0400

The ADT-1 is a neat little machine. It’s very sleak and designy.

I signed up for the developer version of Android TV because I’d like to port a game I wrote to the platform. And Google was nice enough to send one over.

But out of the box it doesn’t allow streaming videos over your home network, which seems like a must for an HTPC. The user interface on the ADT-1 is pretty much locked down. Google play is limited to very few titles. So let’s set up VLC.

Enable USB debugging on your ADT-1. Just go into your developer settings and turn it on. Connect to the ADT-1 using adb. I did this over my network with the following command:

adb connect {ip}:4321

Download a VLC nightly APK. Install the apk using adb:

adb install *.apk

This might take a minute. My wi-fi is as slow as molasses.

If you want to be fancy and start VLC over the network. This can be done in two stages. First, shell in to your ADT-1:

adb shell

Now you can load VLC with am:

am start -n org.videolan.vlc/org.videolan.vlc.gui.MainActivity

Or if you’re into doing things the easy way, you to Apps under settings and you should see VLC.

And that’s it. Now if you walk over to your ADT-1 you will see the bright orange face of VLC smiling at you. Have a nice day!

Canvas Spinny

Ben Wendt — Thu, 04 Sep 2014 20:00:00 -0400

This will make a nice semi transparent spinner for you.

var spinnyCanvas = function(left, top) {
    var conti = false;
    var self = this;
    this.stop = function() {
        this.conti = false;
    }
    this.start = function() {
        console.log(self.conti)
        if (!self.conti) {
            self.conti = true;
            draw();
        }
    }
    var dim = 48
    var canvas = document.createElement('canvas')
    var context = canvas.getContext('2d')
    document.body.appendChild(canvas)
    canvas.height= dim
    canvas.width = dim
    canvas.theta = 0
    canvas.dtheta = -.09
    canvas.style.position = 'fixed'
    canvas.style.top = top + 'px'
    canvas.style.left= left + 'px'

    var draw = function() {
        canvas.width = canvas.width
        var r = dim / 16
        var x, y
        var i, L = 5
        for (i = 0; i &lt; L; i++) {
            x = dim / 2 + dim * .3 * Math.sin(canvas.theta + i * 1.1 * Math.PI / L)
            y = dim / 2 + dim * .3 * Math.cos(canvas.theta + i * 1.1 * Math.PI / L)
            context.beginPath()
            context.fillStyle = "rgba(128,128,128," + (1 - i / L) + ")"
            context.arc(x, y, r, 0, 2*Math.PI, false)
            context.fill()
            context.closePath()
        }
        canvas.theta += canvas.dtheta
        if (self.conti) {    
            requestAnimationFrame(draw)
        }
    }
    return this
}

Emulating the Javascript With Statement in PHP

Ben Wendt — Tue, 01 Jul 2014 20:00:00 -0400

Javascript has a with statement that you probably shouldn’t use. I’ve never seen it used non-jokingly in JavaScript in the past decade or so, other than the occasional clever hack. Visual Basic also has a with statement, and I did see it used a fair bit in that realm, back in the day. In my experience it’s not something that developers are clamoring for.

The main advantage of using with is not having to retype the name of the object you are working with repeatedly in a block of code. The drawback is that this harms readability; an assortment of new variables are presented in the block, and the scope has metaphorically changed gears. A big part of writing readable code is maintaining a good flow and preventing context switches. Because of this, with use is rare.

However, suppose you did want to implement some PHP code where you didn’t want repeated array or object references and you didn’t want to pollute your scope with a call to extract. You could use the following abomination:

call_user_func(function () use ($withVariable) {
    if (is_object($withVariable)) {
        $withVariable= get_object_vars($withVariable);
    }
    extract($withVariable);
    // do stuff.
});

The cumbersome use keyword and its white-list approach to close scope make this a difficult and cumbersome block of code. PHP doesn’t allow immediate execution of anonymous functions directly, so we have to pass the function to call_user_func. In other languages, with will bring variables from the outer scope into the with scope, but that will not happen here unless you add the desired variables into to use arguments.

In conclusion, with is a misfeature and attempting to implement it in PHP is neither very fruitful or elegant.

Migrating SQLite for PHP 5.2 to PHP 5.4 on Ubuntu

Ben Wendt — Fri, 13 Jun 2014 20:00:00 -0400

PHP 5.4 discontinued support for SQLite 2 databases. Updating an old legacy PHP application with a SQLite database to a new server is not very difficult.

To start off, you will need the SQLite binaries so that you can convert the database between versions:

sudo apt-get install sqlite sqlite3

Now that you have the binaries installed, you can convert your version 2 SQLite database to version 3:

sqlite version2.db .dump | sqlite3 version3.db

Now you will need to update your PHP script with the following translations

Old command

New command

sqlite_escape_string()

SQLite3::escapeString()

sqlite_fetch_array($result)

$result->fetchArray()

sqlite_exec($handle, $query)

$handle->exec($query)

sqlite_query($handle, $query)

$handle->query($query)

$handle = sqlite_open($file)

$handle = new SQLite3($file)

A Lottery Model

Ben Wendt — Wed, 11 Jun 2014 20:00:00 -0400

Having studied math in university, I am well aware that buying lottery tickets is a losing proposition. In Ontario, where I live, there is a lottery called 6/49. The rules are simple: pick 6 numbers out of 49 possibilities, if your choice matches a random choice of 6 numbers made in the draw, you win the jackpot. The probability of winning this is:

I.e. the odds are roughly one in 14 million.

However, when you live in a densely populated city like Toronto, you notice that a lot of convenience stores have signs up in their windows proclaiming the jackpot winners who have bought tickets there. Generally these take the form of big banners reading something along the lines of “$16 million dollar ticket sold here!”

Each neighbourhood in the city only has so many convenience stores, each serving a neighbourhood that only has so many people buying lottery tickets, each of whom may or may not play in every draw. It was enough to make me wonder how long a given population size of occasional lottery players would take before one of their members recorded a win.

So I set up a model with the following characteristics:

A configurable population size, I start with a population of 100,000, roughly the population of Guelph, which is a city near where I grew up.
A configurable probability for population wide ticket buying. For the sake of my model, I chose 1/8, which is slightly more than one ticket buy per month (there are bi-weekly draws).
Every week there is a draw, and 6 numbers are chosen from the 49. Every member of the population may or may not play. If they play, they also choose 6 numbers randomly from the 49, if these match, they win and the simulation ends.

Here’s my PHP code that represents the model I’ve described.

<?php
function getRandomChoice($from, $number, $min = 1) {
    $from = range($min, $from);
    $choices = [];
    for ($i = 0; $i &lt; $number; $i++) {
        $index = mt_rand(0, count($from) - 1);
        $choices[] = $from[$index];
        unset($from[$index]);
        $from = array_values($from);
    }
    sort($choices);
    return $choices;
}

$population = 100000;
$playProbability = 1 / 8;

$match_found = false;
$count = 0;
while(!$match_found) {
    $r = getRandomChoice(49, 6);
    $bought = 0;
    for ($i = 0; $i &lt; $population; $i++) {

        $rand = mt_rand() / mt_getrandmax();
        if ($rand &gt; $playProbability) {
            continue;
        }
        $s = getRandomChoice(49, 6);
        $bought ++;
        if ($r == $s) {
            echo "match after $count draws, population member $i\n";
            echo implode(" ", $r) . ", "  . implode(" ", $s) . "\n";
            break 2;
        }

    }
    $count++;
    if ($count % 52 == 0) {
        $year = $count / 52;
        echo "tried $count times, $bought bought, year $year.\n";
    }
}

What I found from this was not very heartening for lottery players. It generally takes several years for even one member of the 100000 population to ever win the jackpot. But I do think that the results demonstrate why there are so many of those jackpot banners around town.

Counting sort

Ben Wendt — Thu, 08 May 2014 20:00:00 -0400

I have a thing for sorting algorithms. They’re fairly accessible as far as algorithms go, and it’s always fun to look under the hood of how computers go about their business. Thinking algorithmically makes you a better programmer too.

So I was interested to see this post on hacker news about count sort in C, and decided to implement it in languages more within my milieu.

PHP

function count_sort($in) {
    $counting = array_fill(0, max($in), 0);

    for ($i = 0, $l = count($in); $i &lt; $l; $i++) {
        $counting[$in[$i] - 1]++;
    }
    for ($i = 0, $j = 0, $l = count($counting); $i &lt; $l; $i++) {
        while ($counting[$i] &gt; 0) {
            $in[$j] = $i + 1;
            $counting[$i]--;
            $j++;
        }
    }

    return $in;
}

Javascript

function count_sort(a) {
    var i, j, l = Math.max.apply(null, a), counting = [];    
    for (i = 0; i &lt; l; i++) {
        counting[i] = 0;
    }
    l = a.length;

    for (i = 0; i &lt; l; i++) {
        counting[a[i] - 1]++;
    }
    for (i = 0, j = 0, l = counting.length; i &lt; l; i++) {
        while (counting[i] &gt; 0) {
            a[j] = i + 1;
            counting[i]--;
            j++;
        }
    }

    return a;
}

python

    def countsort(a):
    counts = [0] * max(a)
    l = len(a) 
    for i in range(0, l):
        counts[a[i] - 1] += 1
    j = 0
    l = len(counts)
    for i in range(0, l):
        while counts[i] &gt; 0:
            a[j] = i + 1
            counts[i] -= 1
            j += 1
    return a

c

static int[] countSort(int[] a) {
            int i, j, l = a.Count();
            int[] counting = new int[a.Max()];    
            for (i = 0; i &lt; l; i++) {
                counting[i] = 0;
            }

            for (i = 0; i &lt; l; i++) {
                counting[a[i] - 1]++;
            }
            for (i = 0, j = 0, l = counting.Count(); i &lt; l; i++) {
                while (counting[i] &gt; 0) {
                    a[j] = i + 1;
                    counting[i]--;
                    j++;
                }
            }

            return a;
        }

powershell

Function countSort ($in) {
    $l = $in | Measure -Maximum
    $l = $l.Maximum
    $counting = @()

    for ($i = 0; $i -lt $l; $i++) {
        $counting += 0;
    }

    $l = $in.length

    for ($i = 0; $i -lt $l; $i++) {
        $counting[$in[$i] - 1]++;
    }

    $l = $counting.length
    $j = 0

    for ($i = 0; $i -lt $l; $i++) {
        while ($counting[$i] -gt 0) {
            $in[$j] = $i + 1;
            $counting[$i]--;
            $j++;
        }
    }

    return $in;

}

Conclusion

The sort only works with positive integers.

As it mentions in the blog I linked that inspired this column, the time and space of the algorithm are O(n + k), where k is the maximum element size in the array. That’s the real problem with the algorithm. If you sort something like [1, 3, 5, 1287614], you end up with a count array of 1287614 elements, which is definitely excessive for the array being sorted. Sorting an array shouldn’t fail because the elements in the array are too big. Running the PHP sort in a standard configuration with 5000000 in the input array will cause an out of memory exception.

Using the HSL colour space on Android

Ben Wendt — Tue, 22 Apr 2014 20:00:00 -0400

For my fun projects that include color, I like to use the following palette in the HSL space:

random hue
full saturation
50% lightness

This is easy in HTML because HSL is in the CSS specification and it has great cross browser support.

It’s a tiny bit harder in an Android app.

Java on Android uses the rgb color space by default, but there are some handy methods for using HSL.

float[] colors = {(float)Math.random() * 360,255,127}; 
int rgb = Color.HSVToColor(colors); 
r = Color.red(rgb); 
g = Color.green(rgb); 
b = Color.blue(rgb);

Using Underscore.PHP and Newton's Method to approximate pi

Ben Wendt — Tue, 15 Apr 2014 20:00:00 -0400

Underscore.js has been ported to Underscore.PHP.

For a simple example, let’s using Newton’s method to approximate pi.

$iterations = 20;
$x = 3;

$_ = new __();

$f = function($x) {return 1 + cos($x);};
$g = function($x) {return -sin($x);};
$h = function() {
    global $x;
    global $f;
    global $g;
    $x = $x - $f($x)/$g($x);
};

$j = [];

for ($i = 0; $i &lt; $iterations; $i++) {
    $j[] = $h;
}

$_->each($j, function($k) {$k();});

echo $x;

Setting up a simple blog using laravel

Ben Wendt — Tue, 08 Apr 2014 20:00:00 -0400

Setting up a very simple blog with laravel is quite easy.

Installing Laravel

The easiest install method is to use composer.

Setting up routing

Edit your /app/routes.php and add the following routes:

Route::get('/', function()
{
    $posts = Post::orderBy('id', 'DESC')->get();
    return View::make('blog')->with('posts', $posts);
});


Route::get('post/{id}', function($id) {
    $post = Post::find($id);
    return View::make('post')->with('post', $post);
})->where('id', '[0-9]+');

What this is saying is that the root request should use the “blog” view, using the collection of all blog posts, ordered by descending id.

It also sets up an option to load individual posts at /post/{id}, and the id is validated to be only integers.

Setting up views

In /app/views create a file called layout.blade.php which contains the following:

<!doctype html>
<html>
    <head>
        <title>@yield('title')</title>
    </head>
    <body>
        <h1>@yield('title')</h1>

        @yield('content')
    </body>
</html>

This is just a wrapper template where the title and yield values can be swapped in from other templates.

Next, in /app/views create a file called blog.blade.php, which contains the following:

@extends('layout')
@section('title')
    The Blog @stop
@section('content')
    @foreach($posts as $post) 
        <h2>
            <a href="/laravel-blog/post/{{ $post->id }}">
                {{ $post->title }}
            </a>
        </h2>
        <div class="post">{{ $post->content }}</div>
    @endforeach
@stop

And make another view called post.blade.php containing the following:

@extends('layout')
@section('title')
    {{ $post->title }} @stop
@section('content')
    {{ $post->content }}
@stop

These view set up values for title and content. In the blog view, we show all the posts, and in the post view, we only show one. This mirrors what we set up in the routes.

Setting up the database

First, configure your MySQL database in app/config/database.php. Now you have to set up a migration. The migration that you set up will create the tables. When you installed laravel, you will have had the artisan command line app set up for you. Run the following command:

php artisan migrate:make create_posts_table

This will have made a file in the /app/database/migrations folder with a name like {datetime}_create_posts_table.php. In this file, there will be a class CreatePostsTable which will have an up and a down method. Replace these methods with the following:

public function up()
    {
        Schema::create('posts', function($table)
        {
            $table->increments('id');
            $table->string('title');
            $table->string('content');
            $table->timestamps();
        });
    }

    public function down()
    {
        Schema::drop('posts');
    }

Save the file, then from your command line, run:

php artisan migrate

This will create the table in your database (as configured in app/config/database.php) as configured in the up method of your migration class.

Conclusion

You have now set up a very simple, non-user friendly, and extremely plain-looking blog. You could add posts by making inserts directly to the table, or by setting up a new route that creates a Post then calls the save() method on it.

Check out my FirefoxOS game Protect Your Thing!

Ben Wendt — Sun, 23 Mar 2014 20:00:00 -0400

FirefoxOS development is really easy if you have experience with front-end web development. Apps are very similar to chrome extensions, with .manifest files set up to control the application, and html / css to handle the content and appearance.

With this technology, it’s pretty easy to shoehorn a webpage into an app. So if created an HTML app using canvas, as I have done with my Protect Your Thing game, you can use something like the FirefoxOS boilerplate app to get yourself up and running quickly. Adding your app to the marketplace is as easy as uploading a zip file and filling out a form. Unlike the Google Play Store, all apps are QA checked before going online, so expect a 12-24 hour delay.

If you have a FirefoxOS device, you can install Protect Your Thing! for FirefoxOS.

A key-value store that forgets

Ben Wendt — Wed, 26 Feb 2014 19:00:00 -0500

Humans, like computers, have long term and short term memory. An interesting feature of human memory is that if you don’t use a memory for a while, you will eventually forget it.

So, for example, assume that 20 years ago you read War and Peace. Rather than keep the details of all 1000 pages in your memory, your brain sees that the memory hasn’t been used in a while, and eventually it forgets the details of the book.

This memory feature could be useful for AI in a video game, or for clearing out old memory that hasn’t been used in a while. In the latter case, you’d want to have some re-lookup functionality added to your code. In our analogy this would be akin to re-reading the book after having forgotten the details.

We can model this behaviour in javascript using an interval that clears out old unused memories.

var Forgettable = function(duration) {
    var forgettable = {
        "duration" : duration,
        "values" : {},
        "timeouts" : {}
    };
    forgettable.set = function (k, v) {
        if (!v) {
            return;
        }
        this.values[k] = v;
        this.timeouts[k] = this.duration;
    };
    forgettable.check = function() {
        var i;
        for (i in this.timeouts) if (this.timeouts.hasOwnProperty(i)) {
            if (this.timeouts[i] == 0) {
                delete this.timeouts[i];
                delete this.values[i];
            } else {
                this.timeouts[i] --;
            }
        }
    };
    forgettable.get = function(k) {
        var value = this.values[k];
        this.set(k, value);
        return value;
    };
    (function(f) {
        setInterval(function() {
            f.check();
        }, 1000);
    })(forgettable);
    return forgettable;
};

The data structure above works by having a duration of time that it will retain a memory. When a certain memory hasn’t been looked up in a certain amount of time, it will be deleted.

Rather than using the interval, the same code could be implemented by moving the contents of the check function in before the contents of the get function, or using timeouts.

Getting Started With Espruino

Ben Wendt — Thu, 06 Feb 2014 19:00:00 -0500

I was excited yesterday to see that my Espruino, which I backed on Kick Starter, had arrived in the mail. Espruino is micro controller that is controlled via JavaScript. I’ve long wanted to experiment with micro controllers, having visions of working with arduinos or a raspberry pi, but the JavaScript control combined with the reasonable price tipped the scale in favour of Espruino.

There is an excellent Quick Start Guide on the Espruino site that got me started in a flash. You can control the LED lights on the board by calling things like digitalWrite(LED1, 1).

There are three LED variables, so I decided to write an interval to loop through them:

var state = 0,
    interval = setInterval(function() {
      state = (state + 1) %3;
      digitalWrite(LED1, 0);
      digitalWrite(LED2, 0);
      digitalWrite(LED3, 0);
      switch (state) {
        case 0:
          digitalWrite(LED1, 1);
          break;
        case 1:
          digitalWrite(LED2, 1);
          break;
        case 2:
          digitalWrite(LED3, 1);
          break;
      }
    }, 50);

And here it is in action:

And here is code to turn on and off LED1 when BTN1 is pressed:

var on = false;
setWatch(function(e) {
  on = !on;

  digitalWrite(LED1, on);
}, A1, { repeat: true, edge: "falling" });

Interfacing with an Espruino from c#

Ben Wendt — Thu, 06 Feb 2014 19:00:00 -0500

Communication with an Espruino is done by sending JavaScript in string format over a serial port interface. This can be done in c# using the System.IO.Ports.SerialPort class. You can see the default serial port connection settings for interfacing on the Espruino site.

The following code will check abevigoda.com to see if Abe Vigoda is still alive. If he is still alive, a blue light will turn on. If he has passed away, a red light will display.

using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO.Ports;
using System.IO;
using System.Net;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            WebClient client = new WebClient();

            Stream Abe = client.OpenRead("http://www.abevigoda.com/");
            StreamReader reader = new StreamReader(Abe);
            string HTML = reader.ReadToEnd();

            int LEDNumber;
            if (HTML.Contains("alive"))
            {
                LEDNumber = 3;
            }
            else
            {
                LEDNumber = 1;
            }

            SerialPort port = new SerialPort(
                "COM4",
                9600,
                Parity.None,
                8,
                StopBits.One
            );

            port.Open();
            port.Write("digitalWrite(LED" + LEDNumber + ", 1);n");
            port.Close();

        }
    }
}

Here’s the current results:

Adding Arbitrary Code to your in Magento

Ben Wendt — Wed, 05 Feb 2014 19:00:00 -0500

Magento’s built in addJs method in the Mage_Page_Block_Html_Head class assumes that files will be hosted locally. This isn’t always desirable. E.g. you may want to use Google’s CDN to host jQuery.

This can be accomplished by adding the following to your layout xml:

<reference name="head">
            <block type="core/text" name="my_head"></block>
        </reference>

Now, in your controller you can add anything into this block that you want, like so:

$google_url = 'http://google.com/jquery.js';
$this->getLayout()->getBlock('my_head')->setText("

            <script src='$google_url'></script>

        ");

An implementation of the memento pattern in PHP

Ben Wendt — Sun, 08 Dec 2013 19:00:00 -0500

The memento pattern is a design pattern used to store and revert states for objects which support this capacity. It is accomplished by having a Caretaker object which manages a set of states, encoded in Memento objects. The Memento objects handle the storage of state; the implementation of this can vary, but it necessitates some level of deep-copying the object. A shallow copy will not suffice in general because it will not always capture the whole state of an object, due to the fact that most languages implement memory access for objects as references. Because of this, I use serialize and unserialize in my example below. Of course you could use other methods, like clone or just copying what you know you will need if memory is a concern.

Let’s take a look at how it works…

/**
* The memento class is very simple; it simply serializes and
* unserializes incoming data.
*/

class Memento {
    private $state = null;
    public function __construct($state) {
        $this->state = serialize($state);
    }
    public function revertState() {
        return unserialize($this->state);
    }
}

/**
* The Caretaker class manages a group of Memento objects.
*/

class Caretaker {

    private $states = array();

    public function set($state) {
        $this->states[] = new Memento($state);
    }

    public function get() {
        $memento = array_pop($this->states);
        return $memento->revertState();
    }

}

And here is a sample of usage:

$ct = new Caretaker();

$ct->set('3');
$ct->set('2');
$ct->set('1');
$ct->set(new stdClass);

var_dump($ct->get());
echo $ct->get() . "n";
echo $ct->get() . "n";
echo $ct->get() . "n";

Weighted merging of multiple Markov Chains

Ben Wendt — Tue, 03 Dec 2013 19:00:00 -0500

Suppose you have ten text sources, and you generate a new block of text trained from each, and you want to give each its own weighting. You have to weave multiple markov chains together. Here is a class that does just that.

class MultiMarkov {
    private $files;
    private $chains = array();
    private $weights = array();
    private $total_weight = 0;
    public function __construct($init) {
        $total_weight = 0;
        foreach($init as $label => $file_settings) {
            $file = $file_settings['file'];
            $weight = $file_settings['weight'];
            if (!is_integer($weight) || $weight &lt;= 0) {
                throw new Exception (
                    "Weight $weight is not a positive integer.");
            }
            $total_weight += $weight;
            $this->weights[$label] = $weight;
            $words = get_all_words_in_file($file);
            $chain = new Chain($words);
            $this->chains[$label] = $chain;
        }
        $this->total_weight = $total_weight;
    }
    public function setWeights($weights) {
        foreach($weights as $label => $weight) {
            $this->weights[$label] = $weight;
        }
    }
    public function getWeightedRandomChain() {
        $rand = rand(0, $this->total_weight);
        $running_total = 0;
        foreach($this->weights as $label => $weight) {
            $running_total += $weight;
            if ($running_total > $rand) {
                break;
            }
        }
        // echo "$labeln";
        return $this->chains[$label];
    }
    public function getRandomWord() {
        $from_chain = $this->getWeightedRandomChain();
        return $from_chain->getRandomWord();
    }
    public function getChainOfLength($in_word, $length) {

        while (!isset($word) || $word == false) {
            $from_chain = $this->getWeightedRandomChain();
            $word = $from_chain->getNextWord($in_word);

        } 
        if ($length > 1) {
            return $word . $this->getChainOfLength($word, $length - 1);
        } else {
            return $word;
        }
    }
}

To use this class you could do something like so:


<?php 

include 'markov.php';
include 'multi-markov.php';

$files = array(
    'joyce' => array(
        'file' => 'text/joyce.txt',
        'weight' => 10
    ),
    'weitz' => array(
        'file' => 'text/weitz.txt',
        'weight' => 20
    ),
    'sontag' => array(
        'file' => 'text/sontag.txt',
        'weight' => 15
    ),
    'berger' => array(
        'file' => 'text/berger.txt',
        'weight' => 10
    ),
    'tolstoy' => array(
        'file' => 'text/tolstoy.txt',
        'weight' => 19
    ),

);

$builder = new MultiMarkov($files);
$starting_word = $builder->getRandomWord();

$newSentence = $builder->getChainOfLength($starting_word, 160);
echo wordwrap($newSentence, 70, "n");

This text is trained from a variety of great writers writing about aesthetics and the nature of art. Here’s a sample of the output:

praise worthy on the conditions under which has drawn it seemed to me that of reassembling what we would call these conditions ù a novel, painting, to its subject, remains the concept shows, has hardly broken into the standard of all other world, have painted the result of color that “This is often experienced as to himself; so, thanks to fate or love to see in the faculty of a common properties of the world that supreme quality of true painting or theory again. Of course,”Art is very perception of art. Whatis central as a fading coal. The destiny of the word has experienced, and everything else… For myself, I am speaking now that basket from everything else. Each claims that are made by accident when he experiences the oar or that its claim to be answered yes or even seem to be maintained is definable as works of application of movements, lines, colors, shapes, volumes – and sufficient conditions.

Registry Pattern

Ben Wendt — Sun, 17 Nov 2013 19:00:00 -0500

Here’s another super-simple design pattern, implemented in PHP.

class Registry {
    private $values = array();

    public function get($key) {
        if (!isset($this->values->$key)) {
            throw new OutOfBoundsException("$key not in registry");
        }
        return $this->values->$key;
    }

    public function set($key, $val) {
        if (!isset($this->values->$key)) {
            throw new OverflowException("$key already in registry");
        }
        $this->values->$key = $val;
    }
}

The registry pattern is used to store information that can be used throughout your application. You could use a registry to store a bunch of application settings, for example.

I’ve seen this implemented with the magic getters and setters in php, but then you end up with an object that appears to just be setting public properties. It’s not the most readable solution and it is unintuitive to expect an exception when setting a public property.

It is often implemented as a singleton, but it doesn’t have to be. In general singletons are bad because they introduce global state into your code and are hard to write test cases for. If you don’t want to use your registry as a singleton, just do an inversion of control. The downside is having to pass around the registry as a parameter, but trust me: it’s worth the effort.

Generating the vertices and edges of an n-cube in javascript

Ben Wendt — Fri, 25 Oct 2013 20:00:00 -0400

An n-cube is a geometric shape analogous to a cube, but in an arbitrary number of dimensions.

The algorithm I’ve laid out does a recursive routine for finding the vertices and then looks for all edges within one unit of each to define the edges.

The function becomes non-response with values over 14. This is because the number of points in an n-cube is given by 2ⁿ and the number of edges is n2^n-1. So the expected memory usage of this function is O(2ⁿ), which is going to blow up with any fairly large value.

var nCube = function(n) {

    if (n == 1) {
      return {
        "points" :[[0], [1]],
        "edges" : [[0, 1]]
      };
    } else {
      var L, i, j, prev = nCube(n - 1), out = {
        "dimensions" : n,
        "points" : [],
        "edges" : []
      };

      for (i = 0; i < prev.points.length; i++) {
        for (j = 0; j < 2; j++) {
          (function() {
            var fit = prev.points[i].slice();
            fit.push(j);
            out.points.push(fit);

          })();
        }
      }
      // surely there is a faster recursive method of defining edges, but...
      for (i = 0; i < out.points.length; i++) {
        for (j = i; j < out.points.length; j++) {
          if (i == j) continue;
          if (distance(out.points[i], out.points[j]) == 1) {
            out.edges.push([i,j]);
          }
        }
      }

      return out;
    }
  };

A PHP spell checker

Ben Wendt — Sun, 20 Oct 2013 20:00:00 -0400

How to Write a Spelling Corrector, by Peter Norvig, is a popular resource for instructions on how to produce spell check functionality. That page does some statistical Analysis of how this algorithm works and is definitely worth reading.

I’ve rewritten this alghorithm in PHP. It’s not as eloquent as the python code in the original, but I suspect it gains a tiny bit of readability in the trade-off for terseness.

A basic outline of the routine is as follows:

Train the spell checker in what words it should know. Any large block of text in the desired language that contains all the words you want to be spell-checkable, and no others, will do. This introduces a chicken-egg issue as you may have misspellings in your training text. Ideally, you would use a previously verified dictionary for this step. But you could use anything, like a bunch of crawled text from wikipedia, or this blog, or the complete works of Charles Dickens, or whatever reference you choose.
Generate a list of candidate words within a given Levenshtein distance of the word you are attempting to correct.
- All candidate words will be found within a predefined number of character deletions, character transpositions, and character insertions on the original word.
- Candidates will be in the trained text.
- A given candidate can be reached in more than one way depending on which deletions, transpositions, and insertions are performed. We will exploit this fact.
The list of suggestions will be ranked by which suggestions arise most frequently.

Choose one or more suggestions from your list as required

Here’s the PHP class:

<?php



class checker {
    private $nwords = array();
    private function words($text) {
        return preg_split(‘/s+/‘, $text);
    }
    public function train($words) {
        $this->nwords = array_flip($this->words($words));
    }
    private function edits1($word) {
        $word = strtolower($word);
        $alphabet = range(‘a’, ‘z’);
    $splits = array();
    for ($i = 1; $i &lt; strlen($word); $i++) {
        $splits[] = array(substr($word, 0, $i), substr($word, $i));
    }
    $deletes = array();
    foreach($splits as $split) {
        $deletes[] = $split[0] . substr($split[1], 1);
    }
    $transposes = array();
    foreach($splits as $split) {
        if (isset($split[1][1])) {
            $transposes[] = $split[0] . $split[1][1] . $split[1][0] . substr($split[1], 2);
        }
    }
    $replaces = array();
    foreach($alphabet as $letter) {
        foreach($splits as $split) {
            $replaces[] = $split[0] . $letter . substr($split[1], 1);
        }
    }
    $inserts = array();
    foreach($alphabet as $letter) {
        foreach($splits as $split) {
            $inserts[] = $split[0] . $letter . $split[1];
        }
    }
    return array_merge($deletes, $transposes, $replaces, $inserts);
}
private function edits2($word) {
    $edits2 = array();
    foreach($this-&gt;edits1($word) as $e1) {
        foreach($this-&gt;edits1($e1) as $e2) {
            if (isset($this-&gt;nwords[$e2])) {
                $edits2[] = $e2;
            }
        }
    }
    return $edits2;
}
private function known($word) {
    $known = array();
    if (isset($this-&gt;nwords[$word])) {
        $known[] = $word;
    }
    return $known;
}
public function correct($word) {
    $candidates = array();
    if ($this-&gt;known($word)) {
        $candidates[] = $word;
    }
    foreach($this-&gt;edits1($word) as $possible) {
        if ($this-&gt;known($possible)) {
            $candidates[] = $possible;
        }
    }
    foreach($this-&gt;edits2($word) as $possible) {
        if ($this-&gt;known($possible)) {
            $candidates[] = $possible;
        }
    }
    $counts = array();
    foreach($candidates as $candidate) {
        if (!isset($counts[$candidate])) {
            $counts[$candidate] = 0;
        } else {
            $counts[$candidate]++;
        }
    }
    $most = 0;
    $word = '';
    foreach($counts as $candidate =&gt; $count) {
        if ($count &gt; $most) {
            $most = $count;
            $word = $candidate;
        }
    }
    return $word;
}
}

Usage:

<pre class="brush: php; title: ; notranslate" title="">$spell = new checker();

$spell->train(file_get_contents(‘dictionary.txt’));

$corrections = $spell->correct(‘speling’);

echo “$correctionsn”;

A Lattice class

Ben Wendt — Tue, 08 Oct 2013 20:00:00 -0400

Consider if you need a 7-dimensional lattice data structure. An array is great for one-dimensional data but once you add dimensions things quickly get difficult to manage. Here’s a data structure that takes away some of that headache:

<?php

class Lattice {

    private $lattice = null;
    private $dimensions = 0;
    private $boundaries = null;

    public function setDimensions($d) {
        if (is_integer($d)) {
            $this->dimensions = $d;
        } else {
            throw new Exception('setDimensions expects an integer. given: ' . $d);
        }
    }
    public function getDimensions() {
        return $this->dimensions;
    }
    public function setBoundaries(array $boundaries) {
        $iteration = count($boundaries);
        if ($this->dimensions > 0 && $this->dimensions == count($boundaries)) {
            $this->boundaries = $boundaries;
            $array = null;
            while ($iteration > 0) {
                $iteration --;
                $array = array_fill(0, $boundaries[$iteration], $array);
            }
            $this->lattice = $array;
        } else {
            throw new Exception("Boundary count should match dimension count");
        }
    }
    public function setNode(array $coordinates, $value) {
        if (count($coordinates) != $this->dimensions) {
            throw new Exception("Passed coordinates have dimension mismatch");
        }
        $iteration = 0;
        $array = &$this->lattice;
        while ($iteration < count($coordinates) - 1) {
            $coordinate = $coordinates[$iteration];

            if ($coordinate > $this->boundaries[$iteration] - 1) {
                throw new Exception("coordinate $iteration is out of bounds");
            }
            $array = &$array[$coordinate];
            $iteration++;
        }
        $coordinate = $coordinates[$iteration];
        if ($coordinate > $this->boundaries[$iteration] - 1) {
            throw new Exception("coordinate $iteration is out of bounds");
        }
        $array[$coordinate] = $value;
    }
    public function getNode(array $coordinates) {
        if (count($coordinates) != $this->dimensions) {
            throw new Exception("Passed coordinates have dimension mismatch");
        }
        $iteration = 0;
        $array = &$this->lattice;
        while ($iteration < count($coordinates) - 1) {
            $coordinate = $coordinates[$iteration];

            if ($coordinate > $this->boundaries[$iteration] - 1) {
                throw new Exception("coordinate $iteration is out of bounds");
            }
            $array = &$array[$coordinate];
            $iteration++;
        }
        $coordinate = $coordinates[$iteration];
        if ($coordinate > $this->boundaries[$iteration] - 1) {
            throw new Exception("coordinate $iteration is out of bounds");
        }
        return $array[$coordinate];
    }
    public function toArray() {
        return $this->lattice;
    }
}

Using Shared Strings to Reduce Memory Usage

Ben Wendt — Thu, 26 Sep 2013 20:00:00 -0400

As of Excel 2007, files are saved in the Open XML format. This format is comprised of a grouping of XML files and assets, which are then zipped up and given the .xlsx extension. It’s a lot more readable from other programs than an old fashioned .xls file.

One means that was used to reduce the file size was setting up a shared strings table. Strings stored in a spreadsheet are given a numeric index and this numeric index is then stored in the xml file. In general, if a string is reused frequently the overhead of the shared string map will be payed off by the saving of only storing string indices.

For example, consider the following spreadsheet:

<td>
  reused string
</td>

<td>
  reused string
</td>

<td>
  <td>
    reused string
  </td></tr> 

  <tr>
    <td>
      other string
    </td>

    <td>
    </td>

    <td>
      reused string
    </td>

    <td>
      reused string
    </td>

    <td>
      other string
    </td>
  </tr>

  <tr>
    <td>
      reused string
    </td>

    <td>
      reused string
    </td>

    <td>
      reused string
    </td>

    <td>
      reused string
    </td>

    <td>
    </td>
  </tr>

  <tr>
    <td>
      reused string
    </td>

    <td>
      reused string
    </td>

    <td>
      reused string
    </td>

    <td>
      reused string
    </td>

    <td>
      other string
    </td>
  </tr>

  <tr>
    <td>
      other string
    </td>

    <td>
    </td>

    <td>
    </td>

    <td>
      reused string
    </td>

    <td>
    </td>
  </tr></table>

This could be shortened to:

1 => reused string
2 => other string

reused string

1	1	1		1
2		1	1	2
1	1	1	1
1	1	1	1	2
2			1

The Open XML format has other space savings, like not storing empty cells, but that is not relevant here.

Even with the clunky HTML table structure above, using shared strings has reduced the number of characters used to store this data from 547 down to 362.

The same idea can be applied when you have a program that is saving thousands of reused strings. Saving integers is much more efficient. The size of a PHP integer is platform dependent, but generally 32-bits, while a string will use 1 or more bytes per character. If you know you are dealing with something with a lot of string reuse, something like this can be useful.

Here’s an implementation in PHP. You can start by throwing strings into the collection by calling getIndex. Then, when you are ready to pull the strings back out, call getString. Note that the class has an insertion mode and an extraction mode, and that when it changes mode a call to array_flip is made on the internal listing of entries. This is done to speed up the process (by using hashing on both operations rather than doing array searches); the down side is that if you are frequently changing back and forth between reading and writing, it will be slow. It’s meant to be written to all at once, then read off later.

class SharedString {
    private $map = array();
    private $extraction_mode = false;
    public function count() {
        return count($this-&gt;map);
    }
    public function getIndex($string) {
        if ($this-&gt;extraction_mode) {
            $this-&gt;map = array_flip($this-&gt;map);
            $this-&gt;extraction_mode = false;
        }
        $string =  preg_replace('/s/', ' ', $string);
        $position = isset($this-&gt;map[$string]) ? $this-&gt;map[$string] : false;
        if ($position === false) {
            $position = count($this-&gt;map);
            $this-&gt;map[$string] = $position;
        } 
        return $position;

    }
    public function getString($index) {
        if (!$this-&gt;extraction_mode) {
            $this-&gt;map = array_flip($this-&gt;map);
            $this-&gt;extraction_mode = true;
        }
        return $this-&gt;map[$index];
    }
    static function mapArray(SharedString $string, $array) {
        foreach($array as $key =&gt; $value) {
            $array[$key] = $string-&gt;getIndex($value);
        }
        return $array;
    }
}

PHP doesn’t have great control over types, but you can make further memory reductions, as Open XML does, but not saving numerics as strings. These use less memory when stored as their appropriate type. E.g. in Ascii 12345 takes 5 bytes, but stored in a signed integer it only takes 16 bits.

The Knuth-Morris-Pratt algorithm implemented in JavaScript

Ben Wendt — Thu, 19 Sep 2013 20:00:00 -0400

The Knuth-Morris-Pratt string search algorithm is an algorithm for finding a substring within another string that uses information calculated about the substring to speed up the search process.

Before looking through the text to be searched, a table of jump lengths is calculated from the search string. Rather than iterating through the whole string, you can determine how many characters to skip over based on the structure of the word being searched for. As you scan through the search text and you find a mismatch, you can start comparing characters again from the last matching substring between the search term and the search text.

E.g. if you are searching through T-Bone by Neil Young for the words “Ain’t got no T-BonenT-Bone”, every time you find an instance of “”Ain’t got no T-Bonen” that isn’t followed by a line-break, you can skip 20 characters of checking the lyrics and restart from the beginning of your words, because you know that they can’t occur again within that span because you have already checked those and they don’t match the begging of the search words. Basically you already know that index is the next occurence of Ain’t, so you can start again from there.

var makeKMPTable = function(word) {
    if(Object.prototype.toString.call(word) == '[object String]' ) {
        word = word.split('');
    }
    var results = [];
    var pos = 2;
    var cnd = 0;

    results[0] = -1;
    results[1] = 0;
    while (pos &lt; word.length) {
        if (word[pos - 1] == word[cnd]) {
            cnd++;
            results[pos] = cnd;
            pos++;
        } else if (cnd &gt; 0) {
            cnd = results[cnd];
        } else {
            results[pos] = 0;
            pos++;
        }
    }
    return results;
};

var KMPSearch = function(string, word) {
    if(Object.prototype.toString.call(string) == '[object String]' ) {
        string = string.split('');
    }
    if(Object.prototype.toString.call(word) == '[object String]' ) {
        word = word.split('');
    }

    var index = -1;
    var m = 0;
    var i = 0;
    var T = makeKMPTable(word);

    while (m + i &lt; string.length) {
        if (word[i] == string[m + i]) {
            if (i == word.length - 1) {
                return m;
            }
            i++;
        } else {
            m = m + i - T[i];
            if (T[i] &gt; -1) {
                i = T[i];
            } else {
                i = 0;
            }
        }
    }
    return index;
};

var test = 'potential';

var string = "This fact implies that the loop can execute at most 2n times. For, in each iteration, it " +
    "executes one of the two branches in the loop. The first branch invariably increases i and does not " +
    "change m, so that the index m + i of the currently scrutinized character of S is increased. The second " +
    "branch adds i - T[i] to m, and as we have seen, this is always a positive number. Thus the location m " +
    "of the beginning of the current potential match is increased. Now, the loop ends if m + i = n; " +
    "therefore each branch of the loop can be reached at most k times, since they respectively increase " +
    "either m + i or m, and m = m + i: if m = n, then certainly m + i = n, so that since it increases by " +
    "unit increments at most, we must have had m + i = n at some point in the past, and therefore either " +
    "way we would be done.";

result = KMPSearch(string, test);

Replacing Removed Whitespace to Restore Words From a Dictionary

Ben Wendt — Wed, 11 Sep 2013 20:00:00 -0400

Consider a spell checker program: a user is entering input, and for whatever reason they miss some white space while entering input, so that something like The Sun Also Rises is entered as The Sun AlsoRises.

AlsoRises is not an English word, but obviously both Also and Rises are. Finding words within this string can be done using the following algorithm:

Choose a list of words to search for. This could be the whole English dictionary, or anything, really.
Sort these words in descending order of word length.
Look for each word and skip to the next unassigned slot when you find a match.
Any non-matching characters should be kept.

Here is an implementation of this algorithm. Ideally this would remember the disregarded tokens and wouldn’t lowercase everything, but as a proof of concept that is not necessary:

class Demungler {

  private $parts = array();
  private $shortest_word_length = 0;

  public function getDemungledString($string, $separator = ' ') {
    $string = strtolower($string);
    $parts = array();
    $tokens = token_get_all('&lt;?php ' . $string);
    foreach($tokens as $token) {
      if (isset($token[0]) && $token[0] == 307) {
        $parts[] = $token[1];
      }
    }
    $output = array();


    foreach($parts as $part) {

      $current_partial_word = '';
      $limit = strlen($part) ;- $this->shortest_word_length;
      for($i = 0; $i &lt; $limit; $i++) {
        $add_char = true;
        foreach($this->words as $word) {
          $search_string = substr($part, $i, strlen($word));

          if ($search_string == $word) {

            $i += strlen($word) - 1;
            if ($current_partial_word != '') {
              $output[] = $current_partial_word;
              $current_partial_word = '';
            }
            $output[] = $word;
            $add_char = false;
            break;
          }
        }
        if ($add_char) {
          $current_partial_word .= $part[$i];
        }
      }
      if (!empty($current_partial_word)) {
        $output[] = $current_partial_word;
      }

    }

    return implode($separator, $output);
  }

  public function setDictionary($words) {
    $words = array_map('strtolower', $words);
    usort($words, function ($a, $b) {
      return strlen($a) &lt; strlen($b);
    });
    $this->shortest_word_length = strlen($words[count($words) - 1]);
    $this->words = $words;
  }
}

And here is a test of it:

$d = new Demungler();
$d->setDictionary(
  array('a', 'are', 'fun', 'of', 'is', 'test', 'tests',
  'the', 'this', 'tokens', 'too')
);
$s = $d->getDemungledString('thisburgleisatestess ofthedemungler.;tokensarefuntestsarefuntoo');
echo "$sn";

The output of which is

this burgle is a test ess of the demungler tokens are fun tests are fun too

Note: It’s possible that you could find the wrong words using this algorithm. E.g. given the input “visiteductionalattractions”, the result would be “visited u cat ion al attractions”, instead of the intended “visit educational attractions”. The only way to fix this would be to try a variety of different words, when more than one fits, then complete the algorithm for each possibility, and see which result maximizes the average word length and the number of found words. Even so, you’re left in a best guess situation with many different possibilities.

A timer class

Ben Wendt — Sun, 08 Sep 2013 20:00:00 -0400

It’s often useful to time aspects of your applications. In environments without access to profiling tools like xdebug, it is necessary to roll your own. Here’s one that relies heavily on calls to microtime. Unfortunately making many thousands of calls to microtime takes a significant amount of time on its own.

But a manual timing class like this can still be quite useful in identifying problem blocks of code:

class Timer {

    private $starts = array();
    private $calls = array();

    private $times = array();

    public function start($identifier) {
        if (!isset($this-&gt;calls[$identifier])) {
            $this-&gt;calls[$identifier] = 0;
        }
        $this-&gt;calls[$identifier]++;
        $this-&gt;starts[$identifier] = microtime(true);
    }

    public function stop($identifier) {
        $end = microtime(true);
        if (!isset($this-&gt;times[$identifier])) {
            $this-&gt;times[$identifier] = 0;
        }
        $this-&gt;times[$identifier] += ($end - $this-&gt;starts[$identifier]);
    }

    public function getTimes() {
        return $this-&gt;times;
    }

    public function getCallCounts() {
        return $this-&gt;calls;
    }

}

A NavigableMap inspired class for PHP

Ben Wendt — Sun, 25 Aug 2013 20:00:00 -0400

Java has a nifty class named NavigableMap. It abstracts away some of the logic for mapping ranges to values. Java is reknowned for it’s ridiculously large library of collections.

Here’s a PHP implementation of some of its functionality.

class NavigableMap {
  private $data = array();
  private function checkKeyNumeric($key) {
    if (!is_numeric($key)) {
      throw new Exception("keys must be numeric. Given $key");
    }
  }
  public function put($key, $value) {
    $this-&gt;checkKeyNumeric($key);
    $this-&gt;data[$key] = $value;
    ksort($this-&gt;data);
  }
  public function firstEntry() {
    return key($this-&gt;data);
  }
  public function lastEntry() {
    $end = key($this-&gt;data);
    reset($this-&gt;data);
    return $end;
  }
  public function floorEntry($point) {
    return $this-&gt;data[$this-&gt;floorKey($point)];
  }
  public function floorKey($point) {
    $this-&gt;checkKeyNumeric($point);
    $lowest = $this-&gt;firstEntry();
    if ($lowest &gt; $point) {
      throw new OutOfBoundsException("no point exists " .
            "below $point. Lowest map entry is $lowest");
    }
    foreach($this-&gt;data as $key =&gt; $value) {
      if ($key &gt; $point) {
        return $previous_key;
      }
      $previous_key = $key;
    }
    return $previous_key;
  }
  public function ceilingEntry($point) {
    return $this-&gt;data[$this-&gt;ceilingKey($point)];
  }
  public function ceilingKey($point) {
    $this-&gt;checkKeyNumeric($point);
    $highest = $this-&gt;lastEntry();
    if ($highest &lt; $point) {
      throw new OutOfBoundsException("no point exists " .
            "above $point. Highest map entry is $highest");
    }
    $keys = array_reverse(array_keys($this-&gt;data));
    foreach($keys as $key) {
      if ($key &lt; $point) {
        return $previous_key;
      }
      $previous_key = $key;
    }
    return $previous_key;
  }
}

A Bloom Filter in c#

Ben Wendt — Thu, 15 Aug 2013 20:00:00 -0400

A bloom filter is a probabilistic data structure meant for checking whether a given entry does not occur in a list. It is meant to be quite fast, and is used as a way of not doing costly queries when it can be determined that no results will be returned. E.g., if you could turn this:

costly_lookup(key)

Into this:

if (!cheap_check_that_key_isnt_there()) {
    costly_lookup()
}

Then that’s a win. The way that a bloom filter goes about this is by performing a series of k hashes that return a value between 0 and n (in my example code below I use k=2 and m=32). When you add a value to your heavy data store, you will also run each of these hashes and store the return values in the bloom filters master list of returned values. Then when you want to see if something is in there, you run it through all of your hashes; if a value is returned that isn’t in your list of previous hash results you know that the new entry isn’t in your set. If only previously hashed values are returned the new item may be in there.

For a discussion of the rate of false positives, optimizing k and m, and a more in-depth discussion of this issue, I recommend Bloom Filters by Example by Bill Mill.

As is usual on my blog, below I will outline the general concept of this data structure. You should note that the values of m and k are hard-coded here, and that the hashing functions rotate and rotateMore aren’t proper hashing functions, they just illustrate the idea.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Collections;

namespace Bloom
{
    class Bloom
    {
        private BitArray bits = new BitArray(32);

        // two toy hashing functions,
        private Int16 rotateMore(String AddString)
        {
            Int16 ReturnValue = 0;
            for (int i = 0; i &lt; AddString.Length; i++)
            {
                ReturnValue += (Int16)((int)AddString[i] * i);
                ReturnValue = (Int16)(ReturnValue % 32);
            }
            return ReturnValue;
        }

        private Int16 rotate(String AddString)
        {
            Int16 ReturnValue = 0;
            for (int i = 0; i &lt; AddString.Length; i++)
            {
                ReturnValue += (Int16) ((int)AddString[i]);
                ReturnValue = (Int16) (ReturnValue % 32);
            }
            return ReturnValue;
        }
        public void add(String AddString)
        {
            Console.WriteLine("adding " + AddString);

            Int16 Point1 = this.rotate(AddString);
            Int16 Point2 = this.rotateMore(AddString);
            this.bits[Point1] = true;
            this.bits[Point2] = true;

        }
        public bool contains(String CheckString)
        {
            Int16 Point1 = this.rotate(CheckString);
            Int16 Point2 = this.rotateMore(CheckString);
            if (this.bits[Point1] && this.bits[Point2])
            {
                return true;
            }
            else
            {
                return false;
            }
        }
        public void checkFor(String key)
        {
            if (this.contains(key))
            {
                Console.WriteLine(key + " may be in there");
            }
            else
            {
                Console.WriteLine(key + " is not there");
            }
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            Bloom bloom = new Bloom();
            bloom.add("string");
            bloom.add("fresh");
            bloom.add("basketball");
            bloom.checkFor("basketball");
            bloom.checkFor("soccer");
            Console.ReadLine();
        }
    }
}

The output of this will be:

adding string
adding fresh
adding basketball
basketball may be in there
soccer is not there

Skip List Implementation in PHP

Ben Wendt — Mon, 12 Aug 2013 20:00:00 -0400

A skip list is similar to a linked list, but it is always sorted and maintains multiple pointers. The multiple pointers allow fast traversal of the list so that you can quickly look for elements, essentially performing a binary search on the data.

Here’s an implementation of skip list in PHP:

<?php 
class SkipList {

    // # implement a skip list with a given depth.
    // # this data structure is used to rapidly search
    // # through a multiply-linked (sorted) list,
    // # mimicking the functionality of a binary
    // # sort using references.

    // # The private properties here are the first
    // # node and the depth, which is the number
    // # of levels of references to maintain.
    // # E.g. a depth of 3 will maintain 3 levels of
    // # pointers, so that each subsequent level of
    // # reference will allow for more fine-grained
    // # access than the one before.

    private $_first = array();
    private $_depth = 0;

    // # We construct with an array. This is then
    // # sorted (so use a sortable array) and references
    // # are made for the given or predefined depth.

    public function __construct($list, $depth = null) {
        if (!is_array($list)) {
            throw new Exception(
                'SkipList constructor called with invalid list parameter'
            );
        }
        if (!is_numeric($depth)) {
            throw new Exception(
                'SkipList constructor called with invalid depth parameter'
            );
        }
        $this->_depth = $depth;
        sort($list);
        foreach($list as $lkey => $item) {
            $list[$lkey] = new SkipListNode($item);
        }
        for ($i = 0; $i < $depth; $i++) {
            for ($j = 0; $j < count($list) / pow(2, $i); $j++) {
                $index = $j * pow(2, $i);
                $next_index = ($j + 1) * pow(2, $i);
                $next_node = isset($list[$next_index]) ?
                    $list[$next_index] : null;
                $list[$index]->setNext($next_node, $i);
            }
        }
        $this->_first = $list[0];
    }

    public function getFirst() {
        return $this->_first;
    }

    public function getDepth() {
        return $this->_depth;
    }

    // # for a perfect skip-list implementation you'd
    // # reweight the references every so often, so
    // # that you don't end up with giant amounts of
    // # data between two references and very little
    // # between others. This implementation does not
    // # do that.

    public function insertValue($value) {
        // inserts the value at the lowest level.
        // ideally you'd shuffle everything around
        // once in a while to ensure it's still efficient.
        $new_node = new SkipListNode($value);
        $node = $this->_first;
        do {
            if (false) {
                 "oops never implemented. lol";
            } else {
                $node = $node->getNext(0);
            }
        } while ($node != null);
    }

    // # The whole point of this is to be able to tell quickly
    // # whether or not a list contains a given value.

    public function contains($value) {
        $found = false;
        $node = $this->_first;
        $depth = $this->_depth - 1;
        do {
            if ($node->getValue() == $value) {
                $found = $node;
            } if ($node->getValue() > $value) {
                break;
            } else if ($node->getNext($depth) != null && $node->getNext($depth)->getValue() > $value) {
                $depth--;
                continue;
            } else {
                $node = $node->getNext($depth);
            }
        } while ($depth >= 0 && $node != null);

        return $found;
    }

}

class SkipListNode {
    private $_value;
    private $_nexts = array();
    public function setValue($value) {
        $this->_value = $value;
    }
    public function getValue() {
        return $this->_value;
    }
    public function getNext($depth) {
        return $this->_nexts[$depth];
    }
    public function setNext($node, $depth) {
        $this->_nexts[$depth] = $node;
    }
    public function __construct($value) {
        $this->setValue($value);
    }
}

Example Usage:

$array = array('this', 'is', 'a', 'way', 'to', 'test');
$depth = 5;
$list = new SkipList($array, $depth);
if ($list->contains('is')) {
        echo "list contains 'is'.n";
}

Finding near-solutions to Fermat’s Last Theorem in C#

Ben Wendt — Thu, 25 Jul 2013 20:00:00 -0400

Fermat’s Last Theorem States that there are no non-trivial integer solutions solutions to the equation:


x^{n} + y^{n} = z^{n}, n > 2

This theorem went unproven for centuries, and was proven to be true in 1995 by Andrew Wiles. The Treehouse of Horror VI episode of the Simpsons aired the same year. People with a keen interest in humourous background gags could have done a freeze frame and seen the following (source:

The gag here is that if you punched in

1782¹² + 1841¹²

to a pocket calculator, you would get the same answer as if you punched in:

1922¹²

So this equation would seem to be true, but it’s not. It’s just really close, and the reason you see the same number is due to floating point rounding. I.e. it’s not actually true, but it’s close enough to fool the calculators of the day. Sadly I don’t know who found this near-solution to give credit. I remember hearing a commentary by David X. Cohen, who is a mathematician and writer for The Simpsons and Futurama, mentioned some details about this.

Finding near-solutions to FLT is actually quite easy though. You just have to do a brute force search and you can output whatever numbers you find that are close enough to what you are looking for. Just define your ranges of exponents and bases and loop through, looking for solutions that match a certain threshold. Here’s an example written in C#:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {

            string lookingFor = ".00000000";
            for (double exponent = 11; exponent &lt; 15; exponent++)
            {

                for (double doing = 701; doing &lt; 20002; doing++)
                {

                    double result = Math.Pow(doing, exponent);

                    for (double x = Math.Floor(doing * 0.5); x &lt; Math.Floor(doing * .95); x++)
                    {
                        double y1 = Math.Ceiling(Math.Pow(result - Math.Pow(x, exponent), 1 / exponent));
                        double y2 = Math.Floor(Math.Pow(result - Math.Pow(x, exponent), 1 / exponent));

                        // check the high number
                        double sum = Math.Pow(Math.Pow(y1, exponent) + Math.Pow(x, exponent), 1 / exponent);
                        string sumstring = sum.ToString();
                        if (sumstring.IndexOf(lookingFor) &gt; -1)
                        {
                            Console.WriteLine(doing + "^" + exponent + " = " + x + "^" + exponent + " + " + y1 + "^" + exponent + ", actually: " + sumstring);
                        }

                        // then the low.
                        sum = Math.Pow(Math.Pow(y2, exponent) + Math.Pow(x, exponent), 1 / exponent);
                        sumstring = sum.ToString();
                        if (sumstring.IndexOf(lookingFor) &gt; -1)
                        {
                            Console.WriteLine(doing + "^" + exponent + " = " + x + "^" + exponent + " + " + y2 + "^" + exponent + ", actually: " + sumstring);
                        }
                    }
                }
            }

            Console.ReadLine();
        }
    }
}

And doing this you can find some interesting near-solutions like:

[

19639^{11} = 15797^{11} + 19469^{11}, actually: 19639.0000000074

]

[

4472^{12} = 3987^{12} + 4365^{12}, actually: 4472.00000000706

]

[

14051^{13} = 11184^{13} + 13994^{13}, actually: 14051.0000000076

]

A demonstration of the usefulness of Memoization in Lua

Ben Wendt — Tue, 23 Jul 2013 20:00:00 -0400

Memoization is a programming technique where you save expensive computations in memory to speed up function execution time. E.g. If you were writing a CMS and you wanted a getSignedInUserName() method, you wouldn’t want to make two database calls to show the user name at the top and bottom of the page, so you’d save it in memory for use later.

My canonical example of the speed increase you can get from this technique is with a Fibonacci calculator. Here is a non-optimized version:

fib = function(n)
    if n==1 or n==0 then
        return 1
    else
        return fib(n-1)+fib(n-2)
    end
end

print(fib(40))

This takes about 22 seconds to run on my development machine. Here’s a rewritten calculator that uses memoization:

results = {}
fib = function(n)
    if results[n] then
        return results[n]
    else
        if n==1 or n==0 then
            result = 1
        else
            result = fib(n-1)+fib(n-2)
        end
        results[n] = result
        return result
    end
end

print(fib(40))

This finishes in well under one second. In fact, calling the memoized function with fib(100) finishes in under a second too. The real benefit here is that you aren’t clogging up your call stack with hundreds of thousands of calls, each waiting on other calls. By having previous results on hand, the function can easily move on to the next step.

Parsing mathematical expressions and calculating the result

Ben Wendt — Thu, 18 Jul 2013 20:00:00 -0400

Given a string like 2 * 2 + 2, how would you calculate the value? It’s necessary to tokenize the string, parse those tokens, and apply the order of operations to the parsed data.

Here is the function to throw a string at.

<?php

function calculate($input) {
    $tokens = tokenize($input);
    $parsed = parse_tokens($tokens);
    $result = calculate_from_parsed($parsed);
    return $result;
}
</pre>

And I just rely on PHP&#8217;s built in tokenizer, which I throw a wrapper around.

<pre class="brush: php; title: ; notranslate" title="">function tokenize($input) {

    $tokens = token_get_all("<?php $input");
    return $tokens;
}

So here is the first function that really does anything. Note that only pedmas rules are followed. The parser doesn’t deal with anything like exponentiation, trig functions, etc.

function parse_tokens($tokens) {

    if (!is_array($tokens)) {
        // invalid input.
        return false;
    }

    $expecting = null;

    $parsed_tokens = array();
    $skip_to = 0;

    foreach($tokens as $token_number => $token) {
        if ($token_number < $skip_to) continue;
        if (is_array($token) && isset($token[0])) {
            switch($token[0]) {
                case 305 :
                case 306 :
                    if (!is_null($expecting) && $expecting != 'number') {
                        exit('error 1: unexpected token ' . print_r($token, true) . "nn");
                    }
                    $expecting = 'operator';
                    $parsed_tokens[] = $token[1];
                case 372:
                case 375:
                    // whitespace
                    continue;

                default :
                    exit('error: unhandled token ' . print_r($token, true) . "nn");
            }
        } else {

            if (!is_null($expecting) && $expecting != 'operator' && $token != '(' && $token != ')') {
                exit('error 2: unexpected token ' . print_r($token, true) . "nn");
            }
            switch($token) {
                case '(' : 

                    $new_tokens = array();
                    $parentheses_count = 1;
                    for ($i = $token_number + 1; $i < count($tokens); $i++) {
                        if ($tokens[$i] == '(') {
                            $parentheses_count ++;

                        } else if ($tokens[$i] == ')') {
                            $parentheses_count --;
                        }

                        if ($parentheses_count != 0) {
                            $new_tokens[] = $tokens[$i];
                        } else {
                            $skip_to = $i;
                            $expecting = 'operator';
                            break;
                        }
                    }

                    $parsed_tokens[] = parse_tokens($new_tokens);
                    $expecting = 'operator';
                    break;
                case '+' :
                case '-' :
                case '*' :
                case '/' :
                    $parsed_tokens[] = $token;
                    $expecting = 'number';
                    break;
            }
        }

    }
    return $parsed_tokens;
}

Now run through the parsed tokens, apply the order of operations, and run until there is a result.

function calculate_from_parsed($parsed_tokens) {

    if (count($parsed_tokens) == 1 && !is_array($parsed_tokens[0])) {
        return $parsed_tokens[0];
    } else if (count($parsed_tokens) == 1 && is_array($parsed_tokens[0])) {
        return calculate_from_parsed($parsed_tokens[0]);
    } else {
        foreach($parsed_tokens as $token_number => $parsed_token) {
            if (is_array($parsed_token)) {
                $parsed_tokens[$token_number] = calculate_from_parsed($parsed_token);
            }
        }

        while (count($parsed_tokens) > 1) {
            $continue = false;
            foreach($parsed_tokens as $token_number => $parsed_token) {
                $previous_token_pair = get_previous_token_pair($parsed_tokens, $token_number);
                $previous_token = $previous_token_pair[0];
                $previous_token_index = $previous_token_pair[1];
                if ($parsed_token == '*' || $parsed_token == '/') {
                    if ($parsed_token == '*') {
                        $parsed_tokens[$token_number] = $previous_token * $parsed_tokens[$token_number + 1];
                        unset($parsed_tokens[$previous_token_index], $parsed_tokens[$token_number + 1]);
                        $continue = true;
                        break;
                    } else if ($parsed_token == '/') {
                        $parsed_tokens[$token_number] = $previous_token / $parsed_tokens[$token_number + 1];
                        unset($parsed_tokens[$previous_token_index], $parsed_tokens[$token_number + 1]);
                        $continue = true;
                        break;
                    }
                }
            }
            if ($continue) continue;

            $parsed_tokens = array_values($parsed_tokens);

            foreach($parsed_tokens as $token_number => $parsed_token) {
                $previous_token_pair = get_previous_token_pair($parsed_tokens, $token_number);
                $previous_token = $previous_token_pair[0];
                $previous_token_index = $previous_token_pair[1];
                if ($parsed_token == '+' || $parsed_token == '-') {
                    if ($parsed_token == '+') {
                        $parsed_tokens[$token_number] = $previous_token + $parsed_tokens[$token_number + 1];
                        unset($parsed_tokens[$previous_token_index], $parsed_tokens[$token_number + 1]);
                        break;
                    } else if ($parsed_token == '-') {
                        $parsed_tokens[$token_number] = $previous_token - $parsed_tokens[$token_number + 1];
                        unset($parsed_tokens[$previous_token_index], $parsed_tokens[$token_number + 1]);
                        break;
                    }
                }
            }

            $parsed_tokens = array_values($parsed_tokens);
        }
        if (count($parsed_tokens) == 1) {
            $parsed_tokens = array_values($parsed_tokens);
            return $parsed_tokens[0];
        }
    }
}

And since we pop results out of the array as we go we need a helper function for retrieving the previous populated element in an array.

function get_previous_token_pair($tokens, $token_number) {
    $return = false;
    for ($i = $token_number - 1; $i > -1; $i--) {
        if (isset($tokens[$i])) {
            $return = array($tokens[$i], $i);
            break;
        }
    }
    return $return;
}

And here is a little test suite, all of which pass:

$tests = array(
    array('(1.1 + ((1)))', 2.1),
    array('2 + 2', 4),
    array('1 + 1 + 1 + 1 + 1    + 1 + 1 + 1 + 1 + 1      + 1 +1 + 1 + 1 + 1', 15),
    array('2*(1 + 1 + 1 + 1 + 1)    + 3*(1 + 1 + 1 + 1 + 1)      + 4*(1 +1 + 1 + 1 + 1)', 45),
    array('(1 + 1 + 1 + 1 + 1)*2 + (1 + 4)*3 + (1 +1 + .5+.5 + 1 + 1)*5', 50),
    array('2 * 2', 4),
    array('2 * (1-1)', 0),
    array('2 * 2 + 2', 6),
    array('2+ 2 * 2', 6),
    array('(2+ 2) * 2 + 2', 10),
    array('(2+ 3 + 1) / 3 + 7', 9),
    array('2+ 2 * 2 + 2', 8),
    array('729 / 3 / 3 / 3 / 3', 9),
    array('729 / 3 / (3 / 3) / 3', 81),
    array('2 + 1', 3)
);

foreach($tests as $test) {
    $input = $test[0];
    echo "$input = ";
    $result = $test[1];
    $generated_result = calculate($input);
    if ($generated_result == $result) {
        echo "$generated_result passedn";
    } else {
        echo "$generated_result <-- FAILEDn";
    }
}

Viable promises in PHP using pthreads

Ben Wendt — Tue, 09 Jul 2013 20:00:00 -0400

I’ve looked at making promises in PHP before, but it was a bit pointless due to PHP’s synchronous nature.

But PHP isn’t necessarily synchronous. You can add threading capabilities by installing pthreads. Using this library it is possible to set up functioning promises (with a few limitations) in PHP.

Pthreads supports stacking threads, which is essentially a promise. As such, this blog post is essentially a rehash of any of the pthreads examples of Stackable. The basic idea is to set up a class that inherits from Worker, initialize an instance, start it going, then stack on an instance of a class inheriting from Stackable.

My implementation of this pattern will allow the passing of arbitrary functions to these classes, which is what the promise pattern is all about. It works, but there is a limitation. You can pass function names in (e.g. count_to_a_million), but not closures (e.g function() {echo "foo";}). [Aside: for some reason PHP calls anonymous functions closures even though they are only related concepts.] It appears that pthreads has some hidden serialization of parameters going on under the hood, and PHP does not support serialization of closures. Because of this, my implementation only supports the passing of function names (although it could be modified to accept parameters as well, as those could be serializable).

Here are the classes:

<?php

class PromiseClass extends Worker {
    private $_promise = null;
    public function run() {
        $func = $this->_promise;
        $func();
    }
    public function __construct($promise) {
        $this->_promise = $promise;
    }
}

class ThenClass extends Stackable {
    private $_promise = null;
    public function __construct($promise) {
        $this->_promise = $promise;
    }
    public function run() {
        $func = $this->_promise;
        $func();
    }

}

As you can see PromiseClass and ThenClass have the same extended properties and methods, but are based on different classes.

Here is how to use these classes to implement Promises in PHP:

function then_function() {
    echo "and then...n";
}

function promise_function() {
    echo "promise function called...n";
}

$promiser = new PromiseClass('promise_function');

$then = new ThenClass('then_function');

$promiser->start();
$promiser->stack($then);

for ($i = 0; $i < 20; $i++) {
    echo "testn";
}

The output for the above example should be something like the following (the outputs of “test” are there to show that this work is asynchronous):

promise function called...
test
test
...
test
and then...
test
test
...

Promises are most useful as a means of waiting for data before performing an action with it. In that regard this is not an ideal solution as it would require some form of kludge. If PHP had proper closures it would be reasonable but in this form it would likely need to require use of the global operator which is never nice.

Non-functional Sleep Sort Implemented in PHP using pthreads

Ben Wendt — Sun, 07 Jul 2013 20:00:00 -0400

I’ve looked at Sleep Sorting before. The basic idea is that each scalar in your collection to be sorted will be used as it’s own weight, which is then used as the delay before outputting it as output.

In a perfect world, all elements are sent to this time-based output buffer at the same instant, in which case the results will be accurate.

When I did this before in javascript, the results were accurate because javascript is an asynchronous language. When you call setInterval, other things can happen before that interval is complete.

PHP is not asynchronous. You could imagine looping through a collection and calling sleep before each echo and that this would be a basic sleep sort implementation. This doesn’t work because in PHP sleep is blocking. The result will be a script that waits the sum of the array seconds in total, and outputs the order unchanged.

function sleepcount($sleepnum) {
    sleep($sleepnum);
    echo "$sleepnumn";
}
foreach($nums as $num){
    sleepcount($num);
}

There are several methods of implementing threading in PHP. A beta PHP extension called pthreads is one way to do this. Here’s an implementation based on the pthreads Async example:

function sleepcount($sleepnum) {
    sleep($sleepnum);
    echo "$sleepnumn";
}

class Async extends Thread {

    public function __construct($method, $params){
        $this->method = $method;
        $this->params = $params;
        $this->result = null;
        $this->joined = false;
    }

    public function run(){
        if (($this->result=call_user_func_array($this->method, $this->params))) {
            return true;
        } else return false;
    }

    public static function call($method, $params){
        $thread = new Async($method, $params);
        if($thread->start()){
            return $thread;
        }
    }

}


$nums = array( 6, 2, 4, 1);

foreach($nums as $num){
    $future = Async::call("sleepcount", array($num));
}

On my development machine this does not product the correct results. It only sorts elements pairwise, so my result is

2, 6, 1, 4

This is likely because my development machine has a dual core processor. That’s just part of the fun of concurrent programming I suppose.

There may be a better way of implementing this in pthreads. I wouldn’t know. I’m only currently getting my feet wet with it.

Sleep Sort in Javascript

Ben Wendt — Sun, 07 Jul 2013 20:00:00 -0400

Sleep Sort is a humourous algorithm for sorting. The idea is to output the numeric array elements after a time interval proportional to the value of the array element. So if you had an array [3, 2, 1], 3 could be output three seconds after the sort, 2 two seconds after and 1 one second after. The result is that you’d see 1, 2, 3 three seconds later.

Of course this is a terrible idea, but it’s also a heck of a lot of fun!

Here’s an implementation of the idea in javascript:

function opnode(a) {
    el = document.createElement('div');
    el.innerHTML = a;
    document.body.appendChild(el);
}

var temporalSort = function(ar) {
    for (i = 0; i < ar.length; i++) {
        (
            function(a){
                window.setTimeout(
                    function() {
                        opnode(a);
                    },
                100 * a);
            }
        )(ar[i]);
    }
};

var i, array = [6, 14, 1, 12, 8, 3, 9, 2, 10, 15, 4, 11, 7, 13, 5];

temporalSort(array);

Simple Observer Pattern in PHP

Ben Wendt — Tue, 02 Jul 2013 20:00:00 -0400

The observer pattern is a nifty way to decouple objects from one another. Rather than having methods explicitly rely on, create, and access other objects, the other objects can subscribe to an object and enact their own changes as necessary.

Our implementation will have two general groups of objects, subscribers and observers. A subscriber class will hook in to an observer class and make actions when the observer publishes certain messages. The observer class will give objects methods to subscribe and unsubscribe from its messages.

We’ll begin by setting up an interface for our subscribers. Our subscribers could really be anything, so we want to specify some general behaviour that they will have. We don’t want to be restrictive and have an abstract class that they will inherit from because that would restrict functionality of all subscribers and limit the usefulness of this pattern.

interface Subscriber {
    public function EventCall($str);
}

Now we’ll set up an abstract class for Observables. This may be more useful as a trait but we actually do have to implement some functionality here.

abstract class Observable {
    private $_subscribers = array();
    public function Subscribe($o) {
        // give objects an ability to add themselves to the subscribers list.
        if (!in_array($o, $this->_subscribers)) {
            $this->_subscribers[] = $o;
        }
    }
    public function Unsubscribe($o) {
        // give objects an ability to remove themselves from the subscribers list.
        if (in_array($o, $this->_subscribers)) {
            foreach($this->_subscribers as $key => $value) {
                if ($o == $value) {
                    unset($this->_subscribers[$key]);
                }
            }
        }    
    }
    public function Event($event) {
        // when the event occurs, call the corresponding method on the clients.
        foreach($this->_subscribers as $subscriber) {
            $subscriber->EventCall($event);
        }
    }
}

Now that the abstract class and interface are ready, we can make some concrete classes based on these.

class Observer extends Observable {
    public function Talk() {
        echo "I am an observern";
    }
}

class Subscriber1 implements Subscriber{
    public function EventCall($str) {
        echo "Subscriber1 event occured $strn";
    }
}

class Subscriber2 implements Subscriber {
    private $data;
    public function __construct($data) {
        $this->data = $data;
    }
    public function EventCall($str) {
        echo "Subscriber2 event occured $str data is " . $this->data . "n";
    }
}

class Subscriber3 implements Subscriber {
    public function EventCall($str) {
        echo "Subscriber3 event occured $strn";
    }
}

And now let’s make some instances of these:

$observer = new Observer();
$subscriber1 = new Subscriber1();
$subscriber2 = new Subscriber2('one');
$subscriber2too = new Subscriber2('two');
$subscriber3 = new Subscriber3();

$observer->Subscribe($subscriber1);
$observer->Subscribe($subscriber2);
$observer->Subscribe($subscriber2);
$observer->Subscribe($subscriber2too);
$observer->Subscribe($subscriber3);

$observer->Event('wow check out this awesome message that is being passed, bro.');
$observer->Talk();

$observer->Unsubscribe($subscriber2);

$observer->Event('what unheard of madness will happen next?');

And of course the output is:

Subscriber1 event occured wow check out this awesome message that is being passed, bro.

Subscriber2 event occured wow check out this awesome message that is being passed, bro. data is one

Subscriber2 event occured wow check out this awesome message that is being passed, bro. data is two

Subscriber3 event occured wow check out this awesome message that is being passed, bro.

I am an observer

Subscriber1 event occured what unheard of madness will happen next?

Subscriber2 event occured what unheard of madness will happen next? data is two

Subscriber3 event occured what unheard of madness will happen next?

Using Interfaces to reduce coupling

Ben Wendt — Thu, 27 Jun 2013 20:00:00 -0400

Consider the following three scenarios:

  class Dog {
...
    public function Bark($str) {
        echo $str;
    }
...
}

class AnimalCommunication {
...
    public function DogBark(Dog $dog, $str) {
        $dog->Bark($str);
    }
...
}

  class Dog {
...
    public function Bark($str) {
        echo $str;
    }
...
}


class AnimalCommunication {
...
    public function AnimalCommunicate($str) {
        echo $str;
    }
...
}

Interface IAnimal {
    public function Speak($str);
}

Class Dog implements IAnimal {
...
    public function Speak($str) {
        $this->Bark($str);
    }
....
}
class AnimalCommunication {
...
    public function AnimalCommunicate(IAnimal $animal, $str) {
        $animal->speak($str);
    }
...
}

Note the following about these examples:

The first is tightly coupled to the dog class, and hence is not very extensible.
The second is coupled to echo. It does not allow different “animals” to output their text differently. What if we later added a TelepathicGecko class, which didn’t echo out to speak, but rather published to some ESP API somewhere? Clearly the coupling here is not ideal either.
The third is best. By programming to an interface, we reduce coupling to the minimum necessary amount.

Pointless Promises in PHP

Ben Wendt — Tue, 18 Jun 2013 20:00:00 -0400

A promise is a way to defer the execution of a given routine until the data it needs to run is ready. This is a very useful pattern in asynchronous languages, so using promises in a language like javascript is a great idea.

Of course PHP is (without forking) totally synchronous so there is really no reason to implement the promise pattern in PHP.

But the motto of every programmer is “if it’s a bad idea, I will do it!” (no, it isn’t), so here’s an implementation of promises in PHP:

class PromiseClass {
    private $callbacks = array();
    private $last_return;
    function promise($promise) {
        if (get_class($promise) == 'Promise') {
            return $promise;
        } else if (is_callable($promise)) {
            $this-&gt;then($promise);
            return $this;
        }
    }
    function then (callable $callback) {
        $this-&gt;callbacks[] = $callback;
        return $this;
    }
    function resolve () {
        $callback = array_shift($this-&gt;callbacks);
        if (is_callable($callback)) {
            $this-&gt;last_return = $callback($this-&gt;last_return);
        }
        if (count($this-&gt;callbacks) &gt; 0) {
            $this-&gt;resolve();
        }
    }
}

A few things to note here:

First you will have to make an instance of the class.
You start by passing a function to the promise method. You could use then but the code wouldn’t look as descriptive or read as well.
You can then add any functions that would be run after with successive calls to then.
None of the functions that have been set up will be run until you call the resolve method on the object.

Here’s an example of usage of this useless and pointless class:

$promiser = new PromiseClass();

$promiser->promise(function() {
        echo "sleepingn";
        sleep(3);
        return 3;
    })
    -&gt;then(function($args) {
        echo "that farn$argsn";
        sleep(1);
    })
    -&gt;then(function() {
        echo "even farthernn";
    });

$promiser->resolve();

Note: I’ve added some sleep statements here so it almost seems like something asynchronous is happening. Really sleep is just blocking. The output will be something like:

sleeping
that far
3
even farther

Longest Common Substring in PHP

Ben Wendt — Mon, 10 Jun 2013 20:00:00 -0400

Longest common substring is a function that can be useful once in a while. Here’s a PHP implementation. Be forewarned, this runs in O(mn) time.

function longest_common_substring($string1, $string2) {
    $L = array();
    $length = 0;
    $pos = 0;
    $array1 =str_split($string1);
    $array2 =str_split($string2);
    foreach ($array1 as $i =&gt; $c1) { 
        $L[$i] = array();
        foreach ($array2 as $j =&gt; $c2) { 
            $L[$i][$j] = 0;
            if ($c1 == $c2) {
                if ($i == 0 || $j == 0) {
                    // initialize that this character position exists.
                    $L[$i][$j] = 1;
                } else {
                    // increment previous or reset.
                    if (isset($L[$i-1][$j-1])) {
                        $L[$i][$j] = $L[$i-1][$j-1] + 1;
                    } else {
                        $L[$i][$j] = 0;
                    }
                }
                if ($L[$i][$j] &gt; $length) {
                    $length = $L[$i][$j];
                }
                if ((isset($L[$i][$j]))&&($L[$i][$j] == $length)) {
                    $pos = $i;
                }
            }
        }
    }
    if ($length &gt; 0) {
        return substr($string1, $pos - $length + 1, $length);
    } else {
        return '';
    }
}

Usage:


$string1 = 'sadjjasdf this is the string  sdlkjhaskl';
$string2 = 'eriuhysdfnbasi this is the stringbhdjubsdi';

echo longest_common_substring($string1, $string2);

Finding the main content element on a page in javascript

Ben Wendt — Sun, 09 Jun 2013 20:00:00 -0400

Short of going to something more complex like measuring information or doing some natural language processing, you can estimate which element on a page contains the content by determining which element has the highest ratio of contained content to contained markup. Here’s a javascript snippet that does just that:

// not perfect obviously. Not terrible neither.

var id, tag;
var all = document.querySelectorAll('body *'), max = 0, el, i, L;

// list some commons ids that denote the outermost element on a page.
var badIds = {
    "wrapper" : 1,
    "container" : 1,
    "wrapper-content" : 1
};

// we don't want to include content from certain tags.
var badTags = {
    "SCRIPT" : 1,
    "STYLE" : 1,
    "HEADER" : 1
}

// the goal rate of markup per content
var contentPercent = 0.45;

var contentRatio = function(el) {
    var i, L, totalScript = 0, scripts = el.getElementsByTagName("script");
    for (i =0, L= scripts.length; i &lt; L; i++) {
        totalScript += scripts[i].length;
    }
    totalScript = 0;
    return (el.textContent.length - totalScript) / el.innerHTML.length;
};

for (i = 0, L =all.length; i &lt; L; i++) {
    id = all[i].getAttribute('id');
    tag = all[i].tagName;
    if (all[i].textContent && all[i].textContent.length &gt; max && (contentRatio(all[i]) &gt; contentPercent) && !badIds[id] && !badTags[tag]) {
        max = all[i].textContent.length;
        el = all[i];
    }
}

// show the results.
console.log(el)
console.log(el.textContent.length / el.innerHTML.length)

Simple Markov Chain in PHP

Ben Wendt — Wed, 29 May 2013 20:00:00 -0400

Here’s a simple Markov chain implementation in PHP, loosely adapted from this excellent write up about implementing Markov chains in javascript:

class Link {
    private $nexts = array();
    public function addNextWord($word) {
        if (!is_string($word)) {
            throw new Exception('addNextWord method in Link class is run with an string parameter');
        }
        if (!isset($this-&gt;nexts[$word])) {
            $this-&gt;nexts[$word] = 0;
        }
        $this-&gt;nexts[$word]++;
    }
    public function getNextWord() {
        $total = 0;
        foreach($this-&gt;nexts as $word =&gt; $count) {
            $total += $count;
        }
        $randomIndex = rand(1, $total);
        $total = 0;
        foreach($this-&gt;nexts as $word =&gt; $count) {
            $total += $count;
            if ($total &gt;= $randomIndex) {
                return $word;
            }
        }
    }
}

class Chain {
    private $words = array();
    function __construct($words) {
        if (!is_array($words)) {
            throw new Exception('Chain class is instantiated with an array');
        }

        for($i = 0; $i &lt; count($words); $i++) {
            $word = (string) $words[$i];
            if (!isset($this-&gt;words[$word])) {
                $this-&gt;words[$word] = new Link();
            }
            if (isset($words[$i + 1])) {
                $this-&gt;words[$word]-&gt;addNextWord($words[$i + 1]);
            }
        }
    }
    public function getChainOfLength($word, $i) {
        if (!is_string($word)) {
            throw new Exception('getChainOfLength method in Chain class is run with an string parameter');
        }
        if (!is_integer($i)) {
            throw new Exception('getChainOfLength method should be called with an integer');
        }
        if (!isset($this-&gt;words[$word])) {
            return '';
        } else {
            $chain = array($word);
            for ($j = 0; $j &lt; $i; $j++) {
                $word = $this-&gt;words[$word]-&gt;getNextWord();
                $chain[] = $word;
            }
            return implode(' ', $chain);
        }
    }
}

And here is an example of usage:

function get_all_words_in_file($file) {
    return preg_split('/s+/ ', file_get_contents($file));
}

$file = 'testtext2.txt';

$words = get_all_words_in_file($file);
$chain = new Chain($words);
$newSentence = $chain-&gt;getChainOfLength('The', 200);
echo wordwrap($newSentence, 80, "n");

Conceptually, a Markov chain captures the idea of likelihood of traversing from state to state. You can populate this data for a block of text by passing through a block of text and counting the number of occurrences of words that follow a given word. You can then use this data to generate new blocks of text.

A Mandelbrot Set Viewer via Javascript and Canvas

Ben Wendt — Thu, 23 May 2013 20:00:00 -0400

In mathematics, a fixed point is an input x for a function f such that

f(x) = x

The Mandelbrot Set is a visualization of which points near 0 in the complex plane are fixed points for the function , where . Points that are fixed are rendered in black, while all other points are colour-coded based on how quickly repeated application of the function to a point diverges.

Images of the Mandelbrot set should be familiar to most everyone. It’s an infinitely detailed, self-similar set of rainbows surrounding bubbles.

So here’s my javascript and canvas based Mandlebrot viewer:

var xScale, xOffext, yScale, yOffset, xVal, yVal;
var canvas, h, w;
updateScalesAndOffsets = function() {
    xScale = parseFloat(document.getElementById('xScale').value);
    xOffset = parseFloat(document.getElementById('xOffset').value);
    yScale = parseFloat(document.getElementById('yScale').value);
    yOffset = parseFloat(document.getElementById('yOffset').value);
};
updateCanvasAndDimensions = function() {
    canvas = document.getElementById('m');
    h = canvas.getAttribute('height');
    w = canvas.getAttribute('width');
};
doMandelbrot = function() {
    var iteration, max_iteration = 1000, l, x, y, x0, y0, xtemp;
    updateCanvasAndDimensions();
    var ctx = canvas.getContext('2d');
    updateScalesAndOffsets();

    for (var i=0; i < w; i++) {
        for (var j=0;j < h; j++) {
            // for each point in the image, generate the color value.
            x0 = xScale * (i / w) + xOffset;
            y0 = yScale * (j / h) + yOffset;

            x = 0;
            y = 0;

            iteration = 0;

            while (x*x + y*y < 4 && iteration < max_iteration) {
                // this is parametrically performing the complex function f(z) = z^2 -1.
                xtemp = x*x - y*y + x0;
                y = 2*x*y + y0;
                x = xtemp;
                iteration++;
            }

            if (x*x + y*y < 4) {
                ctx.fillStyle='rgb(0,0,0)';
            } else {
                l = iteration < 50? iteration : 50;
                // set colors using hsl so that the number of iterations to diverge maps to the hue.
                ctx.fillStyle='hsl('+Math.floor((iteration/max_iteration)*256)+',100%,' + l + '%)';
            }

            ctx.fillRect(i,j,i+1,j+1);
        }
    }
};
mouseMove = function(e) {


    xVal = xScale * (e.clientX / w) + xOffset;
    yVal = yScale * (e.clientY / h) + yOffset;
    var xCoordinateElement = document.getElementById('xCoordinate'), yCoordinateElement = document.getElementById('yCoordinate');
    xCoordinateElement.innerHTML = xVal;
    yCoordinateElement.innerHTML = yVal;
};

zoomIn = function() {
    document.getElementById('xScale').value = parseFloat(document.getElementById('xScale').value) / 2;
    document.getElementById('xOffset').value = xVal - parseFloat(document.getElementById('xScale').value) / 2;
    document.getElementById('yScale').value = parseFloat(document.getElementById('yScale').value) / 2;
    document.getElementById('yOffset').value = yVal - parseFloat(document.getElementById('yScale').value) / 2;
    doMandelbrot();

}
zoomOut = function() {
    document.getElementById('xScale').value = parseFloat(document.getElementById('xScale').value) * 2;
    document.getElementById('yScale').value = parseFloat(document.getElementById('yScale').value) * 2;
    doMandelbrot();

}
moveDir = function(dir) {
    switch (dir) {
        case 'up' : document.getElementById('yOffset').value = yOffset - yScale / 10; break;
        case 'down' : document.getElementById('yOffset').value = yOffset + yScale / 10; break;
        case 'right' : document.getElementById('xOffset').value = xOffset + xScale / 10; break;
        case 'left' : document.getElementById('xOffset').value = xOffset - xScale / 10; break;
    }
    updateScalesAndOffsets();
    doMandelbrot();
}

Reactive Programming in PHP

Ben Wendt — Wed, 22 May 2013 20:00:00 -0400

Wikipedia has this to say about reactive programming:

In computing, reactive programming is a programming paradigm oriented around data flows and the propagation of change. This means that it should be possible to express static or dynamic data flows with ease in the programming languages used, and that the underlying execution model will automatically propagate changes through the data flow.

Inspired by projects like knockout.js and reactor.js, I thought I’d give it a shot in PHP. Here is an example implementation:

$Signal = function($v) {
    if (is_callable($v)) {
        return function() use ($v) {
            return $v();
        };
    } else {
        return function($a = null) use ($v) {
            static $return;
            if (is_null($return)) {
                $return = $v;
            }
            if (!is_null($a)) {
                $return = $a;
            }
            return $return;
        };
    }
};

And here is an example of usage:


include 'Reactor.php';

$foo = $Signal(1);

$bar = $Signal(function() use ($foo) {
    return $foo() + 1;
});
$bar2 = $Signal(function() use ($foo, $bar) {
    $val = $foo();
    return $val * $val + $bar();
});

echo $foo() . "n";
echo $bar() . "n";
echo $bar2() . "nn";

$foo(2);

echo $foo() . "n";
echo $bar() . "n";
echo $bar2() . "nn";

$foo(0);

echo $foo() . "n";
echo $bar2() . "nn";

Ben Wendt's blog

File Integrity Monitoring in Python

Tell A Story To Write Better Code

Generating fractals from Dask Distributed

Branch Coverage, A Cautionary Tale

Rotary MIDI

Cookie Clicking

Using ollama to make an epub synopsis

An Intro to Data Build Tool (dbt) With Hive

Development Environment Setup

Setting up DBT.

An Intro to Data Build Tool (dbt) Hooks

An Intro to Data Build Tool (dbt)

Introduction: Getting to Know Data Build Tool (dbt)

Installation of Project Bootsrapping.

DBT Profiles

Database Seeds

Models

Macros

Schemas

Data Tests

Analyses

Python models

Conclusion

Finding Sator Squares

An uninformative error in bigquery

Schemas and SqlTransform in Beam

Solving Spelling Bee and Letter Boxed

Safe Navigation in Python

A Tale From Microservice Hell

Running a shell command for each entry in a PCollection

Transferring Musical Likes Between Services

Doing some biological modeling

Diabolical Answers In Wordle

Messing Around In Beam, π Day, And More Wordle

BrainF Interpreter In Scratch

Automatic Word of the Day

What is Canada's Largest Transitless Municipality?

"Infinite Word Ladder"

I Did a Deep Q Learning Course

3D Printed Rocket

The Best And Worst Words In Wordle

Using an Artie 3000 as a Plotter

Finding Latent Faces-with Non-negative Matrix Factorization

A "100 emoji" maker

Fibonacci Spheres in Blender

Archiving photos from iOS to a network drive

Uploading and sorting photos from android over sftp using termux

Fizzbuzz in Different Paradigms / Techniques

1. conditional logic

2. pattern match

Feeling a little let down.

0. array programming

1. exception handling

2. boolean operators (ifs in disguise)

3. multiplication (ifs in a better disguise)

4. done declaratively in sql

5. list reduction

6. oop

7. functions

8. code generation

9. declaratively in css

Conclusion

Deep Learning Fundamentals

A Ruby Pipe

Introduction to J

Popping errors out from ruby threads using a Queue

Intro to Bayesian Classifiers

A Simple Example

A mind-expanding example

Learning Through Bayes’ Theorem

A proper learning example

So how would you implement this?

references

Adding Database Constraints Using the `rein` Gem

My first slack bot

The World's Simplest IRC Bot - Again

The World's Simplest IRC Bot

Designing Models for 3D Printing in openSCAD

Playing with Vue.js

2. boolean operators (`if`s in disguise)

3. multiplication (`if`s in a better disguise)