Ocassional Notes
14. February 2015
In my previous post, I discussed some tokenization techniques and mentioned that a whitespace-only tokenizer will make tokens that are sub-optimal for indexing. I also mentioned that a simple solution to this is created a whitespace + punctuation tokenizer.
14. February 2015
Wanting to expand my horizons a little, I decided to give android game development a try. I quickly prototyped a game in canvas + js called protect your thing to give myself an idea of how it would work on the android platform. This was a natural prototyping choice for me because I’ve been working with JS code daily for at least a decade now, and I love playing with canvas. (I frequently will whip up a little graphic in canvas reminiscent of my early QBasic coding days, but flavoured by the years of studying mathematics that followed).
31. January 2015
A text index is a way to store information about textual data to improve ease of retrieval. The basic idea is to store the information in such a way that it is possible to retrieve what you are looking for without doing a full scan of the data. A text index will do this by storing information about each indexed word separately. So where in the raw text you might have a text field with the contents “this is a description”, a full text index would store information about each individual word separately, so it would store a form of “this”, “is”, “a”, and “description”.