Log Analyzer and 3D File Browser

Keitai PictureYet another entry that our non technologically minded readers might want to skip (read: any person sane enough to keep nighttime exclusively dedicated to sleep and/or reproductive purpose rather than inane technological musings and script coding).

First is a little contribution to this massive venture in ego stroking that are personal websites. Or ego-crushing, as the case might be, but that’s none my business if nobody cares about your collection of vintage dr Pepper bottles from the 70s and your logs remain desperately empty (tip: try incorporating a sex angle to the whole thing). Anyway, if you have a site and want to monitor who comes and see what, I put together a small php script to parse and display log results matching a specific text string.

Why bother when there are many much more advanced log parsing utilities out there? Well, because, oddly enough, Awstats, that I use a lot and thanks to which I know that at least one person a month gets here by searching for “pokemon rape pictures” in Google, Awstats is unable to give me something as simple as the number of single visitors to a specific URL on a daily basis. Despite the fact that it’s sometimes damn useful. For example to know how many so-called friends have been checking our announcement for this great party we are throwing in Yoyogi next week. Hence this script.
You can download it here.
Using it is ridiculously simple (provided you know what PHP and Apache logs are, otherwise I strongly suggest doing a bit of research beforehand). Drop as many Apache log files inside the “data” directory as you want. Direct your browser toward the “quicklogs” directory, appending “?match=something” to look for a specific word or expression. Results should come neatly presented with little horizontal bars for each day.
Oh and yea: as usual, Creative Commons GPL 2 license blah blah blah etc.

In other totally unrelated, albeit not any less geeky, news: I’ve been playing with fsv, a “3D File System Visualizer” for X-Window (for trivia fans out there, I reckon this is a clone of the actual SGI program used in the movie Jurassic Park… you know, when that girl knowingly explains “This is Unix! I know this.”). It does what you would expect and display directories and files on your desktop in a cool, though slightly dated, 3D look. You can move the camera around, zoom and explore directories.

Playing with fsv got me thinking about all these alternate file-browsing system and why nobody had come up with more insightful ways to present organized data yet. Explanation: while fsv or XCruise 3D navigation systems are definitely neat-looking and in fact somewhat useful, the major turn-off is that they use file size to decide what importance to give files (e.g. how big, how bright or how close to the camera it should be). In an era where most hard drives are filled with files of disproportionate sizes (a 15 megs MP3 here, 600 megs movie there and a 30k text file in the middle), this comes with little use and ends up creating useless scales where a few files take all the visual field and a myriad of others are barely visible.

Further more, most visual implementation (not all, but most) still rely entirely on the file hierarchy (i.e. what file is in what directory) to display files together, be it as 3D block (fsv) or stars in a virtual galaxy (XCruise). This is also thoroughly unproductive in those days…

Why not, for example, incorporate criteria such as:

  • Creation and last modification date/time. Both to judge the relevance of a given file but also to create artificial contingences between files: if you’ve edited two files, each one in a separate location on your HD, on the very same date and time, chances are they are somehow related.
  • File type: not just a dumb extension matching or similar, but a classification by type of media. If you start heading toward a jpeg file, chances are you are looking for a picture on your hard drive, therefore the system should privilege files that belong to that type (not only jpeg: any graphic files, possibly even movies or less closely related file types).
  • File Content: of course, that would become more of a performance issue, but, using a smart keyword indexing system, we could get files of similar content presented closer together.

Another thing of importance, I think, is that these criteria should be evaluated dynamically (or cached whenever possible, but it’s not a good idea for a file-system browser to take a half-hour parsing the HD each time it starts): navigation should never be interrupted to scan the content of a folder or read metadata. Relevant data should be integrated progressively as they are read and file representation morphed accordingly to fit their new status (e.g. for a program like fsv, block should grow or shrink, change color and move around when their contingence and relevance becomes clearer to the system).

It would be essential that such a system integrates a powerful search tool to help navigating or filtering. This should be easy to do but I have not seen much of this in any projects lately. With HD routinely containing thousands if not millions of files, it is just vital to have ways to query a certain file by its name or type.

All these features, conjugated with one of these 3D file-browser project could give a seriously kick-ass software and help get past their sole purpose as futuristic eye-candy.

And now for the complaining about how little time I got and how I’d love to implement such a thing one day soon… yea whatever…

1 comment

Leave a Reply