Projects

In this area you'll find other things I'm working on and/or interested in.

But have I mentioned my beautiful son lately?

23:00:59 27-Sep-2006
Dezi search platform

This week I announced the initial release of Dezi, a new search platform based on Swish3, Apache Lucy, OpenSearch and Plack.

As of about 15 minutes ago, there are now PHP and Perl clients available.

21:47:46 01-Oct-2011
libswish3 1.0.0 released

I am happy to announce the 1.0.0 release of libswish3:

http://swish-e.org/swish3/libswish3-1.0.0.tar.gz

libswish3 is at the core of multiple Swish3 implementations, and has reached a stable enough API that a 1.0.0 release seems appropriate.

From the README:

libswish3 is a document parser compatible with the Swish-e 2.4 -S prog API. libswish3 is a C library for parsing documents into a data structure that can then be stored and searched with a variety of IR backends.

There are currently four different implementations available of Swish3.

  • swish_xapian (C++ using libxapian, included in libswish3 distribution)
  • SWISH::Prog::Xapian (Perl using Search::Xapian)
  • SWISH::Prog::Lucy (Perl using Apache Lucy)
  • SWISH::Prog::KSx (Perl using KinoSearch)

All the Perl implementations are available from CPAN. They each rely on SWISH::3 (the Perl bindings to libswish3) and the core SWISH::Prog project, a Perl rewrite of the swish-e 2.x C binary and accompanying helper scripts. The SWISH::Prog distribution includes a 'swish3' command line interface with options very similar to the swish-e 2.x command line tool.

Xapian, KinoSearch and Apache Lucy all offer robust UTF-8 and incremental indexing support, as well as the ability to scale to many millions of documents across multiple servers.

You can read more about Swish3 at the devel site.

UPDATE: Mailing list announcement here.

22:03:59 21-Sep-2011
Search::OpenSearch::Server with REST API

Just uploaded several modules to CPAN that together implement a full REST API for KinoSearch indexes, using Search::OpenSearch::Server::Plack.

% curl -XPOST http://localhost:5000/foo \ -d '<doc><title>bar</title>foo</doc>' \ -H 'Content-Type: application/xml' [response:] { "success":1, "doc":{ "orgs":[], "places":[], "people":[], "topics":[], "summary":"", "title":"bar", "author":[] }, "total":"21581", "code":"200" }

The modules are:

  • Search::OpenSearch 0.11
  • Search::OpenSearch::Server 0.05
  • Search::OpenSearch::Engine::KSx 0.08
  • SWISH::Prog::KSx 0.17
  • SWSIH::Prog 0.49

    • 13:56:43 26-May-2011
    False Laziness

    One of the three virtues of programming is Laziness. Beware of false laziness. Andy Lester writes on the problem aptly when he describes an interaction with another programmer:

    This person was one of those programmers who tried for the premature optimization of saving some typing. He forgot that typing is the least of our concerns when programming. He forgot that programmer thinking time costs many orders of magnitude more than programmer typing time, and that the time spent debugging can dwarf the amount of time spent creating code.

    03:40:32 23-Dec-2010
    The Interruptible Programmer

    A brilliant and humane essay on changing work habits.

    Excuse me while I get up and stretch.

    I can vouch for the writer's experience, though for me it has been less about back pain (though I have that too) than eye strain (going on 7 years now). Biggest of all though has been having children and working from home: that is the interruption formula in a nutshell.

    10:43:06 15-Oct-2010
    CPAN test failures

    SWISH::3 0.08_04 is passing all tests all over the CPAN testers universe, so that is encouraging.

    However, some reports (notably on FreeBSD) report false failures because of a Wstat issue.

    I've posted about it at PerlMonks and hope someone out there has an easy fix.

    Update: finally found a fix for this. The problem is that Perl has its own my_setenv() function that interferes with the native setenv() called by libswish3.c. The fix was to set the magic Perl var PL_use_safe_putenv as shown here. This took many hours and googling to track down. Glad to be done with it (I hope!).

    00:37:31 11-Oct-2010
    Swish3 progress report

    There's been a ton of work on Swish3 in the last year. I've actually started planning a 1.0 release, after 5 years of work.

    Lately I've been focusing on three things: (1) making the Perl bindings easier to install; (2) indexing of compressed documents; and (3) supporting XInclude of document fragments. The first is accomplished: you can install the entire library via CPAN. The last two are aimed at large doc sets where I want to keep the XML compressed on disk for space reasons, and where I want to re-use subsets of the document collections in building multiple indexes.

    23:33:18 08-Jun-2010
    Open Source Handbook

    Open Source Handbook reviewed.

    23:03:03 16-Mar-2010
    make test

    Invoking make test in a project and watching as 1000s of successful tests scroll by, culminating in the All tests successful. message, gives me the same thrill of satisfaction as when I used to paint houses, and having finished a long day of sweaty labor at sanding and chipping old paint off, I could stand back and survey the structure, primed and ready for a fresh coat of paint. It's the anticipation that thrills, in the same way that a trip to the grocery store and a full fridge, or several loads of clean laundry folded and stowed safely away in drawers, thrills me. The knowing that I am prepared, belt cinched tight, all tests successful.

    21:56:14 03-Mar-2010