So, the basic idea is that there’s a core library (beyond-core), a server (connecting to MySQL or SQLite), and a set of clients (connecting to the server via a web service, and each with their own local data store for offline work).

Core library

This’ll be the main workhorse, used by both the web service and the clients. I’m planning on writing it in Python, though I may end up porting it to Objective-C as well for the desktop and iPhone clients. We’ll see.

The core will manage the records, conflict resolution, and all of that. More details to come, of course.

Data interface

Also used by both the web service and the clients, this will be an in-between layer so I can keep the database stuff separate from the main logic. I’m hoping to make it relatively easy to plug in new data stores for those who want to extend Beyond; the only ones I’ll be working on will be MySQL (for the main server) and SQLite (for all the clients).

So, if someone wants to write a Beyond client that uses plain text files for the data, they’ll be able to. That’s what I’m aiming for.

Web service

I’m leaning towards using web.py for this and the web client. It’ll basically be a web-exposed API to the core library.

Clients

The clients will work on a local store (SQLite for most and HTML5 offline storage for the web client) and then periodically sync their changes with the web server.

Note: for brevity’s sake I’ve left off the data interface and pipes in the top part of the diagram.

What’s next

I’ll be writing a functional spec (use cases and stuff) and a technical spec (data model, web service API, etc.) and will post them to Google Docs as public read-only documents. I’ll also continue posting on here, of course.

Once I feel like the design is where it needs to be, then it’s coding time.

Feedback

I want feedback. Seriously. Let me know what you think (if you have an opinion, of course :)) about the architecture behind this, especially if you notice any flaws or ways to improve it.

One last thing

Confession: I’m seriously itching to start coding already. To scratch that itch, I’m going to write a smaller, minimalist web-based genealogy app on the side. More on that once it’s actually underway.

| Date | Description |
| 12.21.09 | Finished transcribing the November documents |

Which produced a table that looked like this:

Super simple. It assumes that if you have more than one row, your first row is a header row. Compare that to the HTML I’d have to write instead:

<table>
    <thead>
        <tr>
            <th>Date</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>12.21.09</td>
            <td>Finished transcribing the November documents</td>
        </tr>
    </tbody>
</table>

Sure, you couldn’t do much more than this with it — no control over alignment, no possibility for complex setups — but it was enough. Being easy to write goes a long way. (And again, if you need more control, you can always revert to HTML.)

Now look at PHP Markdown Extra’s syntax:

Date     | Description
-------- | -----------
12.21.09 | Finished transcribing the November documents

Or MediaWiki’s (Wikipedia):

{|
|Date
|Description
-
|12.21.09
|Finished transcribing the November documents
|}

Ew. Both are a pain to write and MediaWiki’s is ugly to boot.

My philosophy: if I want to write a complex table, then I’ll use HTML. Most of the time I don’t, however, and I want something simple, fast, and beautiful, like the syntax up at the top.

I found a Python-Markdown extension, Simple Tables, which uses a variant on the syntax above, but it seems to have been replaced by PHP Markdown Extra, which uses the yucky syntax.

Anybody know of any Markdown extensions (preferably PHP, since that’s what my WordPress blog is running on) that do nice tables like this?

In other news, at work I’ve been coding a board game in Flash. It’s effectively my first Flash project ever (years and years ago I edited a company map in Flash, but it was so long ago that I can hardly remember it, and I was only maintaining it, so it doesn’t really count). Flash is smooth. I’m not completely satisfied with ActionScript, but it certainly works well enough, and I’m sure more experience with it will make it better. Overall, my time with Flash has been good and fun.

Finally, I’m hoping to get back into more Ruby/Python/Perl coding before too long. My new job’ll require some XML magic, for which I’ll probably use Python and XSLT. And in my typographical work on the side I’ll be doing a lot with TeX and either Ruby or Python.

All of which is to say, hopefully I’ll start blogging here more often. :)

In the meantime, I think I want to master regular expressions next. I’m familiar with them and have used them often, but there’s a lot of power in them there regexes. In other news, I’ll be coding a family website in Rails over the next few months. It’ll be a good testing ground for my Beyond work. And at work, it looks like this web app I’m working on will be almost all Javascript. Not what I expected, but I’d much rather work in Javascript than ASP.NET. :)

[tags]Rails, Javascript, ASP.NET[/tags]

So, I’m going back to the basics. Simplicity is good, and so I’ve taken the app I’m working on and pared it down as far as possible. So far, in fact, that I’m pretty sure I’ll be able to do the whole thing with Javascript, HTML, and CSS, with a web service providing a connection to the server (written in ASP.NET most likely, though I’d prefer Python and may end up going that route), and anything else in Python. I might use MochiKit or Dojo to make the Javascript end easier to code, but I’m not sure yet if I really need it. (Why not Rails? Well, the server is unfortunately a Windows box running IIS, and I’ve heard that Rails doesn’t perform all that well on IIS. I wish I could get the server moved to Linux, but that’s probably not going to happen anytime soon. In the meantime, Python is still very nice to work with, and I haven’t caught wind of any performance issues with it on IIS.)

At any rate, this stack feels a lot better. When I was working with OpenLaszlo, I felt like I was in a straitjacket. Not much fun. Or “open.” (Again, keep in mind that these are subjective impressions; who knows, maybe OpenLaszlo is your style.)

In the meantime, I’m reading Paul Graham’s ANSI Common Lisp and liking it.

[tags]OpenLaszlo, Flex, Javascript, XML, Lisp, Python, MochiKit, Dojo, Paul Graham[/tags]

So I spent half an hour throwing together a quick prototype of our app — no functionality, just the layout — and I think I’m liking it. It’s basically XML with a healthy dose of Javascript. Using XML this way feels almost like Lisp somehow, incidentally. :) So far I haven’t run into any huge roadblocks, so we’ll see if OpenLaszlo works for what I need.

[tags]OpenLaszlo, Flash, Pandora, XML, Javascript, Lisp[/tags]

In writing software, it seems like there are two distinct creative philosophies: planned vs. organic. With the first, you blueprint the whole project out in advance, thinking through everything as much as possible, sketching out the program with broad strokes first. You then go through the blueprint and implement it. The second philosophy takes a more laid-back approach, starting with the bare minimum and then letting the needs of the users define the design. It’s an iterative process, something more grown than engineered.

As is often the case in life, both of these methods have their place. For me, the general flow of a project should be organic (if possible), but the details should be planned. Let me explain.

When writing software, you often don’t know what’s best for the user. You have ideas, sure, but you don’t know for certain. Rather than bloat the software up with unnecessary features, it’s better to let the project’s requirements evolve as you proceed. Experience will dictate the course.

Down in the trenches, however, it helps a lot to have a plan. Not a long-term plan, mind you, but a right-here-and-now list of what needs to happen in the next hour or day. Yes, I can code without a plan, but my thinking gets murky and there’s only so much state you can hold in your mind before you start to lose things. Writing out a list of to-do items, along with pseudocode for whatever it is I’m working on, has proven invaluable.

My other lifesaver is freewriting. If I get stuck, I open up my work log in Google Docs and start writing about what I’m working on, raising questions that need to be answered, describing obstacles in my path, trying to map out the next few steps so I don’t stay bogged down. It’s worked almost every time. And when it doesn’t work, I know it’s time to take a break and do something else for a bit — read a book, draw, take a walk — anything to recharge the batteries.

Oh, and I’m going to try to be better about posting here more than once a month. :)

[A CMS] ought to make your content more useful simply by virtue of the content being in the system. But more often than not, it doesn’t. Most of time, you actually make your data (read: content) as dumb as possible by way of entering it into a CMS. Seriously.

He’s got a good point — with most content management systems, we flatten out the structure (or square-peg a round hole, as he puts it) and lose important information. Rolling your own is sounding better and better every day. I’ve been meaning to do that for Riverglen Press, and it wouldn’t be a bad idea to extend it to Blank Slate either (once I figure out what on earth I want Blank Slate to be — at the moment it’s just a neglected child, most of the time forgotten while its siblings bask in the spotlight, but ironically it gets more traffic than the others).

At any rate, I’m itching to make these sites more my own, and writing a custom CMS is a great way to do that. (I also really need to revamp the graphic design…) The main thing is to decide what I need this CMS to do. Most features I don’t really need, really, and my tastes have sharply turned to lightweight lately.

More to come later, once I have more time. :)

So, the BYUFHLC is on hold for now due to time constraints at work. (I have to get all the new extraction software done by a month from now when we present it at the ICAPgen conference.) After ICAPgen I should have enough time to finish it, though.

Haven’t coded much Ruby (or Rails) lately. I may start using it for prototypes for this extraction software, though… Considering that we’ll be switching to a Linux server sometime within the next few months, it almost feels like a waste to code the stuff in .NET (not to mention that I much much much prefer Rails), but it’ll have to do. At least I can rest assured that I’ll be able to recode it in Rails five times as quickly. :)

In other news, Matz is coming to BYU to give a colloquium two weeks from tomorrow. (11 a.m. on the 19th in 1170 TMCB, if you’re interested and on campus.)

[tags]BYU, Family History Library, ICAPgen, Ruby, Rails, Ruby on Rails, Matz[/tags]