Monthly Archives: September 2018

Revisiting Embedded InnoDB

Many people these days don’t know InnoDB was originally developed as an independent database engine apart from MySQL. Its author, Heikki Tuuri, modeled InnoDB after Transaction Processing: Concepts and Techniques, the seminal transaction processing book authored by Turing Award laureate James “Jim” Gray and Andreas Reuter. It wasn’t until later InnoDB was integrated with MySQL.

While InnoDB contains a very simple, barebones, parser and executor, it’s major advantage was in its solid transactional storage engine implementation. The advantage to using MySQL for the high-level database functionality was that a full SQL parser, optimizer, et al didn’t have to be developed and InnoDB could focus on its strengths; this was especially true when compared to the two other popular storage engines of the day, MyISAM and the Berkeley DB (BDB) Storage Engine, which–when I met with Keith Bostic in 2005–it seemed Sleepycat had no interest in pursuing. Several years later, after InnoDB was already considered the defacto production-ready storage engine for MySQL, Innobase Oy released the Embedded InnoDB, which enabled developers to embed the open-source InnoDB database engine directly within their applications. As I had done some work years before to extract the storage engine from MySQL and use it in my own project, Prophet, this was a pretty awesome turn of events. Sadly, however, it ended-up being overlooked and died pretty quickly.

Given the death of the official Embedded InnoDB, developer Stewart Smith took up the reins, forked the codebase, and began a separate project he called HailDB. This continued for some time and resulted in a couple releases. Eventually, however, the same disinterest Innobase saw resulted in the project being canned as well.

For posterity, I’ve uploaded a copy of the original release of the official Embedded InnoDB source code as well as the documentation not included in the source distribution. Likewise, I’ve committed the code to the last official release of HailDB along with example code demonstrating how to develop against the embedded database engine.

You may ask, “what would it take to get the latest InnoDB working in embedded mode?” Well, I’ve done a cursory look and I believe it would take a week or so to re-work things from the current C++-based MySQL server codebase and swap out the items required for it to be standalone again. While I have several cool uses for this, including my old drop-in replacement libraries for Btrieve and Microsoft Jet/Extensible Storage Engine (ESENT), it’s not worth my free-time to work on.


Open Source ODBC Drivers for Oracle

In the good old days, database-agnostic applications were written using drivers that implemented the Microsoft Open Database Connectivity (ODBC) API, especially on Windows. Much like JDBC, ODBC provided developers with a single, interoperable, C-based programming language interface that made it possible for applications to access data from a variety of database management systems.

When developing an ODBC-based application with Oracle, there were a variety of drivers to choose from including:

While several of these companies offer wire-protocol implementations of their drivers, which didn’t require Oracle Client, the majority of supportable ones (or ones that required advanced functionality) are based on top of the Oracle Call Interface, including the Oracle ODBC driver itself. With the exception of Oracle’s driver, however, the other commercial drivers are rather expensive. So much so, in fact, that I’ve seen companies pay almost the cost of a database license just to deploy the driver across their enterprise.

To get around this, several open-source ODBC drivers for Oracle Database had been developed. The two most notable drivers are as follows:

Easysoft ODBC Driver for Oracle (in C)
First, before it became a commercial product, Easysoft released their driver as open-source. Like other ODBC drivers for Oracle available at the time, it was based on top of OCI. Over the years, however, this seemed to get lost. For that reason, when I ran across the code for it in my archive the other day, I figured I’d upload it to GitHub for all the world to see. You can find it here. Thanks to the legal principle of promissory estoppel, while the C code is no longer publicly available from Easysoft, the Easysoft Public License (EPL) which it was originally distributed under is still applicable. The EPL appears to be very Apache-like in nature, permitting inclusion in a commercial application as long as any changes to the driver itself are made available and the copyright acknowledgement is made.

EDO Driver for Oracle (in C++)
This driver was written by Edwig “Edo” Huisman, who ran into a number of the bugs and limitations found in the ODBC driver for Oracle9i and who also found the commercial offerings to be too expensive. Unlike the Easysoft driver, which is designed for UnixODBC, I’ve never tried to compile this on UNIX/Linux. I do think, however, someone with a little C++ and ODBC driver development knowledge could easily get it to compile within a day or so. While it’s currently released under the GPL, Edwig has said he’s looking to re-license it under something more liberal–perhaps even the MIT license–provided portions of the code are re-written. While the project for this is on SourceForge, you can find my repo of the code here.

The last update to the open-source Easysoft driver was made in 2002 and, of the two, Edo’s driver is certainly more readable, recent, and well-tested. Likewise, Edo seems to be a pretty responsive and good guy. If I find time, I’d like to get his driver going on UNIX/Linux.

A Word About Wire-Protocol ODBC Drivers
Having reverse-engineered the drivers from several commercial ODBC vendors, it seems most have done an almost verbatim copy and source-to-source translation of the wire protocol code from Oracle’s JDBC driver. While this certainly isn’t a legal form of reverse engineering, I guess they’re not big enough for Oracle to go after. Likewise, while every one of these drivers will work with newer versions of Oracle, they all appear to use rather antiquated calls emulating the JDBC driver and don’t take advantage of many improvements in OCI over the years. YMMV.