Procurios TechBlog RSS

Locking multiple objects in MySQL

Sun, 17 Jan 2010 12:03:05 +0100

I was looking for a way to lock several objects in a single transaction. It took me some time to figure it out, but it turned out to be very simple.

Subject

MySQL, using InnoDB tables, placing WRITE locks on (a distributed representation of) "objects"

The Problem

An object in a programming language usually needs to be stored in a database, for later retrieval. Most developers are using a relational database to do this. This means that an object is stored as a record in a table. And if it stays that simple, there is no problem. However, once an object gets more complex, it will span many more records, in several tables. And when two or more processes start editing the same object at the same time, data corruption may occur. The objects need to be locked explicitly for writing, so that the other process waits for the first one to finish. But how do you lock "objects"?

I will give you an example of data corruption:

A relation is represented by this main record (table: 'relation_main'):

[relation_id, lastname, firstname]

this relation has a single e-mail address in a separate table ('relation_address', because he can have many addresses)

[relation_id, e-mail]

This is an object that spans several (2) tables. At some point an extra e-mail address needs to be added to this relation and two processes (either people and or automated processes) enter the new address at the same time. The processes both check if the e-mail adres is present already, and if not, add it. But since both processes are doing this at the same time, the new address will have been entered twice.

If your application has comparable situations, think of locking the objects: one of the processes acquires the lock first, the other process needs to wait until the other process is completely done with the object, before it can make its change.

Now MySQL has a function for just this purpose: GET_LOCK(str, timeout). It enables you to acquire a single lock on any given identifier. It could be the relation_id of our example. RELEASE_LOCK(str) will release the lock again.

If this was all there is to it I wouldn't be writing this article. The point is this: on the request of a second GET_LOCK(), MySQL will release the first lock. So the function does not allow you to acquire multiple locks. In a transaction it is often necessary to lock multiple objects at once. To be able to do that, we need another technique. I thought of some terribly hopeless alternatives that I will not expose here. In stead, let's look at the conclusion.

The Solution

To lock an "object", change a single special record that is part of the object, or that represents it, by updating it.

How does that work? Well, remember the simple case where an object is represented by a single record? There is no need for an explicit object lock, because MySQL will place a write lock on the record automatically. You can use this principle to place a lock on an object with multiple records as well. All you need to do is to mark one of the object's records as special, and make sure that all processes that modify the object, modify this special record first. This way, by writing to a single record, a lock is acquired for the entire object.

You can use a special "locking" table for this purpose, or use a special record inside of the object. It doesn't matter what update is made to the main record: update its last-modified time if you like. But writing the existing value of a field right back to it will do just fine:

UPDATE 'relation_main' SET 'firstname' = 'firstname' WHERE 'relation_id' = 1234;

No change is made to the data, but MySQL will still place a write lock on the record.

When a process performs this update it may need to wait for another process to release this "lock". Only after the UPDATE has completed, the process should read the object's current data from the database, because the other process have have changed it.

Mark unread tabs in FireFox

Sun, 03 Jan 2010 14:19:38 +0100

I'm a heavy FireFox user. When doing research these days my primary source of information is the web and the browser is my primary application. So I'm interested in everything that makes my FireFox life easier. I'd like to share the best tweak I know.

When reading a page I open a lot of tabs. Search results to be checked out later. Interesting topics I stumble upon by serendipity. References an article links to. A simple ctrl-click and they're being loaded in the background for later reading. The following tweak marks an unread tab with any style you like.

Open the file chrome/userChrome.css in your FireFox profile directory. If this file does not exist you can create an empty one.

Put the following simple css rule in it, restart FireFox and you're done.

#content tab:not([selected]) {
	color:red !important;
}

See the Customizing Mozilla page on mozilla.org for more information about the chrome css files and lots of other tweaks.

Upgrade to PHP5.3 using dotdeb debian packages

Wed, 12 Aug 2009 16:02:56 +0200

Currently we are testing PHP 5.3 on some of our production servers. It has a quite stable feel to it.

I expected much more problems. Although we had to modify our sources a bit, (To avoid this bug for example. Using empty() within our own database class extending mysqli) the effort into getting it to work wasn't too big.

In the past years we've always used the dotdeb.org repository for php packages. Guillaume is doing a terrific job in keeping his packages updated with the last PHP source version. Typically it takes about 2-3 days for packages tot pop up on dotdeb after a new PHP release. Because 5.3 is such a big change, Guillaume has setup a separate repository for it.

The dotdeb php 5.3 repository is still lacking some packages, so we had to build a few packages ourselves. The packages php5-apc, php5-ffmpeg and php5-mailparse built the dh-make-pecl way without problems.

Building php5-imagick gave us some troubles. I don't recall if that was because we wanted them running on ubuntu too. The ImageMagick package is in transition from libimagick to -core and -wand packages. Also debian and ubuntu packages names are different. I eventually decided to build ImageMagick ourselves too. That way I have the same setup on production and my ubuntu development laptop.

We are using the syck pecl package, that one built as a charm too. We use the libsyck0-dev package from debian unstable to build it.

We also used the dbase functions that were available by default in PHP 5.2. From 5.3 on they have been moved to a pecl package, but it is still without maintainer. Currently it lives in SVN only at http://svn.php.net/repository/pecl/dbase/. This one wasn't easy to build. You have to make a file structure that complies with pecl packages, before you can use dh-make-pecl on it. If you want I could provide you a debian source package for this one.

Now the only thing missing is the Suhosin patch for PHP 5.3. Good news on that front: Arnold Daniels ported the patch http://blog.adaniels.nl/articles/suhosin-patch-for-php-53/. Although it is not done by Stefan Esser of the Suhosin team himself.

Did you switch to PHP 5.3 already? Did you run into any problems?

Cache a large array: JSON, serialize or var_export?

Thu, 09 Jul 2009 16:20:40 +0200

While developing software like our framework you will need to cache a large data array to a file at some point sooner or later. At such a point you need to choose what caching method you will be using. In this article I will compare three methods: JSON, serialization and var_export() combined with include().

Too curious? Jump right to the results!

JSON

The JSON method uses the json_encode and json_decode functions. The JSON-encoded data is stored as is into a plain text file.

Code example

// Store cache
file_put_contents($cachePath, json_encode($myDataArray));
// Retrieve cache
$myDataArray = json_decode(file_get_contents($cachePath));

pros

Pretty easy to read when encoded
Can easily be used outside a PHP application

cons

Only works with UTF-8 encoded data
Will not work with objects other than instances of the stdClass class.

Serialization

The serialization method uses the serialize and unserialize functions. The serialized data is, just like the JSON data, stored as is into a plain text file.

Code example

// Store cache
file_put_contents($cachePath, serialize($myDataArray));
// Retrieve cache
$myDataArray = unserialize(file_get_contents($cachePath));

pros

Does not need the data to be UTF-8 encoded
Works with instances of classes other than the stdClass class.

cons

Nearly impossible to read when encoded
Can not be used outside of a PHP application, without having to write custom functions

Var_export

This method 'encodes' the data using var_export and loads the data using the include statement (no need for file_get_contents!). The encoded data needs to be in a valid PHP file so we wrap the encoded data in the following PHP code:

<?php
return /*var_export output goes here*/;

Code example

// Store cache
file_put_contents($cachePath, "<?php\nreturn " . var_export($myDataArray, true) . ";");
// Retrieve cache
$myDataArray = include($cachePath);

pros

No need for UTF-8 encoding
Is very readable (assuming you can read PHP code)
Retrieving the cache uses one language construct instead of two functions
When using an opcode cache your cache file will be stored in the opcode cache. (This is actually a disadvantage, see the cons list).

cons

Needs PHP wrapper code.
Can not encode Objects of classes missing the __set_state method.
When using an opcode cache your cache file will be stored in the opcode cache. If you do not need a persistant cache this is useless, most opcode caches support storing values in the shared memory. If you don't mind storing the cache in memory, use the shared memory without writing the cache to disk first.
Another disadvantage is that your stored file has to be valid PHP. If it contains a parse error (which could happen when your script crashes while writing the cache) your application will not work anymore.

Benchmark

In my benchmark I used 5 different data sets with different sizes (measured in memory usage): 904B, ~18kB, ~250kB, ~4.5MB and ~72.5MB. For each of these data sets I did the following routine for each encoding method:

Encode the data 10 times
Calculate the string length of the encoded data
Decode the encoded data 10 times

Results

Yay, results! In the result tables you see the length of the encoded string, the total time used for encoding and the total time used for decoding. The benchmark was done on my laptop: 2.53GHz, 4GB, Ubuntu linux, PHP 5.3.0RC4.

904 B array	JSON	Serialization	var_export / include
Length	105	150	151
Encoding	0.0000660419464111	0.00004696846008301	0.00014996528625488
Decoding	0.0011160373687744	0.00092697143554688	0.0010221004486084

18.07 kB array	JSON	Serialization	var_export / include
Length	1965	2790	3103
Encoding	0.0005040168762207	0.00035905838012695	0.001352071762085
Decoding	0.0017290115356445	0.0011298656463623	0.0056741237640381

290.59 kB array	JSON	Serialization	var_export / include
Length	31725	45030	58015
Encoding	0.0076849460601807	0.0057480335235596	0.02099609375
Decoding	0.014955997467041	0.010177850723267	0.030472993850708

4.54 MB array	JSON	Serialization	var_export / include
Length	507885	720870	1059487
Encoding	0.13873195648193	0.11841702461243	0.38376498222351
Decoding	0.29870986938477	0.21590781211853	0.53850317001343

72.67 MB array	JSON	Serialization	var_export / include
Length	8126445	11534310	19049119
Encoding	2.3055040836334	2.7609040737152	6.2211949825287
Decoding	4.5191099643707	8.351490020752	8.7873070240021

We've done the same benchmark on eight other machines including Windows and Mac OS machines and some webservers running Debian. Some of these machines had PHP 5.2.9 installed, others already switched to 5.3.0. All had the same (relative) results, except for a macbook in which serialize was faster encoding the largest dataset.

Conclusion

As you can see the var_export (without opcode cache!) method doesn't come out that well and serialize seems to be the overall winner. What bothered me though was the largest dataset in which JSON became faster than serialize. Wondering whether this was a glitch or a trend I fired up my OpenOffice spreadsheet and created some charts:

The charts show the relative speed of each method compared to the fastest method (so 100% is the best a method can do). As you can see both JSON and var_export become relatively faster when the data set gets big (arrays of 70MB and bigger? Maybe you should reconsider the structure of your data set :)). So when using a sane sized data array: use serialize. When you want to go crazy with large data sets: use anything you like, disk i/o will become your bottleneck.

ProtoFish: advanced hover menu based on Prototype

Wed, 19 Aug 2009 12:22:33 +0200

ProtoFish is an advanced hover menu based on Prototype. You can easily add a delay to your menu (on mouseout) and choose your own hover class. All ProtoFish menu's are keyboard accessible conform ARIA best practices. On top of that: it's fully customizable and free to use!

Quickstart

You've come here to implement ProtoFish into your website. Due to ProtoFish' popularity, we've decided to give ProtoFish it's own home at http://protofish.procurios.nl/. All implementation details can be found there!

Why ProtoFish?

Most people know the parent of all hover menu's: the SuckerFish dropdown menu published by A List Apart in 2003. Obviously, the name ProtoFish is a combination of the Prototype JS framework and the SuckerFish technique published by A List Apart. There are many enhanced versions of the SuckerFish dropdown around on the web, but none of them met the following, to me important, criteria:

The script should be based on Prototype,
The script should be lightweight and
The script should be easy to implement.

Finding a good menu-script based on criteria #1 is already hard enough, so I decided to write my own class and share it with the world!

Can I customize ProtoFish?

Sure! You're allowed to use ProtoFish without any constraints. Rip the code apart or add some features - use it the way you like! If you download the non-minified ProtoFish version, you can easily get a grip on the code, by reading the comments!

You can also contribute to ProtoFish on github: http://github.com/pesla/ProtoFish/.

Feedback

If you've found a bug, want to improve ProtoFish or just want to give some feedback, feel free to contact me on Twitter (@pesla) or leave a comment below this post.

Semantic web marvels in a relational database - part II: Comparing alternatives

Sun, 13 Sep 2009 11:18:04 +0200

In this article I will compare the basic technical details of current relational database alternatives.

In the first article in this series I explained the relational database mapping of our semantic web implementation. In this article I will place this work into perspective by exploring related techniques.

The last few years developers are looking for ways to overcome certain shortcomings of relational database systems. RDBMSes are general purpose data stores that are flexible enough to store any type of data. However, these are several cases in which the relational model proves inefficient:

An object has many attributes (100+), many of which are optional. It would be wasting space to store all these attributes in separate columns.
Many attributes with multiple values. Since each of these attributes needs a separate table, the object data will be distributed over many tables. This is inefficient in terms of development time, maintenance, as well as query time.
Class inheritance. Since most software is Object Oriented these days the objects in code will need to be mapped to the database structure. In the case of class inheritance, where attributes are inherited from superclasses, it is a big problem to store objects in, and query them from, an RDBMS efficiently.
Types and attributes are not objects. In an RDBMS the data of a model is separate from the metadata (attribute names, datatypes, foreign key constraints, etc.). Types and attributes are not like normal objects. This is inefficient in areas where types and attributes need to be added, changed and removed regularly, just like any other data. It is inefficient to write separate code to manipulate and query types and attributes. In short, first order predicate logic no longer suffices for many new applications. The second order is needed.
Scalability. Is an aspect often named as the reason to leave RDBMS. However, since relational databases have been optimized for decades, they do scale. Nevertheless, in this age of global, real-time webapplications, techniques provided by RDBMS manufacturers may prove to be inadequate, or simply too expensive.

In the following I will provide a simple understanding of the basic principles of alternative database techniques, along with some pointers to more in-depth information. I hope you will forgive me my non-expert view on these subjects. For detailed information on any given subject, look elsewhere. This article is meant to be just a quick overview, aimed to waken some concepts provided by the examples.

RDBMS, or Row-Oriented database

In a relational database management system, pieces of data are grouped together in a record. In this article I will consider the case where the data stored is meant to represent the attributes of an object. Seen this way, a record is a group of attributes of an object. Here's an example of such a table of objects:

object id	color	width	height	name
3	red	100	100	my box
4	green	50	500	old beam

Metadata is shown in gray. Keys / foreign keys are shown in bold typeface.

Need more attributes? Add more columns. Need an attribute with multiple values? Add a table and link it to the first. The RDBMS chooses speed over flexibility. Speed was a big deal 40 years ago, when this database type was designed. And it still is a big deal today. For large amounts of simple data, there is absolutely no need to leave this model.

Semantic net

Storing semantic information as triples is an old idea in the field of Knowledge Representation. As early as 1956, semantic nets were used for this purpose. In this technique the relations between objects are represented by plain labels. Each "record" stores only a single attribute, or one element of an array-attribute. Most notable are the absense of metadata and the fact that object data is distributed over many records.

object id	predicate	value
3	color	red
3	width	100
3	height	100
3	name	my box
4	color	green
4	width	50
4	height	500
4	name	old beam

Need more attributes? No need to change the table structure. Need an attribute with multiple values? Same thing.

Entity-Attribute-Value

The Entity-Attribute-Value model of knowledge representation uses some form of triples, just like the semantic web. Its primary use is described by Wikipedia as "Entity-Attribute-Value model (EAV), also known as object-attribute-value model and open schema is a data model that is used in circumstances where the number of attributes (properties, parameters) that can be used to describe a thing (an "entity" or "object") is potentially very vast, but the number that will actually apply to a given entity is relatively modest. In mathematics, this model is known as a sparse matrix."

Attribute metadata is stored in separate attribute tables, which are not triples. EAV is a sort of middle between semantic nets and semantic web: attributes have explicit properties, but these are fixed in amount.

EAV can be used to model classes and relationships as in EAV/CR.

EAV is used in Cloud computing databases like Amazon's SimpleDB and Google's App Engine.

object id	attribute id	value
3	1	red
3	2	100
3	3	100
3	4	my box
4	1	green
4	2	50
4	3	500
4	4	old beam

attribute id	name	datatype	unique
1	color	char(6)	true
2	width	double	true
3	height	double	true
4	name	string	true

Need more attributes? Add them in the attribute table. Attributes with multiple values? No extra work. The schema of the attributes is stored in the database explicitly, but attributes are treated different from the objects.

Column-Oriented databases

From wikipedia: "A column-oriented DBMS is a database management system (DBMS) which stores its content by column rather than by row."

object id	color
3	red
4	green

object id	width
3	100
4	50

object id	height
3	100
4	500

object id	name
3	my box
4	old beam

Google's BigTable is based, in part, on column-orientation. Their tables use reversed URI's as object and column identifiers, and have a "third dimension" in that older revisions of the data are stored in the same table.

References:

Correlation databases

A correlation database is "value based": every constant value is stored only once. All these values are stored together, except that values are grouped by datatype. All values are indexed. "In addition to typical data values, the data value store contains a special type of data for storing relationships between tables...but with a CDBMS, the relationship is known by the dictionary and stored as a data value."

I have not found a clear example of what this datastructure looks like, but we can infer that the internal structure must look something like the following. Note: I may be completely wrong here!

The values-table (actually there is one table per major datatype; i.e. integers, strings, dates, etc.)

value id	value
1	red
2	green
3	100
4	50
5	500
6	my box
7	old beam
8	<object 1>
9	<object 2>
10	<relationship color>
11	<relationship width>
12	<relationship height>
13	<relationship name>

and then there is at least a table containing the relationships (or: "associations") between the values. The relationships are stored as values themselves:

value id 1	association	value id 2
8	10	1
8	11	3
8	12	3
8	13	6
9	10	2
9	11	4
9	12	5
9	13	7

References:

Value-based storage (VBS) unleashes fast ad hoc query performance.

Hierarchical model, Network model, Navigational database

For the sake of completeness I have to name these models. The hierarchical model stores tree-like structures only, requiring each piece of data to have a single "parent". The network model allows a piece of data to have multiple parents. Both models were superseded by the relational model, but they are still used for special-purpose applications. A navigational database allows to traverse such trees / DAGs by following paths.

Object-Oriented databases

In an object-oriented database all attributes of a class are stored together. From what I've read on the internet I conclude that the actual storage structure of an OODBMS is sort of an implementation detail. This means that performance characteristics of the database will depend heavily on the type of implementation chosen. Development of this model was first in the hands of the ODMG, but control was transferred to the Java Community Proces that build the Java Data Objects specification. This specification names the conditions for such a database, but does not guide the implementation.

Some special properties:

Class inheritance is supported in the data model.
Object nesting: an object can contain (not just link to) other objects

Mapped to an RDBMS, a so called ORM (Object Relational Mapping), objects are commonly stored in a standard relational way: one column per (single valued) attribute. To implement inheritance, the columns of all base classes of an object are joined. This can be done at design-time (create a big table containing the columns of all parent classes) or at query-time (join parent class tables).

class id	object id	color	width	height	name
101	3	red	100	100	my box
101	4	green	50	500	old beam

class id	class name	parent class
101	Object
102	Bar	101

References:

Document based databases

A document based database is a different beast altogether. It lacks a database schema completely, and a complete object is stored in a single cell. In the case of CouchDB, this is done by encoding the object (or: document) in JSON. Real-time querying of the source table is thus impossible, one needs to create views on the data.

object id	document
3	{"color":"red","width":100,"height":100,"name":"my box"}
4	{"color":"green","width":50,"height":500,"name":"old beam"}

References:

CouchDB: Technical Overview

Triplestores

Some triplestores are publicly available. Commonly they have an RDF interface. Their performance can be measured using the Lehigh University Benchmark (LUBM). The most advanced open source triplestores are Sesame, and ARC.

object id	attribute id	value
3	101	red
3	102	100
3	103	100
3	104	my box
4	101	green
4	102	50
4	103	500
4	104	old beam
101	104	color
102	104	width
103	104	height
104	104	name

Very little has been made public about the way triplestores are implemented in a relational database. A laudable exception to this is the Jena2 database schema. Unfortunately, the schema appears to be very inefficient, since the URIs are not indexed but are used literally.

A charmingly simple implementation that seems resource intensive was made for expasy4j: triples are stored in a single table, but for query speed, a single column is reserved for each separate datatype.

Another, somewhat better implementation was made for OpenLink Virtuoso: it uses indexed uris, but all constants are placed in a single field datatyped "ANY".

Conclusion

I hope this article has shown you a little bit why developers are looking for alternatives for the familiar RDBMS and which forms these currently have taken. Currently the field is quite diverse and developments are being made by many different parties. It will be interesting to see how this evolves and which alternative(s) will eventually become the successor of the relational database.

Semantic web marvels in a relational database - part I: Case Study

Tue, 15 Sep 2009 09:09:39 +0200

You have heard about the semantic web. You know it is described as the future of the Web. But you are still wondering how this vision is going to make your applications better. Can it speed up application development? Can it help you to build complex datastructures? Can you use Object Oriented principles? This article shows how it can be done. And more.

The semantic web is framework developed by the W3C under supervision of Tim Berners Lee. Its basic assumption is that data should be self-descriptive in a global way. That means that data does not just express numbers, dates and text, it also explicitly expresses the types of relationship these fields have for their objects. Using this uniform datastructure, it will be easier to interchange data between different servers, and most of all, data can be made accessible to global search engines.

That is a big thing. But is that all? Can't you just provide an RDF import / export tool for your data and be done? Are there any intrinsic reasons why you would base you entire datastructure on the semantic web?

In a series of two articles I will try to explain how we at Procurios implemented semantic web concepts, what the theoretical background of our implementation is, and what benefits a semantic web has over a traditional relational database. In this first article I will explain how we implemented a semantic web in a relational database (we used MySQL), added an object oriented layer on top, and even created a data revision control system from it.

Triples

In a classic relational database, data is stored in records. Each record contains multiple fields. These fields contain data that may belong to some object. The relation between the field and the object it belongs to is not represented as data in the database. It is only available as metadata in the form of the column (name, datatype, collation, foreign keys). An object is not explictly modelled, but rather via a series of linked tables.

A semantic web is a network of interrelated triples ("subject-predicate-object" triplets) whose predicates are part of the data themselves. Moreover, each object has an identifier that is not just an integer number that means only something inside the database only. It is a URI that may have a distinct meaning worldwide.

A triple is a record containing three values: either (uri, uri, uri) or (uri, uri, value). In the first form the triple relates one object to another, as in the fact "Vox inc. is a supplier" (Both "Vox inc.", "is a", and "supplier" are semantic subjects identified by a uri). In the second form the triple links a constant value to a subject, as in "Vox inc.'s phone number is 0842 020 9090". A naive implementation would look like this:

CREATE TABLE 'triple' (
    'subject'                varchar(255) NOT NULL,
    'predicate'              varchar(255) NOT NULL,
    'object'                 longtext,
);

This table provides a complete implementation for the semantic web. However, it is too slow to be used in any serious application. Now, there are various ways in which this basic form can be optimized, but to my knowledge there is no best practise available. Several problems have to met:

How to identify a triple uniquely (If this is necessary for your application. The combination of subject, predicate, object itself is not unique)
How to search fast, given a subject and predicate? ("Give me the names of these set of people"?)
How to search fast, given a predicate and an object? ("Give me the persons whose name begins with 'Moham'"?)

To solve these problems we came up with the following changes:

Create a single triple index table that only stores triple ids.
Create separate triple tables for each of the major datatypes needed (varchar(255), longtext, integer, double, datetime)
The triple tables reference the index table by foreign key.
Add two extra indexes for the two ways the tables are used: a subject-predicate combined key and a predicate-object combined key.

Here's the triple index table (we are using MySQL):

CREATE TABLE 'triple' (
    'triple_id'                int(11) NOT NULL auto_increment,
    PRIMARY KEY ('triple_id')
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

and here's the triple table for the datatype "datetime" (the other datatypes are handled similarly)

CREATE TABLE 'triple_datetime' (
    'triple_id'                int(11) NOT NULL,
    'subject_id'               int(11) NOT NULL,
    'predicate_id'             int(11) NOT NULL,
    'object'                   datetime NOT NULL,
    'active'                   tinyint(1) NOT NULL DEFAULT '1',
    PRIMARY KEY ('triple_id'),
    KEY ('subject_id', 'predicate_id'),
    KEY ('predicate_id', 'object'),
    CONSTRAINT 'triple_datetime_ibfk_1' FOREIGN KEY ('triple_id') REFERENCES 'triple' ('triple_id') ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

The table definition should speak for itself, except for the field "active". This field is not necessary at this point, but I will need it in the next section.

The predicate_id refers to a separate "uri" table where the full uris of these predicates are stored. However, this is not necessary and the uris may be stored in the triple_longtext table as well.

The two combined keys have an interesting side-effect: the application developer never needs to be concerned again about using the right keys. Effective keys have been added by default.

To query this triplestore building SQL queries by hand may be a daunting task. It requires a special query language to be effective. More about that below.

All data of a given object can be queried by selecting all triples with a given subject id (one query per datatype triple table). That seems to be inefficient and it is: compared to the situation where an object can be stored in a single record, the triplestore is always slower. However, in a more complex situation a relational database requires you to join many tables to fetch all data. We use 5 separate queries (one per datatype table) to fetch all object data from the triplestore. This turned out faster than a single union query on the five queries. We use the same 5 queries to fetch all data of any desired number of objects. Here a the queries needed to fetch object data from three objects identified by the ids 12076, 12077, and 12078:

SELECT 'object' FROM 'triple_varchar' WHERE 'subject_id' IN (12076, 12077, 12078);
SELECT 'object' FROM 'triple_longtext' WHERE 'subject_id' IN (12076, 12077, 12078);
SELECT 'object' FROM 'triple_integer' WHERE 'subject_id' IN (12076, 12077, 12078);
SELECT 'object' FROM 'triple_double' WHERE 'subject_id' IN (12076, 12077, 12078);
SELECT 'object' FROM 'triple_datetime' WHERE 'subject_id' IN (12076, 12077, 12078);

You can see that the object-data is fetched from the database without having to provide explicit type or attribute information. The type of the object is stored in one of the triples. This is useful in case of inheritance where the exact type of an object can only be determined at runtime.

Arrays and multilinguality

Many object attributes have an array datatype (an unordered set). To model these in a relational database you would need a separate table for each of these attributes. Querying all attributes of a series of objects including these array attributes is far from easy. In the triple store you can model an unordered set as a series of triples having the same subject and predicate and a different object. When you query all object data, you will get the array values the same way as you get the scalar values.

Multilinguality is also a hassle in relational databases. For each of the attributes that need to be available in more than one language the table structure needs to be adjusted and it is hard to avoid data duplication. In a triplestore you can treat a multilingual attribute almost like an array element. The only thing is that the predicates are similar but not the same. We use the following uri's for the representation of different language variants of an attribute: http://our-business.com/supplier/description#nl, http://our-business.com/supplier/description#en, http://our-business.com/supplier/description#de (in the tables these predicates are replaced by their integer ids for faster querying).

Data revision control

Version control is pretty common for developers when it comes to storing previous versions of their code. It allows you to track the changes of the code, revert to a previous version, and to work on the same file together. Still, when it comes to data, version control is very uncommon. And I think that is mainly because the overhead to create such a system is huge in a traditional relational database.

One of the requirements for our framework was that there should be some form of data-change history available. And when you think of it, it is actually really simple to keep track of all the changes that are made to the data if you use triples. And that's because from a version-control point of view, all that changes each revision is that some triples are added, and others are removed.

So all that is needed is two more tables, one to keep track of the revision-data, like, who made the change, when, and a short description for future reference, and another to track all the added and removed triples in this revision:

CREATE TABLE 'revision' (
    'revision_id'                int(11) not null auto_increment,
    'user_id'                    int(11),
    'revision_timestamp'         int(11) not null,
    'revision_description'       varchar(255),
    PRIMARY KEY  ('revision_id')
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

CREATE TABLE IF NOT EXISTS 'mod_lime_revision_action' (
    'action_id'                   int(11) NOT NULL AUTO_INCREMENT,
    'revision_id'                 int(11) NOT NULL,
    'triple_id'                   int(11) NOT NULL,
    'action'                      enum ('ACTIVATE', 'DEACTIVATE') NOT NULL,
    'section_id'                  int(11),
    PRIMARY KEY  ('action_id'),
    CONSTRAINT 'revision_triple_ibfk_1' FOREIGN KEY ('revision_id') REFERENCES 'revision' ('revision_id') ON DELETE CASCADE,
    CONSTRAINT 'revision_triple_ibfk_2' FOREIGN KEY ('triple_id') REFERENCES 'triple' ('triple_id') ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Each time a user changes the data, a new revision is stored in the database, along with a list of all triples that are added or deactivated, and a compact description of the change. A triple that was already available in an inactive state is made active. If no such triple was present, an active one is created. Triples are never really removed, they are only set to be inactive.

If you query the triplestore (the set of all triples), you need to ensure that only the active triples are queried.

With this information, you can:

List all revisions made to the data, showing who made the change and when, along with a small description of the change.
Revert changes back to a previous revision, by performing the revisions backwards: activate the deactivated triples, deactivate the activated triples. It is also possible to undo a single revision, that is not even the last one. But beware that revisions following it may have dependencies on it.
Work together on an object by merging the changes made by the two users using the difference in data between the start and end revisions.

Object database

Businesses are used to work with objects. A web of data needs to be structured first before it can be used for common business purposes. To this end we decided to build an object oriented layer on top of the triplestore. But even though the Web Ontology Language (OWL) was designed for this purpose, we did not use it, since we needed only a very small subset anyway and we wanted complete freedom for our modelling activities, because processing speed was very high on our priority list. I will not cover all the details here, since it is a very extensive project, but I want to mention the following features:

The database was set up as a repository: no direct database access is possible by the application developer. Object creation, modification, destruction, and querying is done via the repository API. This provided the OOP principles of information hiding, modularity.
Object types could be associated with PHP classes. This is no requirement, but it proved really easy to generate object types from PHP classes. This provided us with the principle of polymorphism.
Not only simple objects are modelled as objects (a set of triples, having the same subject), but object types as well. Furthermore, the attributes of the types are modelled as objects as well. Objects and their types can be used in the same query.
Object types can be subtyped. The triplestore allows us to query objects of a given type and all its subtypes in a straightforward way.
The attributes of objects can be subtyped as well. This allows you to add datatype restrictions to the attributes of subtypes that were not applicable on a higher level up the type hierarchy.

These features are very powerful. It is possible to build a real Object database using triples as a datastructure only. Types and attributes are treated the same as normal objects. This means that the same code can be used to manipulate normal data as wel as metadata. Also, to implement inheritance is relatively easy, since object data is not chunked in single rows any more.

Query language

After some time we felt that the simple queries we were performing on the object database were too constraining. We wanted the same power that SQL provides. And on top of that, since we continue to use normal relation tables as well, the object queries needed to be able combine the object database with the relational tables. For these reasons, the semantic web query language of choice, SPARQL, was insufficient for our purposes. We now build SQL-like queries using method chaining on a query object. The object then creates the SQL query.

I mention this because you really need to build or use a new query language when starting to work with either a triplestores or an object database. The underlying triple store is too Spartan for an application developer. The lower level SQL queries consist of many self-joins connecting the subject of one to the object of another. Very hard to understand.

Afterword

I wrote this article because I think this sort of detailed information about emerging datastructures is lacking on the internet. It is also great to work for a company (Procurios!) that agrees with me that knowledge should be given back to the community if possible. Gerbert Kaandorp from Backbase once asked me very seriously what I had done to promote the semantic web, like it was some kind of holy mission. I hope this article has made a small contribution and that it inspires some of you to build your own semantic web based object datastore. Let me know what you think!

Syntactic Sugar for MySQLi Results using SPL Iterators

Fri, 15 May 2009 21:12:26 +0200

Ever wondered why you can't use foreach() on MySQLi Results, and instead have to write less convenient while() loops with fetch_row? Actually, you can use foreach() on MySQLi Results. All it takes is some SPL Iterator magic.

Why would we want foreach()?

Well, just take a look at the code examples below and judge for yourself.

$DB = new mysqli('localhost', 'user', 'password', 'db');

// standard approach using a while() loop with fetch_row
$Result = $DB->query($query);
while ($row = $Result->fetch_row()) {
  // do something with $row
}
$Result->free();

// same loop with foreach
foreach($DB->query($query) as $row) {
  // do something with $row
}

// foreach loop on a query that selects 2 columns
foreach ($DB->query($query) as $key => $value) {
  // do something with $key and $value
}

As you can see the foreach loops can be written in a more compact form. Another benefit is that we can't accidentally forget to free() our $Result objects this way, which could lead to unnecessary memory usage.

Whereas the PDO database interface does support foreach() on PDOStatements natively, MySQLi doesn't support foreach() on it's MySQLi_Results.

Keep reading to find out how we can add foreach() support to your MySQLi Results ourselves.

The Iterator Interface

The Standard PHP Library (SPL) contains a set of useful classes, among which we find the Iterator interface. Any class that implements the Iterator interface can be iterated using a foreach() loop.

If we want to use a foreach() loop on our query results we need to write our own ResultIterator class that implements the Iterator interface.

interface Iterator extends Traversable
{
  function rewind();
  function current();
  function key();
  function next();
  function valid();
}

Implementing the Iterator interface means providing implementations for each of the 5 methods in the interface. A nice way to show what each of these methods should do, is using the iterator interface in a standard for() loop.

for ($Iterator->rewind(); $Iterator->valid(); $Iterator->next()) {
  $value = $Iterator->current();
  $key = $Iterator->key();
}

As you can see the loop will start with rewind() and will then continue to call next() untill valid() returns false. The Iterator needs to maintain an internal pointer for the current position and return false from the valid() method as soon as the position has reached the end. Inside each iteration the key() and current() methods are called to get the $key and $value for the current position.

An Iterator for MySQLi_Result objects

The next example shows how you can create an Iterator for MySQLi results.

Once you know what each of the functions should do, implementing them for a MySQLi result is really straightforward. The only tricky part is that the mysqli_fetch_* methods advance the MySQLi_Results' internal pointer, while the current() method on our Iterator shouldn't. We can solve this by calling $Result->fetch_array() in rewind() and next() instead and prefetching the current row.

class ResultIterator implements Iterator
{
  protected $Result;
  protected $fetchMode;
  protected $position;
  protected $currentRow;

  /**
   * Constructor
   * @param MySQLi_Result $Result
   * @param constant $fetchMode (MYSQLI_ASSOC, MYSQLI_NUM, MYSQLI_BOTH)
   */
  public function __construct($Result, $fetchMode = MYSQLI_ASSOC)
  {
    $this->Result = $Result;
    $this->fetchMode = $fetchMode;
  }

  /**
   * Destructor
   * Frees the Result object
   */
  public function __destruct()
  {
    $this->Result->free();
  }

  /**
   * Rewinds the internal pointer
   */
  public function rewind()
  {
    // data_seek moves the Results internal pointer
    $this->Result->data_seek($this->position = 0);

    // prefetch the current row
    // note that this advances the Results internal pointer.
    $this->currentRow = $this->Result->fetch_array($this->fetchMode);
  }

  /**
   * Moves the internal pointer one step forward
   */
  public function next()
  {
    // prefetch the current row
    $this->currentRow = $this->Result->fetch_array($this->fetchMode);

    // and increment internal pointer
    ++$this->position;
  }

  /**
   * Returns true if the current position is valid, false otherwise.
   * @return bool
   */
  public function valid()
  {
    return $this->position < $this->Result->num_rows;
  }

  /**
   * Returns the row that matches the current position
   * @return array
   */
  public function current()
  {
    return $this->currentRow;
  }

  /**
   * Returns the current position
   * @return int
   */
  public function key()
  {
    return $this->position;
  }
}

At this point we have a fully functional ResultIterator that can be used like this:

foreach (new ResultIterator($DB->query($query) as $row)) {
  // do something with $row
}

While this is already quite nice it isn't perfect. We still need to create our own ResultIterator object, which - as i will show in the next part - isn't really necessary.

Extending MySQLi and MySQLi_Result

To avoid having to create our own ResultIterator object we can use the simple IteratorAggregate interface.

IteratorAggregate extends Traversable
{
  function getIterator();
}

As soon as foreach() is called on an object that implements this interface the getIterator() method is called and can then create a new Iterator for the object. We can apply this interface to our MySQLi_Result objects.

To do so, we will have to extend both MySQLi_Result and MySQLi. Let's start by showing how to extend the MySQLi_Result class for this purpose.

class QueryResult extends MySQLi_Result implements IteratorAggregate
{
  public function getIterator()
  {
    return new ResultIterator($this);
  }
}

As you can see this is really simple.

There is just one small problem. Even though we provided an extension for the MySQLi_Result class, the MySQLi::query() method still returns a standard MySQLi_Result object. To solve this problem we need to override the query() method, which can be done like this:

class DB extends MySQLi
{
  public function query($query)
  {
    // query succeeded
    if ($this->real_query($query)) {

      if ($this->field_count) {

        // SELECT, SHOW, DESCRIBE
        return new QueryResult($this)

      } else {

        // INSERT, UPDATE, DELETE
        return true;

      }
    }
  }

This one is a bit tricky. First we use the real_query() method to run the query. Then if there was no error in the query, we use field_count to find out if we just did a select query. If so we create our own QueryResult object and return it, otherwise we just return true to indicate the query was succesful. The trick here is that by creating our own QueryResult after calling real_query() we are basically doing the same as the store_result method does.

At this point we can finally enjoy the results of our work

$DB = new DB('localhost', 'root', '', 'test');
$query = 'SELECT ... ';

foreach ($DB->query($query) as $row) {
  // do something with $row
}

Taking it one step further

At the beginning of this article i showed the following example:

// foreach loop on a query that selects 2 columns
foreach ($DB->query($query) as $key => $value) {
  // do something with $key and $value
}

If we select 2 columns and want the first column as $key and the second as $value all we need is a specialized version of our ResultIterator.

class KeyValueResultIterator extends ResultIterator
{
  public function __construct($Result)
  {
    $this->Result = $Result;
    $this->fetchMode = MYSQLI_NUM;
  }

  public function current()
  {
    return $this->currentRow[1];
  }

  public function key()
  {
    return $this->currentRow[0];
  }
}

Likewise, when we are selecting only 1 column we don't need to return an array from the current method(). For this case we can create a third version of our ResultIterator.

class SingleColumnResultIterator extends ResultIterator
{
  public function __construct($Result)
  {
    $this->Result = $Result;
    $this->fetchMode = MYSQLI_NUM;
  }

  public function current()
  {
    return $this->currentRow[0];
  }
}

The downside of these 2 specialized ResultIterators is that we would have to create them manually again if we wanted to use them. Unless ofcourse, we alter our getIterator() method like so:

public function getIterator()
{
  switch ($this->field_count) {
    case 1:    return new SingleColumnResultIterator($this);
    case 2:    return new KeyValueResultIterator($this);
    default:   return new ResultIterator($this);
  }
}

Conclusion

By creating our own Iterator for MySQLi_Results we can now easily iterate over them using foreach(). We also solved the special cases when selecting only 1 or 2 columns. In the process we saw some of the new things we can do with the SPL.

But what about the performance?

Well, as was to be expected, it's a bit slower than a while() loop using fetch_row() directly. As we saw in the example for() loop, the Iterator interface has the overhead of 3 or 4 functions calls per iteration (depending on whether key() is called or not).
I included some source code with a few benchmarks at the bottom of this article if you need more specifics.

Further improvements

If you want to further improve the QueryResult class or try some other SPL interfaces, you could take a look at the SeekableIterator and Countable interfaces.

Or download the [source code] used for this article to play around with.

Calculate ISO 8601 week and year in javascript

Mon, 27 Apr 2009 14:58:55 +0200

For one of our projects I needed to be able to calculate the current weeknumber and corresponding year of a given date... in javascript. Not really a big deal, just calculate the number of weeks between the requested date and the 1st of January that year and you're done, right? Wrong!

Why do we use ISO week numbering?

In our framework (including modules) we use the ISO 8601 standard for week numbering. We've chosen this standard because it is very related to western european workweeks: weeks start on monday and the first week of the year is the first week with a workday in it. Our clients are mostly businesses operating in western europe and therefore this standard is the most obvious choice.

Why this article?

Our framework is built using PHP. In PHP requesting the week and / or year number according to this standard is easy:

// Week and year number using date()
$w = date('W');
$y = date('o');

// Week and year number using strftime()
$w = strftime('%V');
$y = strftime('%G');

But in javascript no built-in methods are available to calculate these values.

ISO 8601 week numbering

This (part of the) standard primarily defines two rules:

Weeks start on monday
The first week of the year is the week with the first thursday of that year

An alternative to the second rule is that the first week of the year is the week with the 4th of January in it. This rule is mostly more convenient in coding.

As a result of this second rule the week with the 1st of January isn't always week number 1. Another result is that some years have 53 weeks instead of 52.

Extending the javascript Date class

To be able to get the week and year numbers in javascript we extend the Date class with two methods: getWeek() and getWeekYear().

Extending a class in javascript can be done by adding properties to the prototype object of that class. In the code below we define Date.prototype.getWeek() and Date.prototype.getYearWeek(), these new methods can be called directly on instances of the Date class:

var today = new Date();
alert("The current week number is: " + today.getWeek());

Date.getWeek()

First, take a look at the code below:

/**
 * Get the ISO week date week number
 */
Date.prototype.getWeek = function () {
	// Create a copy of this date object
	var target  = new Date(this.valueOf());

	// ISO week date weeks start on monday
	// so correct the day number
	var dayNr   = (this.getDay() + 6) % 7;

	// Set the target to the thursday of this week so the
	// target date is in the right year
	target.setDate(target.getDate() - dayNr + 3);

	// ISO 8601 states that week 1 is the week
	// with january 4th in it
	var jan4    = new Date(target.getFullYear(), 0, 4);

	// Number of days between target date and january 4th
	var dayDiff = (target - jan4) / 86400000;  

	// Calculate week number: Week 1 (january 4th) plus the  
	// number of weeks between target date and january 4th  
	var weekNr = 1 + Math.ceil(dayDiff / 7);  

	return weekNr;  
}

The first thing we do is find out the difference in days between the requested date and the 4th of January that same (calendar) year. We subtract the two Date objects, which will result in a number of milliseconds, and divide that by the number of milliseconds in a day: 86400000. Since weeks always start on monday (rule #1) we are more interested in the difference in days between January 4th and the monday of the target week, so we correct this day difference. Next we convert this amount of days to weeks and add one (since it was the difference between week #1 and the target date).

This was the general part of the method, next we need to correct for the exceptions (remember that the 1st of January might be in a week of the previous year?). If the result of our method is less than 1, we know this date belongs to the last week of the previous year, simply return the result of the getWeek() method for December 31st of the previous year.

On the other hand the last few days of the year might belong to a week of the next year. Since at most 3 days can belong to the next year (remember that the first week is the week with January 4th) we only need to check this exception when the date is after December 28th. We return 1 when the monday of the requested week is after the 28th of December.

Date.getWeekYear()

That was the getWeek method, the most important one of the two. The getWeekYear() method uses getWeek() to determine what year to return.

This method is very simple, just return the calendar year except when:

The requested date is in January but the getWeek() method returns 52 or 53
The requested date is in December but the getWeek() method returns 1

The code below (and it's comments) speak for itself ;)

/**
* Get the ISO week date year number
*/
Date.prototype.getWeekYear = function () 
{
	// Create a new date object for the thursday of this week
	var target	= new Date(this.valueOf());
	target.setDate(target.getDate() - ((this.getDay() + 6) % 7) + 3);
	
	return target.getFullYear();
}

So that's about all you'll need to calculate ISO 8601 year and week numbers in javascript, happy coding!

Optimisation

After reading the site from Dr J R Stockton that Rick recommended in his post I stole a simple trick to make the code a little shorter. Instead of handling the edge cases I first change the date to the thursday of the requested week so the week year number equals the calendar year. I did keep the code more verbose than Stockton did though, just so you understand what exactly happens.

CSS Rendering problems in IE8? Fix your conditional comments!

Wed, 19 Aug 2009 12:22:42 +0200

Are your existing websites incorrectly rendered by IE8? Do you have a hard time getting IE8 to render your CSS correctly? The core problem may be incorrect usage of conditional comments... Ofcourse, a fix is available in this article!

Today, I talked to some friends & colleague front-end developers, who had trouble with the way IE8 renders their existing websites and websites under development. We all know that developing for IE7 (and lower versions) can be a pain. Using conditional comments is a commonly used method to fix IE proprietary bugs. Please note that the point of this article is not to discuss the usage of conditional comments. My opinion on this comes close to what Jens Meiert said about this subject.

Conditional comments are conditional statements interpreted by IE only. They first appeared in IE5 and are supported through at least IE8. An example of a conditional comment:

In the above example, the conditional comment is interpreted only by IE, which loads the ie.css file with IE-specific fixes. The idea is simple and effective. So what's any of this got to do with IE8 rendering problems?

"Fix" IE8 rendering

The most common way the conditional comments were used when IE7 was the latest IE version, was as following:


    < link rel='stylesheet' type='text/css' href='http://techblog.procurios.nlie.css'/>

The idea was to return a CSS-file for all versions of IE to fix the most common version-wide IE bugs, and another CSS-file for IE6 and lower. Since IE8 supports conditional comments, the first comment in the above example affects IE8 as well.

IE8 renders CSS2.1 fine, and doesn't behave nearly as buggy as the previous versions. This means that the fixes in your ie.css probably conflict with IE8's rendering. The solution is to change the first conditional comment into the following:

You'll see that most of the rendering problems in IE8 will disappear. Being aware of this could save you time and prevents you ending up with adding a meta-tag to make IE8 render in IE7 compatability mode. Have fun developing!

Testing Internet Explorer 8: CSS3 for Web Design

Wed, 19 Aug 2009 12:22:52 +0200

CSS3 introduces many exciting new features for webdesigners and webdevelopers. These features can help you create some common webdesign techniques like rounded corners, opacity and dropshadows just by typing out some CSS rules. Let's find out whether Microsofts newly born baby, Internet Explorer 8, supports these CSS3 features or not.

Internet Explorer 8 Final released

It was thursday the 19th of March when Microsoft released Internet Explorer 8. Microsoft said the following about IE8:

Get where you want to go faster with real world performance.
Use more of the web, easier.
Stay safer from real world threats.
Build on real world interoperability, standards, and compatibility.

Nice. A safe, fast and up-to-date browser, released by Microsoft. But how modern is it? Does it support any of the new features CSS3 introduces?

An introduction to CSS3

CSS3 is the new CSS-standard, which is set to take over from CSS2.1. CSS3 introduces a huge variety of ways to create flexible and attractive designs. Even though the CSS3 specification is still in development, there are some very exciting new features for webdesigners to be implemented in the specifications. These features can help you create some common Web design techniques like rounded corners, transparent boxes and dropshadows just by typing out some CSS rules.

I'm aware of the fact that in reality we can't use any of these CSS rules yet, because not every browser fully supports them. Some of the CSS3 features are already implemented in modern browsers like Firefox 3, Safari 3, Google Chrome 1 and Opera 9.6. Does Internet Explorer 8 join this list of modern browsers, supporting the new features CSS3 is going to introduce?

Testing the CSS3 implementation

Transparency using RGBA and opacity

The use of PNG files with alpha-transparency is a modern and common design feature. CSS3 enables you to directly specify an alpha-value or opacity setting in your CSS. A simple test case in which we test the support of the opacity and rgba properties:

This is how it looks in compatible browsers.
Full support in: Firefox 3, Safari 3, Chrome 1.
Partial support in: Firefox 2, Opera 9.6 (no support for rgba).
No support in: Internet Explorer 8.

Rounded corners

Another commonly used webdesign feature is rounded corners, which are usually made by placing background-images or using empty elements. If you need some examples of current techniques, check out this list of 25 rounded corner techniques. CSS3 allows rounded corners to be set in your CSS files, which would eliminate every of the existing techniques. Let's see if your browser supports the border-radius property:

This is how it looks in compatible browsers.
Full support in: Firefox 2 & 3, Safari 3, Chrome 1.
No support in: Internet Explorer 8, Opera 9.6.

Border Image

A new border feature CSS3 introduces, is the border-image property. With this feature you can set an image to be used as a border instead of the normal border of an element. This feature is split up into border-edge-images and border-corner-images. If your browser supports this CSS3 feature, a colorfull border should be displayed in the following test case:

This is how it looks in compatible browsers.
Full support in: Firefox 3.1 (upcoming), Safari 3, Chrome 1.
No support in: Internet Explorer 8, Opera 9.6.

Box Shadow

CSS3 has a nice new feature called box-shadow, to create dropshadows with CSS instead of using images. The property takes four values: three lenghts (the shadow offsets & shadow blur) and the color of the shadow:

This is how it looks in compatible browsers.
Full support in: Firefox 3.1 (upcoming), Safari 3, Chrome 1.
No support in: Internet Explorer 8, Opera 9.6.

Text Shadow

Even though i'm not a fan of heavy usage of dropshadows, CSS3 also introduces a text-shadow feature. Who needs Photoshop? ;) The property text-shadow takes the same four values as box-shadow:

This is how it looks in compatible browsers.
Full support in: Firefox 3.1 (upcoming), Safari 3, Opera 9.6.
No support in: Internet Explorer 8, Chrome 1.

@font-face (custom fonts)

One of the coolest new features of CSS3, is enabling the developer to use his/her own fonts with the Web Fonts module. The @font-face rule provides the bridge between an author's font specification and the font data, which is the data needed to format text and to render the abstract glyphs to which the characters map - the actual scalable outlines or bitmaps. Fonts are referenced by style sheet properties. The @font-face rule takes two properties: the definition of the font-family and the source (url) of the font:

This is how it looks in compatible browsers.
Full support in: Safari 3, Opera 10 (upcoming), Firefox 3.1 (upcoming).
No support in: Firefox 3.0, Internet Explorer 8, Chrome 1.

HSL & HSLA Colors

CSS3 adds numerical hue-saturation-lightness (HSL) colors as a complement to numerical RGB colors. According to the W3C, the RGB color model has two limitations:

RGB is hardware-oriented: it reflects the use of CRTs.
RGB is non-intuitive. People can learn how to use RGB, but actually by internalizing how to translate hue, saturation and lightness, or something similar, to RGB.

HSL is far more intuitive to use. You can guess the colors you want to use, and if needed, tweak them a little. Besides, you can easily create a set of matching colors by setting the hue to a constant and varying the lightness and saturation. The A in "HSLA" stands for alpha, and allows you to add an alpha-value (as in rgba). If your browser supports HSLA colors, you should see three red blocks, each with a different alpha setting:

This is how it looks in compatible browsers.
Full support in: Safari 3, Firefox 3, Opera 9.6, Chrome 1.
No support in: Internet Explorer 8.

Multiple Backgrounds

Another very cool feature CSS3 Background & Borders Module introduces is the ability to add multiple background images to one box. The 'background' property may have multiple values, separated with comma's:

This is how it looks in compatible browsers.
Full support in: Safari 3, Chrome 1.
No support in: Firefox 3, Opera 9.6, Internet Explorer 8.

CSS3 selector test

From w3.org:

In CSS, pattern matching rules determine which style rules apply to elements in the document tree. These patterns, called selectors, may range from simple element names to rich contextual patterns. If all conditions in the pattern are true for a certain element, the selector matches the element.

CSS3 also introduces an enhanced selector-model, which we can test easily using the CSS3 selector test provided by CSS3.info. I've saved the testresult of Internet Explorer 8 (if your browser has scaled the image down: click to zoom in):

Conclusion

Based on the test cases in this article, we can conclude the following:

Internet Explorer 8 supports none of the exciting new features CSS3 introduces to webdesigners and front-end developers.
Internet Explorer 8 only supports 22 of the 43 available CSS3 selectors.

Comparing the CSS3 implementations of IE8 to its modern competitors, we'll sadly notice that IE8 comes last across the finish line. Microsoft did a huge step forward with the release of Internet Explorer 8 where it comes to support of CSS2.1 and rendering speed, but it's still far behind it's competitors.

Thumbs up to Safari, which supports every tested CSS3 features.

Finally

Did you spot a mistake? Please send your feedback and you'll be rewarded with your nickname next to the corrected phrase ;). Feel free to comment or use the contact form on the homepage.

Protection against SYN Floods

Mon, 16 Mar 2009 10:34:27 +0100

Just after lunch our office is a calm and peaceful place. The only sound comes from a handful of developers discussing possible solutions for a challenging problem in a project...

Suddenly our nagios IRC bot politely informs us that the HTTP service is down for one of the servers. Fortunately SSH still works like a charm. A quick inspection reveals no peculiarities, except for an amazing number of Apache clients doing nothing. A quick glance in the logs learns us that Apache has reached its MaxClients setting. An Apache restart doesn't help, it reaches its MaxClients setting again within the minute.

# tail /var/log/apache2/error.log

[Fri Dec 05 14:31:30 2008] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Fri Dec 05 14:50:54 2008] [notice] caught SIGTERM, shutting down
[Fri Dec 05 14:50:56 2008] [notice] Apache/2.2.x (Debian) configured -- resuming normal operations
[Fri Dec 05 14:51:35 2008] [error] server reached MaxClients setting, consider raising the MaxClients setting

Could be a (D)DoS attack. However, the server doesn't show typical signs of a DoS attack like high load or network traffic. Next stop is netstat which shows us a huge number of tcp connections in SYN_RECV state:

# netstat -n -p TCP

tcp        0      0 10.1.1.1:80       91.64.4.146:64979       SYN_RECV   -
tcp        0      0 10.1.1.1:80       84.24.103.112:4005      SYN_RECV   -
tcp        0      0 10.1.1.1:80       79.223.69.239:61510     SYN_RECV   -
tcp        0      0 10.1.1.1:80       67.86.135.44:43312      SYN_RECV   -
tcp        0      0 10.1.1.1:80       86.88.67.226:50600      SYN_RECV   -
tcp        0      0 10.1.1.1:80       173.20.137.110:3813     SYN_RECV   -
tcp        0      0 10.1.1.1:80       84.58.10.121:4878       SYN_RECV   -
tcp        0      0 10.1.1.1:80       91.37.40.151:2408       SYN_RECV   -
tcp        0      0 10.1.1.1:80       173.20.137.110:3441     SYN_RECV   -
...

Lots of half-open connections are a good indication of a SYN Flood attack. So we need to do two things:

Harden the TCP/IP stack
Block attackers

Harden the TCP/IP stack

If you're not familiar to SYN Floods or other TCP-based resource starvation attacks, read this SecurityFocus article. It also covers hardening your TCP/IP stack against SYN Flooding and SYN Spoofing. Most important is enabling SYN Cookies.

Block attackers

Now we've hardened the TCP/IP stack, spoofing has been eliminated. The attacking source address is really the attacker, so it makes sense to block it. However, these days it's not very hard to create, buy or rent a large botnet consisting of thousands of hacked computers ready to be engaged in massive DDoS attacks. So manually block IP's won't work.

We'll have to write a script to automatically block attacking IP's. The script consists of three parts:

Find out which IP's to block
Block IP
Unblock IP after 24 hours

The first part is the hardest part. In our case, we could just take all IP's having too many connections in SYN_RECV state. In your case you might need to use another condition, e.g. count requests to a specific URI.

$netstat = shell_exec('/bin/netstat -apn | grep SYN_RECV');
$lines = preg_split('/[\r\n]+/', $netstat);

$ips = array();
foreach ($lines as $line) {
    $fields = preg_split('/\s+/', $line);
    if(preg_match('/[\d\.]+/is', $fields[4], $match)) {
       $key = $match[0];
       $ips[$key] = empty($curr[$key]) ? 1 : 
      ($ips[$key] + 1);
    }
}

Now we have all IP's and number of concurrent connections in SYN_RECV state. Next step is block IP's with more than 50 connections. This number is quite random, any sane treshold will do.

foreach ($ips as $key => $val) {
    if ($val > 50) {
       exec("sbin/iptables -A INPUT -s " . 
      escapeshellarg($key) . " -j DROP");
    }
}

For the last step you need to create your own administration of blocked IP's and timestamps. Check out the full script [Download] for this and some other features like a whitelist and simple logging.

Reinventing the wheel?

You might wonder why we try to reinvent the wheel. Though avoiding to reinvent the proverbial wheel is a good practice and key principle in software development, it's useful to rethink the wheel concept from time to time. Or to quote Jeff Atwood: Don't Reinvent The Wheel, Unless You Plan on Learning More About Wheels.

Most high profile websites will probably have a dedicated intrusion detection or prevention system (IDS/IPS) in place. IT security companies can implement a commercial solution, e.g. Cisco, IBM or Juniper systems. You could also try to build one yourself. Snort is a free and open source IDS/IPS. It's the de facto standard and used by many commercial solutions as well.

However, for most websites or web applications an IDS/IPS is overkill. Start with good server configuration, hardened OS and server software and a good router and firewall. And add an IDS or custom IP blocking script if necessary.

Finally

In this case we controlled the attack within 5 minutes after we launched the block-ips script. The websites came back online and the attack stopped after three days.

Transforming the button element with sliding doors and image sprites

Fri, 13 Mar 2009 20:05:20 +0100

One of the elements lacking a standardized look is the button element. In this article, we're going to transform the buttons image from fixed to flexible and good looking using HTML and CSS. Let's get styling!

Introduction

In the past years, webdesign and webdevelopment have become mature. The user is the center of development, and deserves an user interface which supports and leads users to accomplish the goal of the website. Building consistent, attractive and cross-browser compatible user interfaces can be a tough job. This especially becomes apparent when you're busy giving your forms the desired look and feel. One of the form-elements lacking a standardized look is the button element. Let's go tame that button, so it fits into any design.

Introducing the button element

First, let's get to know the button element. The HTML 4.01 specification provides the following information about the button element:

Buttons created with the BUTTON element function just like buttons created with the INPUT element, but they offer richer rendering possibilities: the BUTTON element may have content. For example, a BUTTON element that contains an image functions like and may resemble an INPUT element whose type is set to "image", but the BUTTON element type allows content.

Visual user agents may render BUTTON buttons with relief and an up/down motion when clicked, while they may render INPUT buttons as "flat" images.

That sounds great. Even though the W3C recommendations states two advantages of using the button element, it seems that many developers forgot about it, or aren't even aware of its existence.

With that in mind, let's find out the possibilities of the button element. User Interfaces are often heavily dependent on background images, icons and other fancy layout-elements. We want our buttons to blend into that perfectly as well, and accomplish the following:

Use the most semantic element HTML4 offers us to submit a form: the (submit) button element.
Create a flexible, attractive button using:
- The sliding doors technique, introduced by A List Apart.
- Use an image sprite (there's plenty of information about image sprites on the web) for fast hover changes and less HTTP requests (faster page loading).
Create a cross-browser, consistent look.

Our goal

Before we get into technical information and details, let's take a look at what we are going to build. I've created three buttons, two with basic terms, and another one with some more text. The buttons look great, have round corners, a hover state and a focus state. Feel free to move your mouse over the buttons, or even click them ;) (Chrome users will see a minor bug, we'll come to that later):

The markup and images

Let's have a closer look at one of the buttons markup. We'll inspect the following button:

As you can see, the markup is really basic. For flexibility reasons, we wrapped the inner text into a span element. This makes it possible for us to use the sliding doors technique, which enables us to blend the button into any user interface. Without the span element inside the button, we wouldn't be able to create a button with round edges and a flexible width.

To give the button it's look, I used an image sprite containing the three states our button has (the hover and focus state are the same):

The CSS

Now, its time to style our buttons. Let's start with the button element:

	button {
    float: left;
    padding: 0 15px 0 0;
    height: 25px;
    border: 0;
    cursor: pointer;
    text-align: center;
    color: #f9f9f9;
    background: transparent url(images/button-sprite.png) no-repeat right top;
    font: bold 14px "Myriad Pro", arial, sans-serif;
    overflow: visible;
	}

Some of the properties and values (padding, background and font) are written with shorthand properties and values. I want to address three values and explain why they'r used:

float: left;
I wanted to point out that you should, logically, only apply this value if you want your button to float. In my example, I got three buttons floating next to eachother, hence the value is present.
padding: 0 20px 0 0;
The padding-right is used to show the top-right image in our sprite. If you remove the padding, the span element will overlap the button completely, hiding the right edge of the button.
overflow: visible;
For some reason Internet Explorer decides to stack padding depending on the width of the content inside the element. This value eliminates the stacking. More information here and here.

Secondly, the span element inside our button needs some CSS to get the job done.

button span {
    display: block;
    padding: 0 0 0 15px;
    height: 25px;
    line-height: 25px;
    white-space: nowrap;
    background: transparent url(images/button-sprite.png) no-repeat left -25px;
	}

Again, i'd like to address three values in our CSS rule. Even though it may sound really basic, questioning why certain values are present, is a good habit.

display: block;
By default, a span element is an inline element. We need to be able to add box properties like height and padding to our element, so we need to change its display setting to block.
line-height: 23px;
To vertically align our text in the middle of our button element, we'll have to set the line-height to a value within the 20-25px range. I chose 23px.
white-space: nowrap
This settings makes sure that all of the words you want to show on your button are visible in Internet Explorer. If you forget this setting, you'll notice that 'This is my sentence' will display as 'This'.

Thirdly we'll add the CSS necessary to add effects on the hover- and focusevents. Please note that Internet Explorer 7 lacks the implementation of the :focus pseudoclass and Internet Explorer 6 and lower lack the :hover pseudoclass as well. Both can be simulated / fixed using javascript. For example, Dean Edwards javascript library fixes the most common IE6 and IE7 problems.

button:hover,
button:focus {
    background-position: right -56px;
}

button:hover span,
button:focus span {
    background-position: left -84px;
}

button:focus {
    margin-top: 1px;
}

We've come to the point that we're ready to view the buttons in a couple of browsers. Let's see:

You'll notice that there's still a little bug in Firefox and Google Chrome. The differences in browser implementations become really apparent now. Let's find out whats happening, and additionally how to fix the problem.

If you inspect our CSS closely, you'll notice that there isn't a conflicting value that causes the extra padding in our button. So where does it come from? Every browser has implemented User Agent stylesheets. If a browser gets a HTML document served without styling information attached to it, it will still show some basic formatting: the User Agent stylesheets. Jens Meiert has written a good article on User Agent stylesheets, there's even a list with CSS files you can download.

If you open Firefox' forms.css, you'll see the following code:

button::-moz-focus-inner,
input[type="reset"]::-moz-focus-inner,
input[type="button"]::-moz-focus-inner,
input[type="submit"]::-moz-focus-inner,
input[type="file"] > input[type="button"]::-moz-focus-inner {
    padding: 0px 2px 0px 2px;
    border: 1px dotted transparent;
}

Mozilla supports a number of extensions to CSS that are prefixed with '-moz'. These include a number of capabilities, including rounded borders, and the ability to specify different box width and height calculations to more easily support browers that perform such calculations in a non-standard way. An overview of all available settings can be found here.
So apparently the Mozilla extensions are conflicting with our CSS. The first selector is the one we need to fix:

button::-moz-focus-inner /* fixing evil default firefox padding */ {
    padding:0;
    border:none;
}

There we go, the button we created will show correctly in Firefox as well. So what happens in Google Chrome?

Google's goal with Chrome form implementation, is to behave exactly the same as Firefox. Obviously and unfortunately, they introduced the same bug, which in Chrome can't be fixed. I'm still searching for a way to get around it, but I'm sure Google will fix it sooner or later. The bug has been reported to the Google Headquarters.

Summary

We've looked at the button element and discovered it's possibilities and limits. We've seen that its possible to create flexible and good looking button elements, with cross-browser support.

Pros

Semantic HTML;
Fast :hover changes;
Less HTTP requests;
Good looking, flexible buttons.

Cons

Needs a 'hack' to disable the native User Agent CSS in Firefox.
Google Chromes similar bug can't be fixed at the moment (I dont worry too much about Google Chrome yet, though).

The method described in this article, covers the usage of image sprites. It's possible to get this working in Chrome aswell, but you will have to make separated images instead of using a sprite. If there's demand for an article which explains how to get this done, I'll write a follow up.

The code

button {
    float: left;
    padding: 0 15px 0 0;
    height: 25px;
    border: 0;
    cursor: pointer;
    text-align: center;
    color: #f9f9f9;
    background: transparent url(images/button-sprite.png) no-repeat right top;
    font: bold 14px "Myriad Pro", arial, sans-serif;
    overflow: visible;
}

button span {
    display: block;
    padding: 0 0 0 15px;
    height: 25px;
    line-height: 25px;
    white-space: nowrap;
    background: transparent url(images/button-sprite.png) no-repeat left -25px;
}

button:hover,
button:focus {
    background-position: right -56px;
}

button:hover span,
button:focus span {
    background-position: left -84px;
}

button:focus {
    margin-top: 1px;
}

button::-moz-focus-inner /* fixing evil default firefox padding */ {
    padding:0;
    border:none;
}

If you feel something is missed out here, or needs further explanation, feel free to post your comment!