etags and form tags

Starting Use Case

Early experience taught me that browsers don't really pay to expiration times. I could serve up a file set to expire a year from now, and pretty soon the browser would be asking for the file again. This proved to be inefficient.

Reading the protocols quickly clued me in that I needed to support etags. An etag is an opaque string that the client sends with an If-None-Match header set to the value for etag when the file was originally served.

To serve files from a PostgreSQL table, I needed an etag value to go with the file contents. To PG, this is simply a serial field. So simply take whatever the client says is the etag, and compare it to what is in PG, except I have to account for the fact that the data may be toxic in ways I don't understand. It could be carefully crafted to break my parser or do other evil. I needed to be able to know that the etag the client just handed to the server came from the server and is in fact, safe to pass to a C function.

I needed to encrypt a server constant and the serial attribute for the etag from PG and give the client an opaque token that must be checked and decoded before it is used.

The original use has been overshadowed by form tags which provide complete click stream validation, and the extended use has called for the original idea to be enhanced. Currently, this is the third iteration.

How it works

It looks like this
"1cc3afec4a5abb0a3aaafe393fcfb56d.e820eab36f7195d7b1fedff9645fb877f9c05306ee8c55f9d949cd22652c681a"

The first 32 characters are a random 128 bit initiallization vector. The 64 hex characters after the dot are two 64 bit numbers, and 128 bits of payload. They are encrypted using AES and a key that is either from qzforms.conf variable SERVER_KEY, or if not set a value is selected at random.

The first 64 bits past the dot is a server token, a value that is constant for the duration the server is running. It may be set in qzforms.conf as SERVER_TOKEN, or if not set a value is selected at random. Having the correct value only proves a high probability that the tag originated from this server.

The second 64 bits is the domain token. It proves the tag is not being reused outside of its original purpose. There are three domains.

1. Session token - Random at server startup.
2. etag for HTTP_IF_NONE_MATCH get requests - Random for each session
3. Form tag - Required for every http post - Random for each session

The last 32 hex characters are the payload, or etag value storing a 16 byte character array. For etags, this is the value from a serial or big serial attribute in a Postgresql table. For sessions and form tags, this is 15 non-zero bytes followed by one zero byte, which is used a the key for a hash table lookup.

To separate the encoding and decoding of tags from QZForms, a process is forked at startup called tagger. Communication is via the qzforms.conf value for QZ_TAGGER_SOCKET, "run/tagger.sock" by default. The tagger will read requests from the socket.

If the tagger proccess reads 24 bytes, it is interpetted as a 64 bit unsigned integer domain token, followed by 128 bits of payload. A tag is created from the data and written to the requesting socket connection.

If it is 97 bytes (or two bytes more but quoting 97 bytes), and it's hex and the dot is in the right place, it is decoded. If the server token matches, then it's a valid tag so the domain token and payload are written to the requesting socket connection.

The one value for which a tag will not be created is all zeros. A request for a zero tag will return nothing. A request to validate an invalid tag will return all zeroes as the indication of failure.

Form Tags

When a form is created, an HTML attribute is added with the name "form_tag". The tag payload is 15 non-zero bytes, 1-255, followed by 1 zero byte. (119.9 bits of randomness, log(255¹⁵,2) It is used for a hash table lookup to get a form record which contains the form the tag is allowed to access, the primary keys of the Postgresql table row, and some bookkeepking details including an is_valid flag.

Each form served has an expiration time. This is in qzforms.conf as FORM_DURATION. A page may have a Javascript function that will use XMLHttpRequest to extend the expiration time. The function form_refresh is provided in qzforms.js to do this.

Every HTTP request that includes post data must have two validation tokens in order to proceed. There must be an HTTP cookie with the session key and there must be a form tag. The session key is constant for the duration of the session. The form tag is ephemeral with new tags created with every page creating an authorized click stream.

Forms tags can be flagged as use only once, in which case the form is invalidated after the first use. If every form is flagged use only once, and a minor error occurs the session can be left with no valid actions forcing the user to log back in to continue working.

Session Keys

Tags are also used for session keys, but this use is not especially interesting. The cookie created for the session key is given a path of the first segment of the URL, /qz/ in the documentation, but it can be anything. Multiple login sessions are possible by having different base segments mappping to the same QZForms instance in the web server.