Using Vidocq and Hardwood

This article is an AI-assisted translation of the original French content.

To explore new Java frameworks, I rewrote a POC about HTTP range requests: a Spring server exposed static Parquet files, which DuckDB read remotely. The goal was to check that:

1. thanks to range requests and the block-based structure of the Parquet format, DuckDB can read an entire column without downloading the whole file;
2. Spring MVC correctly handles range requests on static files (which led to a small contribution of mine to its documentation).

This time, Vidocq replaces Spring and Hardwood replaces DuckDB. Along the way, a third tool joined in to measure the data exchanged during the tests: JFRUnit.

A quick overview of the frameworks and libraries used Link to heading

Vidocq Link to heading

Vidocq aims to implement Jakarta EE and MicroProfile without any external dependency, on a European and sovereign stack, independent of any commercial vendor. Still under development, it already offers demo releases.

It ticks all the boxes of a modern Java framework: Java 25, virtual threads, code bound at compile time (no reflection), AOT compatibility, jpackage and native compilation, and an architecture made of small independent building blocks.

It also brings Java modules (JPMS, introduced in Java 9) back into fashion: they are ubiquitous in the JDK but shunned by the major frameworks.

We will only use one small building block here: Chappe, a lightweight HTTP server.

Hardwood Link to heading

Hardwood, which appeared less than a year ago, fills a gap: reading Parquet files in pure Java, without a cascade of dependencies (only the compression libraries are optional). Its developers pay particular attention to performance, with benchmarks to back it up. Writing support is coming with version 1.1 (in beta), and a CLI is also available.

JFRUnit Link to heading

Since Java 14, the RecordingStream class makes it possible to consume a JFR recording continuously from code. You can therefore capture events during a test and use them in assertions: performance, disk access, network traffic…

JFRUnit takes care of the JFR plumbing and provides an assertion API. Here, it will be used to measure the data received by an HTTP client.

Implementation Link to heading

A simple JUnit test starts a Chappe server and a Hardwood client that communicate over HTTP on the loopback interface. The client reads the Age column of a Parquet file served by Chappe and computes its average; meanwhile, a JFR recording measures the volume of data exchanged.

Overall software architecture Link to heading

Hardwood extension to read files over HTTP Link to heading

The titanic.parquet file contains data about the Titanic’s passengers. With DuckDB, a simple SQL query was enough thanks to its HTTP extension: select avg("Age") from 'http://localhost:8080/titanic.parquet'.

Hardwood, on the other hand, does not provide an InputFile for HTTP. But the interface is not sealed and only has five methods:

• open: since the JDK’s HttpClient manages its own connection pool, we can’t really “open” a connection here. Instead, we send a HEAD request that validates the URL and the network configuration, warms up a connection that will probably be reused afterwards, and retrieves the file size.
• readRange: returns the body of a partial GET request whose bounds are the ones passed as arguments.
• length: returns the size of the resource, obtained during the call to open.
• name: returns the URL, just like MappedInputFile.
• close: closes the HttpClient.

This gap is hardly surprising for such a young library, especially since its model lends itself very well to it: readRange and length map directly to range requests and HEAD requests.

The implementation, deliberately simple (no optimization and no elaborate error handling), can be found in poc.java.httprange.hardwood.HttpInputFile. While handy for a POC, the JDK’s HTTP client is not necessarily suited for production: no cache, no compression, a few bugs and performance issues.

Hardwood’s cache: RangeBackedInputFile Link to heading

Since the JDK’s HTTP client has no cache, we wrap HttpInputFile in a dev.hardwood.internal.reader.RangeBackedInputFile, which adds Hardwood’s internal cache. This is what the test does.

Hardwood extension to make reading a column easier Link to heading

To simplify reading a column, the poc.java.httprange.hardwood.ParquetFileReaderExtension#streamColumn method exposes its values as a DoubleStream, handling missing values, following the algorithm described in the documentation. Once again, extending the library is straightforward.

Vidocq/Chappe extension to handle HTTP range requests Link to heading

On the server side, Chappe must answer the client’s range requests. A server is declared like this:

try (io.vidocq.chappe.api.Server server = Server.builder()
        .port(8080)
        .handler(handler)
        .build()) {
        server.start();
// The server handles requests...
} // stop() called automatically

handler is an io.vidocq.chappe.api.Handler, a functional interface that receives a request and produces a response. Chappe provides io.vidocq.chappe.api.StaticFileHandler, which serves a folder of static resources, and a builder for io.vidocq.chappe.api.Router, which maps request patterns to handling methods.

StaticFileHandler almost matches our needs, but ignores range requests. So we wrap it in a custom Handler (poc.java.httprange.vidocq.RangeRequestHandler) which:

1. delegates to StaticFileHandler to get the full response;
2. if the request carries a Range header, returns only the requested portion, with a 206 status and a Content-Range header;
3. otherwise, returns the full response as is.

This approach has two limitations. The file is read in full at least once, as StaticFileHandler’s cache only covers small files. More importantly, it fails if the client asks for a compressed response (the most common case): StaticFileHandler compresses the body itself, which breaks the slicing. It is nonetheless good enough for a POC and a nice way to discover the API, in particular io.vidocq.chappe.api.Response.Builder.

Setting up JFRUnit Link to heading

To check that the file is not transferred in full, we measure the bytes read from localhost:8080 (Chappe’s listening port) during the test, by monitoring the jdk.SocketRead event with JFRUnit (setup described in the documentation).

NB: in the current version of JFRUnit, the test class as well as the jfrEvents field must be public, because the library accesses them through reflection.

JFRUnit’s API then makes the computation easy, and its constants for event and attribute names prevent typos:

private long totalBytesReadByHardwoodFromRemoteTitanicParquetFile() {
            // keep only the events related to localhost:8080
    return jfrEvents.filter(SocketRead.INSTANCE.withHost("localhost").withPort(8080))
            // get the amount of data received for each event
            .mapToLong(socketReadEvent -> socketReadEvent.getLong(SocketRead.BYTES_READ.getName()))
            .sum();
}

We can then write an assertion in the test checking that the total amount of data read is positive but less than half the size of the file (it is hard to set a precise threshold without analysing the internal structure of the Parquet file, so we use half the file to make sure the gain is substantial):

assertThat(totalBytesReadByHardwoodFromRemoteTitanicParquetFile())
        .isPositive()
        .isLessThan(Files.size(Path.of(PARQUET_FILE)) / 2);

The full implementation can be found in the CheckBytesTransfered class.

Conclusion Link to heading

In the previous POC, measuring the data exchanged was more complicated: DuckDB relies on a native binary whose traffic escapes the JVM. With Hardwood, we stay within the JVM and can take advantage of JFR in tests, a technique that deserves to be better known. JFR could also be used to compare the performance of different approaches.