An HTTP server to serve GCS files

As many other clouds, Google Cloud Platform provides an Object Storage service, Google Cloud Storage. As many other Object Storage service, Google Cloud Storage provides an HTTP server to deliver your files quickly. When I started to use Google Cloud Storage and its HTTP server I have not been entirely pleased by how it works and therefore I wanted to re-implement the HTTP server so that I can manage it completely.

Being a performance-sensible application, and since I like it, I decided to adopt Go for it.

Since in the environment where I would have deployed it there is an HAProxy as frontend that takes care of HTTPS termination, I didn’t have to implement it at all, just needing to read files from GCS and expose them on HTTP. Also, I needed to expose a /healthz and a /readiness endpoints since it has been designed to work in a Kubernetes cluster. As for file delivery, my goals were:

  • be able to deliver existant files
  • return a 404 if the file does not exist
  • the service has to be able to create a decent throughput but is not required to be completely optimized since it will stay behind a CDN
  • always set the right Content-Type, Content-Encoding, and Content-Length
  • Get the name of the service account key file from the SERVICE_ACCOUNT_KEY environmental variable
  • Get the name of the bucket to use from the BUCKET environmental variable
  • If the LOGGING environmental variable is set to true, log all requests
  • If the DEBUG environmental variable is set to true, use port 10080 (no root power needed) otherwise use port 80 (standard HTTP port)

So, this is what I came out with:

package main

import (
    "context"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "time"

    "cloud.google.com/go/storage"
    "google.golang.org/api/option"
)

func main() {
    log.Println("Starting")

    wwwPort := "80"
    if os.Getenv("DEBUG") == "true" {
        wwwPort = "10080"
    }

    if len(os.Getenv("BUCKET")) == 0 {
        panic("No BUCKET environmental variable is set")
    }
    ctx := context.Background()
    client, err := storage.NewClient(
        ctx,
        option.WithCredentialsFile(os.Getenv("SERVICE_ACCOUNT_KEY")),
    )
    if err != nil {
        panic("Unable to create the client")
    }
    bucket := client.Bucket(os.Getenv("BUCKET"))

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        oh := bucket.Object(r.URL.Path[1:])
        objAttrs, err := oh.Attrs(ctx)
        if err != nil {
            if os.Getenv("LOGGING") == "true" {
                elapsed := time.Since(start)
                log.Println("| 404 |", elapsed.String(), r.Host, r.Method, r.URL.Path)
            }
            http.Error(w, "Not found", 404)
            return
        }
        o := oh.ReadCompressed(true)
        rc, err := o.NewReader(ctx)
        if err != nil {
            http.Error(w, "Not found", 404)
            return
        }
        defer rc.Close()

        w.Header().Set("Content-Type", objAttrs.ContentType)
        w.Header().Set("Content-Encoding", objAttrs.ContentEncoding)
        w.Header().Set("Content-Length", strconv.Itoa(int(objAttrs.Size)))
        w.WriteHeader(200)
        if _, err := io.Copy(w, rc); err != nil {
            if os.Getenv("LOGGING") == "true" {
                elapsed := time.Since(start)
                log.Println("| 200 |", elapsed.String(), r.Host, r.Method, r.URL.Path)
            }
            return
        }
        if os.Getenv("LOGGING") == "true" {
            elapsed := time.Since(start)
            log.Println("| 200 |", elapsed.String(), r.Host, r.Method, r.URL.Path)
        }
    })

    http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("ok"))
    })
    http.HandleFunc("/readiness", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("ok"))
    })

    log.Println("Ready to serve")
    http.ListenAndServe(":"+wwwPort, nil)
}

I’m sure that many additional features can be implemented and that many optimizations can be done as well, but it’s already a working server!

One of the features that I might decide to implement in the future is the ability to deliver index pages if the user asks for a directory and if it contains an index file.

I hope this can help other people to deliver GCS files with a custom HTTP server.