Avatar (Fabio Alessandro Locati|Fale)'s blog

An HTTP server to serve GCS files

April 12, 2018

As many other clouds, Google Cloud Platform provides an Object Storage service, Google Cloud Storage. As many other Object Storage service, Google Cloud Storage provides an HTTP server to deliver your files quickly. When I started to use Google Cloud Storage and its HTTP server I have not been entirely pleased by how it works and therefore I wanted to re-implement the HTTP server so that I can manage it completely.

Being a performance-sensible application, and since I like it, I decided to adopt Go for it.

Since in the environment where I would have deployed it there is an HAProxy as frontend that takes care of HTTPS termination, I didn’t have to implement it at all, just needing to read files from GCS and expose them on HTTP. Also, I needed to expose a /healthz and a /readiness endpoints since it has been designed to work in a Kubernetes cluster. As for file delivery, my goals were:

So, this is what I came out with:

package main

import (
    "context"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "time"

    "cloud.google.com/go/storage"
    "google.golang.org/api/option"
)

func main() {
    log.Println("Starting")

    wwwPort := "80"
    if os.Getenv("DEBUG") == "true" {
        wwwPort = "10080"
    }

    if len(os.Getenv("BUCKET")) == 0 {
        panic("No BUCKET environmental variable is set")
    }
    ctx := context.Background()
    client, err := storage.NewClient(
        ctx,
        option.WithCredentialsFile(os.Getenv("SERVICE_ACCOUNT_KEY")),
    )
    if err != nil {
        panic("Unable to create the client")
    }
    bucket := client.Bucket(os.Getenv("BUCKET"))

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        oh := bucket.Object(r.URL.Path[1:])
        objAttrs, err := oh.Attrs(ctx)
        if err != nil {
            if os.Getenv("LOGGING") == "true" {
                elapsed := time.Since(start)
                log.Println("| 404 |", elapsed.String(), r.Host, r.Method, r.URL.Path)
            }
            http.Error(w, "Not found", 404)
            return
        }
        o := oh.ReadCompressed(true)
        rc, err := o.NewReader(ctx)
        if err != nil {
            http.Error(w, "Not found", 404)
            return
        }
        defer rc.Close()

        w.Header().Set("Content-Type", objAttrs.ContentType)
        w.Header().Set("Content-Encoding", objAttrs.ContentEncoding)
        w.Header().Set("Content-Length", strconv.Itoa(int(objAttrs.Size)))
        w.WriteHeader(200)
        if _, err := io.Copy(w, rc); err != nil {
            if os.Getenv("LOGGING") == "true" {
                elapsed := time.Since(start)
                log.Println("| 200 |", elapsed.String(), r.Host, r.Method, r.URL.Path)
            }
            return
        }
        if os.Getenv("LOGGING") == "true" {
            elapsed := time.Since(start)
            log.Println("| 200 |", elapsed.String(), r.Host, r.Method, r.URL.Path)
        }
    })

    http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("ok"))
    })
    http.HandleFunc("/readiness", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("ok"))
    })

    log.Println("Ready to serve")
    http.ListenAndServe(":"+wwwPort, nil)
}

I’m sure that many additional features can be implemented and that many optimizations can be done as well, but it’s already a working server!

One of the features that I might decide to implement in the future is the ability to deliver index pages if the user asks for a directory and if it contains an index file.

I hope this can help other people to deliver GCS files with a custom HTTP server.

Cloud, GCP, Programming
GoLang, Google Cloud Storage