Rest APIs Server in Go and Redis

Simple REST API service for an imaginary blog with Redis as a mock database.

For starters, clone the project here:

$ git clone https://github.com/jochasinga/go-rest -b blog

Then run the project within the directory with:

$ go build 
$ ./go-rest

Browse to:

http://localhost:8080 to see the welcome message
http://localhost:8080/posts to see all post datas (seeded posts)
http://localhost:8080/posts/1 to see the post with id=1

Send POST request to add more post with:

$ curl -H "Content-Type: application/json" -d '{
        "topic": "Hello from Mars",
        "text": "I am anonymous hehe. Catch me if you can." 
        }' http://localhost:8080/posts

Try browsing steps again to see your new post.

REST Assured

REST (Representational State Transfer) is a uniform way of locating resources on the internet that goes beyond the boundaries of programming languages. On the internet, whether from a web or mobile app, you almost always have to request a resource using HTTP methods like GET and POST. For instance, a specific PDF file (a resource) located at http://somedomain/files/pdfs/id=99 is always available at that location for anyone interfacing with the url in any language.

A web service is a function provided at a network address on the web designed to expose a set of APIs (Application Program Interfaces) for other services to consume. The consumers can range from a web app, mobile app to another service, while the service can range from something as simple as a calculator, today’s temparature to the Google Map APIs. Web service’s resources can be a static binary file as a PDF or dynamic functions and data.

A very universal data format which goes hand-in-hand with REST APIs is JSON (Javascript Object Notation). It has become the de facto of data exchange on the internet (though there are other players like XML and Google’s Protobuf). In this writing, we’ll be creating a simple JSON-based RESTful APIs web service in Go for a simple blog with Redis as a mock database.

What we will build

We will create a simple blog service that writes JSON data as strings to the browser on different url routes. For instance, browsing to http://localhost/will display a welcome message, http://localhost/posts will display all posts in JSON and http://localhost/posts/id will get you the post with the given id.

The Data Models

It is always wise to think a lot before burning the keyboard away. We should first think very hard about what a “concept” of a blog post should look like.

// A Post should contain...
Id            int
User          User
Topic         string
Text          string
Comment       Comment
Timestamp     Time

You might notice that a Comment isn’t just a string. Well, think harder. A Comment itself is another concept with a few fields of its own, some of which are similar to Post.

// A Comment should contain...
Id             int
User           User
Text           string
Timestamp      Time

And so is a User:

// A User should contain...
Id             int
Username       string
Email          string

We’ll leave things like profile photos and passwords out for simplicity’s sake.

With all these set in stones, let’s write these down as Go structs.

// User 
type User struct {
        Id        int
        Username  string       
        Email     string
}
// Comment
type Comment struct {
        Id        int
        User      User 
        Text      string
        Timestamp time.Time   // Go's Time type
}
// Post
type Post struct {
        Id        int
        User      User
        Topic     string
        Text      string
        Comment   Comment
        Timestamp time.Time
}

Make sense? We now have a working data model for our blog. However, we can’t expect a blog with only one post, comment or user. We need a slice for each data structure.

type Posts    []Post
type Comments []Comment
type Users    []User

Let’s make this into a file and name it models.go

The struct tags (within the backticks “) are used to make all fields lowercase when being encoded to JSON.

Endpoints

Next, let’s think of the way to design locations for our data. In RESTful speaks, let’s talk url routes or endpoints. What paths you want consumers to take and what data you want to place at each end.

// Say hello with a stunning home page 
"/"
// GET all Posts here
"/posts"
// POST new Post here
"/posts"
// GET Post(s) with a specific Id
"/posts/id"

Of course, you can think of so many ways to expose your APIs like exposing a User with a specific Id for a profile page. But I’ll leave that to your imagination.

In Go, when you spend more than five minutes thinking about a concept really hard, you should make it a struct.

type Route struct {
        Method string
        Path   string
        Handle mux.Handle   // httprouter package as mux
}

The Method is a HTTP method like “GET” or “POST” (see CRUD if you are unfamiliar with these), while the Path is the url endpoint you designed earlier. The Handle field is the handler function which dispatches the right HTML page and/or data for each route, known as a controller function in some frameworks. As you’ll see I’ll make use of httprouter package and import it as mux (for multiplexer). You can also read the doc.

Install the package:

$ go get github.com/julienschmidt/httprouter
$ go install github.com/julienschmidt/httprouter

Again, there will probably be a lot of routes. A slice would be great.

type Routes []Route

Now let’s create some Route instances for the slice and name this file routes.go.

From here, it’s time the lay out the handler function for each route. For the Index function, we want to display a heading “Hello, welcome to my blog”.

func Index(w http.ResponseWriter, r *http.Request, _ mux.Params) {
        fmt.Fprintf(w, "<h1>Hello, welcome to my blog</h1>")
}

This function does nothing more than writing the HTML string to the browser (ResponseWriter) i.e. the heading.

If you have read the doc on type httprouter.Handle, you’ll see that it’s a function that can be registered to a route to handle HTTP Request like http.HandlerFunc, but with an additional third parameter to capture the params from the url.

For the PostIndex function, we want to display every posts in the database.

func PostIndex(w http.ResponseWriter, r *http.Request, 
               _ mux.Params) {
        var posts Posts
        posts = FindAll()  // We'll work on this
        if err := json.NewEncoder(w).Encode(posts); err != nil {
                panic(err)
        }
}

What the function does is simply encoding a slice of Post instances in the variable posts read from the database to JSON arrays and write to the ResponseWriter to the browser. We have not yet worked on the FindAll() function, but it is nice think of a functionality of a function before creating one. In this case, FindAll() will return a slice of Post instances from the database.

The PostShow function is a little tricky, since it retrieves only the post with the Id specified in the url parameter.

func PostShow(w http.ResponseWriter, r *http.Request, p mux.Params) {
        id, err := strconv.Atoi(p.ByName("postId"))
        if err != nil {
                panic(err)
        }
        post := FindPost(id)   // We'll work on this
        
        if err := json.NewEncoder(w).Encode(post); err != nil {
                panic(err)
        }
}

We use strconv.Atoi (ASCII to integer) method to convert the id parameter retrieved from the url to an integer and store it in the id variable. Next, we retrieve the post with a FindPost() function, which is another conceptual function. Lastly, we encode the data to JSON and write it to the response.

Since we have to deal with error handling very regularly in Go, let’s make it a function to make it more DRY.

func HandleError(err error) {
        if err != nil {
                panic(err)
        }
}

You can place it anywhere, but I’ll be placing it in the main.go file.

For PostCreate, we will have to deal with reading the request body, limiting the buffer size for security reason, unmarshalling the posted JSON to Todo data structure and save to the database.

func PostCreate(w ResponseWriter, r *http.Request, _ mux.Params) {
        var post Post
       
        // Read request body and close it
        body, err := ioutil.ReadAll(io.LimitReader(r.Body,1048576))
        HandleError(err)
        defer r.Body.Close()
        // Save JSON to Post struct
        if err := json.Unmarshal(body, &todo); err != nil {
             if err := json.NewEncoder(w).Encode(err); err != nil {
                     panic(err)
             }
        }
        CreatePost(post)   // We'll work on this
}

We’ll be using cURL to send requests to the service, thus controller function will read the request body sent from cURL. Note another non-existing function CreatePost(). This function is suppose to save our newly created Post to the database.

This sums up the handlers.go.

You might see a lot of Header setting and writing status. It’s considered responsible to do so, although Go can guess that for you most of the time.

Routers

With routes and handler functions out of the way, let’s “activate” those routes and connect them to appropriate controllers.

The code creates a new router instance and iterate through all the Routes to get each’s Route’s Method, Pattern and Handle and registers a new request handle with the given path and method. See the doc.

Database

A service can’t be one without a data storage. To simplify the matter, I decided to use Redis as a mock database. Redis itself is a key-value store and is capable of being a NoSQL database in its own right, but it is more suitable as a data cache between your server code and the database of your choice.

Start by installing Redis by downloading the binary from its homepage, or better yet via your OS’s package manager. I used Homebrew here:

$ brew update
$ brew install redis

To launch a Redis server, use the following command:

$ redis-server /usr/local/etc/redis.conf

The server starts listen at port 6379 as a default.

Optional: Redis Server is a Service
You can skip this paragraph, but just for those who are curious about yet another server — Redis is a service for data storage. It doesn’t offer RESTful APIs like what we’re doing here, but a set of commands pretty close, like SETfor creating and saving and GET for reading data. Our blog service is also consuming data from Redis database service via its APIs. ANALOGY: You might be running a bakery shop but you may source ingredients from other shops, and mixing those ingredients into delicious, meaningful bites is your service. The concept of web services isn’t that different.

Once you’ve started a Redis server, it starts running and ready to accept connections. You can also use Redis’ commandline interface that comes with the installation to inspect your stored data with Redis’ APIs. Open another tab on your terminal and type

$ redis-cli

With the commandline ready, try typing

> KEYS *

This command returns the keys that match anything. We will be saving each Post as JSON string (or more specifically, []byte) with a key post:id format. At this point, you should get a nil meaning there’s nothing stored just yet.

Database Adapters

This part is when all the mysterious functions unfold. We will work on the functions interacting most closely to the database, creating and reading from it directly. In many languages’ framework, this is often a part of the model in the MVC. For us, we can think of models.go and this code db.go as model code.

Before going into the details, you’ll need to install one more package. Redigois an Redis adapter for Go. Install it with go get

$ go get github.com/garyburd/redigo
$ go install github.com/garyburd/redigo

And include the package

import "github.com/garyburd/redigo/redis"

Also, declare two global variables for the each Post’s and User’s id:

var currentPostId int
var currentUserId int

We will be connecting several times to the Redis server, so let’s write a function to wrap that up

func RedisConnect() redis.Conn {
        c, err := redis.Dial("tcp", ":6379")
        HandleError(err)
        return c
}

Now let’s start by filling up the database by creating two Posts on initiation.

func init() {
        CreatePost(Post{
                User: User{
                        Username: "pieOhpah",
                        Email: "[email protected]",
                },
                Topic: "My First Post",
                Text: "Hello everyone! This is awesome.",
        })
        CreatePost(Post{
                User: User{
                        Username: "IronMan",
                        Email: "[email protected]",
                },
                Topic: "My Fight with Thor Today",
                Text: "This is the third time I beat Thor up.",
        })
}

In Go, function named init() will be run automatically upon running. It has a lowercase name since there’s no need for other code to call it from another package. Also note that we did not have to specify all the fields for each Post on creation. The missing ones will be set to zero value (o for int, “” for string, nil for pointer, etc.), Go-style.

Now let’s dive into CreatePost() for creating a Post instance and saving it as JSON to Redis. This function will be called by handler PostCreate.

func CreatePost(p Post) {
        // Increment Ids for each Post created 
        currentPostId += 1
        currentUserId += 1
        // Set Post's and User's Id
        p.Id = currentPostId
        p.User.Id = currentUserId
        // Set Timestamp to now
        p.Timestamp = time.Now()
        c := RedisConnect()
        defer c.Close()
        // Marshal Post to JSON blob
        b, err := json.Marshal(p)
        // Save JSON to Redis
        reply, err := c.Do("SET", "post:" + strconv.Itoa(p.Id), b)
        HandleError(err)
        fmt.Println("GET ", reply)
}

The key to this function is saving JSON string marshaled from a Post to Redis. SET sets the data to a given key. In this case, each JSON string is assigned to a key in the format post:id. This will, in Redis, look something similar to a map in Go:

map[string]string{
        "post:1" : "JSON representation of Post with Id 1",
        "post:2" : "JSON represention of Post with Id 2",
        ...,
}

Now let’s work on FindAll() to read all the posts from the database. It will be called by the handler PostIndex.

func FindAll() Posts {
        var posts Posts
        c := RedisConnect()
        defer c.Close()
        // Get all the keys matching the glob
        keys, err := c.Do("KEYS", "post:*")
        HandleError(err)
        // Iterate through all keys
        for _, k := range keys.([]interface{}) {
            
                var post Post
                // Get the data at each key
                reply, err := c.Do("GET", k.([]byte))   
                HandleError(err)
                // Marshal JSON data to Post
                if err := json.Unmarshal(reply.([]byte), &post); err != nil {
                        panic(err)
                }
                // Append each to posts slice
                posts = append(posts, post)
         }
         return posts
}

There’s a little thing at the iteration phase — we had to do type assertion for the list of keys returned from the KEYS command because Redigo returns almost everything as type interface{} (generic in other tongues). Everything else is straightforward.

Here is a FindPost() to find just a specific post for a given id:

func FindPost(id int) Post {
        var post Post
        
        c := RedisConnect()
        defer c.Close()
        reply, err := c.Do("GET", "post:" + strconv.Itoa(id))
        HandleError(err)
        if err = json.Unmarshal(reply.([]byte), &post); err != nil {
                panic(err)
        }
        return post
}

This function will be called by the handler PostShow.

Save it as a single file db.go.

Main

There is little for the main.go.

We create a new *router instance, and set the HTTP server up to listen on port 8080. The HandleError() is just a helper function we have been using throughout the code to deal with error checking.

Running the server

At this point, we should have the following files inside a project directory

blog/
├── db.go
├── handlers.go
├── main.go
├── models.go
├── router.go
└── routes.go

Make sure you already have Redis running on port 6379. Build and run the project.

$ cd blog
$ go build 
$ ./blog

Try browsing to localhost:8080 to see the welcome message. append “/posts” to see all posts in JSON (hopefully your browser supports viewing JSON in pretty format), and append an id as “/posts/2″ to read only that post. Now we have only two posts created by our init() function.

To try creating new post, use cURL to send POST request to the endpoint localhost:8080/posts:

$ curl -H "Content-Type: application/json" -d '{
        "topic": "Hello from Mars",
        "text": "I am anonymous hehe. Catch me if you can." 
        }' http://localhost:8080/posts

Refresh the browser to see the new post, or try on the redis-cli:

> KEYS post:*   // Find all keys matching the glob
> GET post:3    // Get the JSON string stored at the key

Improvements

This was written very quickly, so there is a lot of repetitive code and rooms for tons of improvement (i.e. Replacing Redis with a real database and use Redis or Go’s maps to work as a cache instead). Please feel free to write comments on how this can be improved. Thanks!