How to Build a Web API with gRPC and Protocol Buffers in Go
Introduction
This article describes gRPC, protocol buffers, and how to use these tools to build a simple web API in Go.
Prerequisites
- Working knowledge of Go.
- Properly installed Go toolchain (Go 1.x installed).
- Knowledge of how APIs work.
gRPC
gRPC (gRPC Remote Procedure Calls) is a modern, open-source, high-performance RPC framework that can run in any environment and uses HTTP 2.0 as the underlying transport protocol. It enables transparent communication between clients and servers and simplifies the building of connected systems.
During server-client communication in gRPC, as in many other RPC systems, a client application can directly invoke a method on another system - as if it was a local call, this makes it easier to build highly efficient distributed services. gRPC clients and servers can run in a variety of environments and be implemented in various supported languages; you can build a gRPC server in Go and implement clients in Ruby, Python, or any one of the other supported languages.
In gRPC, you define a service to specify what methods can be called remotely with the corresponding parameters and return types. On the server-side, this interface is implemented, and a gRPC server is run to handle client calls, while on the client-side, a client that provides access to the same methods as a server - is run.
As we mentioned earlier, gRPC utilizes HTTP 2.0 as the underlying transport protocol. This allows gRPC to get the most out of HTTP 2 advantages, including streaming communication and bidirectional support. Typically, in the request-response communication model built-in HTTP 1.1 used by REST APIs, if a service receives multiple requests from multiple clients - the service has to respond to each request one at a time; this, in turn, slows down the entire system, but in HTTP 2.0's client-response communication model that supports streaming and includes bidirectional support, gRPC can receive multiple requests from multiple clients, and handle those requests simultaneously by constantly streaming information.
gRPC supports the following types of streams:
- Unary streams - a single request-response pair between client and server.
- Server streams - the server responds with a stream of messages to a client's request.
- Client streams - the client sends a stream of messages to the server, and the server, in turn, responds with a single response.
- Bidirectional streams - both streams (client and server) are independent of each other and can both transmit messages in any order. The client is the one who initiates and ends a bidirectional stream.
By default, gRPC utilizes a binary format called Protocol Buffers to serialize data; gRPC offers very high performance and is lightweight.
Protocol Buffers
Protocol Buffers binary format (Protobuf) is an open-source, cross-platform data format used to serialize structured data. You define the structure of the data you want to serialize in a proto file, a normal text file with a .proto extension. Then, use the protocol buffer compiler, protoc - to generate data access classes in your preferred language using your proto definition.
Protocol buffer data is structured as messages, with name-value pairs:
message Student {
string name = 1;
int32 id = 2;
bool active = 3;
}These messages are used as parameters and return types for gRPC services defined in proto files, with RPC methods:
// The Echo service definition
service Echo {
// Sends a greeting back
rpc SayHi (HiRequest) returns (HiResponse) {}
}
// The request message containing the user’s name
message HiRequest {
string name = 1;
}
// The response message containing a greeting
message HiReply {
string message = 1;
}In gRPC, you create a service that has RPC methods; these methods take a request message and return a response message.
gRPC uses protoc with a special plugin to generate code from your proto file; it generates the gRPC server and client code, as well as the regular protocol buffer code for serialiing and retrieving your message types.
Now that we've covered what gRPC and protocol buffers entail, let's get started building our gRPC web API by defining our API resources and installing the required tools.
Identifying our API resources
The first step to building APIs is to define what endpoints will be available via your API. We will be building a simple gRPC web API that creates or reads a user resource from our server; we have the following endpoints:
- API Method FetchUser: Fetch a user
- API Method CreateUser: Creates a user
- API Method UpdateUser: Updates a user
- API Method DeleteUser: Deletes a user
Let's get started with development - by installing the required libraries (gRPC and protocol buffers).
Installing gRPC
To install gRPC, run the following command in your terminal:
$ go get -u google.golang.org/grpcThis will install the gRPC packages to your Go toolchain.
Installing Protocol Buffers
Next, let's install the protobuf compiler:
$ sudo apt install -y protobuf-compilerRunning the above command will install the protocol buffer compiler, which we will be using to generate code from our proto file, later on, using the protoc command.
Creating our project
Let's create our project in our GOPATH:
$ mkdir $GOPATH/src/grpc-apiNow cd into our project directory:
$ cd $GOPATH/src/grpc-apiDefining our proto file
Now create a new folder for our proto files:
$ mkdir protobufLet’s create our proto file:
$ touch protobuf/protobuf.protoOpen our newly created file in your text editor, and add the following lines:
syntax = "proto3";
package protobuf;The first line tells the protocol buffer compiler that we're using proto3 syntax, or else the compiler will assume that we are using proto2. This must be the first line non-empty, non-comment line in a proto file. Then, we specify the package this file belongs to.
Next, we define our message and its field types:
message User {
int32 uid = 1;
string name = 2;
string nationality = 3;
int32 zip = 4;
}This specifies a User message type that takes four fields consisting of two integers (uid and zip), and two strings (name and nationality). You can also use other messages as field types. The number assigned to each field in the message definition is uniquely used to identify the field in the message binary format and should not be changed once the message type is in use.
Next, let’s define our message request/response pair for our FetchUser method:
message FetchUserRequest {
int32 uid = 1;
}
message FetchUserResponse {
User user = 1;
}In our FetchUserRequest message, we accept a user ID (uid) that will be used to find the requested user. We'll return the requested user in a FetchUserResponse message that takes a User message type that we defined earlier as a field.
Then, we define similar request/response message pairs for the rest of our methods:
message CreateUserRequest {
User user = 1;
}
message CreateUserResponse {
User user = 1;
}
message UpdateUserRequest {
User user = 1;
}
message UpdateUserResponse {
User user = 1;
}
message DeleteUserRequest {
int32 uid = 1;
}
message DeleteUserResponse {
int32 uid = 1;
}Above, we define CreateUserRequest and CreateUserResponse, which both take a User message type as fields. So, when a client sends a User in the CreateUserRequest, we send back the created user in a CreateUserResponse message. We also create request/response pairs for the rest of our methods, UpdateUser and DeleteUser.
Now that we’ve defined our message request/response pairs, let’s create our service and register our RPC methods (FetchUser, CreateUser, UpdateUser, and DeleteUser) under the service:
service UserService {
rpc FetchUser(FetchUserRequest) returns (FetchUserResponse);
rpc CreateUser(CreateUserRequest) returns (CreateUserResponse);
rpc UpdateUser(UpdateUserRequest) returns (UpdateUserResponse);
rpc DeleteUser(DeleteUserRequest) returns (DeleteUserResponse);
}Here, we defined a service called UserService, and registered our RPC methods with their parameters and return values, e.g. the FetchUser method accepts a FetchUserRequest as a parameter and returns a FetchUserResponse, etc.
Now our protobuf.proto file should look like this:
syntax = "proto3";
package protobuf;
message User {
int32 uid = 1;
string name = 2;
string nationality = 3;
int32 zip = 4;
}
message FetchUserRequest {
int32 uid = 1;
}
message FetchUserResponse {
User user = 1;
}
message CreateUserRequest {
User user = 1;
}
message CreateUserResponse {
User user = 1;
}
message UpdateUserRequest {
User user = 1;
}
message UpdateUserResponse {
User user = 1;
}
message DeleteUserRequest {
int32 uid = 1;
}
message DeleteUserResponse {
int32 uid = 1;
}
service UserService {
rpc FetchUser(FetchUserRequest) returns (FetchUserResponse);
rpc CreateUser(CreateUserRequest) returns (CreateUserResponse);
rpc UpdateUser(UpdateUserRequest) returns (UpdateUserResponse);
rpc DeleteUser(DeleteUserRequest) returns (DeleteUserResponse);
}To compile our proto file with the protocol buffer compiler and generate the corresponding Go code to use, simply run the command from our project directory:
$ protoc --gogo_out=plugins=grpc:. protobuf/protobuf.protoRunning this command will generate a .pb.go file in our protobuf directory, which contains the equivalent Go bindings from our proto file which implements the gRPC code our application will use, along with server and client methods.
Creating our Server
We can build our server after generating Go code from our protocol buffer compiler (using protoc).
Let's create our server.go file:
$ touch server.goOpen the file in your favorite text editor, and add the following lines:
package main
import (
"context"
"fmt"
"log"
"net"
"errors"
"time"
"math"
"google.golang.org/grpc"
"google.golang.org/grpc/keepalive"
"grpc-api/protobuf"
)We've just imported the required libraries, including our protobuf package, so that we can access the methods and structs generated by protoc.
The protocol buffer compiler generated a UserServiceServer interface for our UserService, which implements the RPC methods that we defined.
Before we go on to define our server, let's create a struct that we can use as our database:
type userDetails struct {
Uid int32
Name string
Nationality string
Zip int32
}
var users = []userDetails{
{
Uid: 1,
Name: "Josh Winters",
Nationality: "American",
Zip: 10111,
},
{
Uid: 2,
Name: "Brian Stone",
Nationality: "British",
Zip: 20212,
},
}We've created a userDetails struct with identical fields to the User type defined in our proto file; we will be using a slice of userDetails as our database for reading, updating, creating, and deleting users. Let's create helper functions to help convert between User and userDetails type:
func toUser(data userDetails) *protobuf.User {
return &protobuf.User {
Uid: data.Uid,
Name: data.Name,
Nationality: data.Nationality,
Zip: data.Zip,
}
}
func fromUser(user *protobuf.User) userDetails {
return userDetails {
Uid: user.GetUid(),
Name: user.GetName(),
Nationality: user.GetNationality(),
Zip: user.GetZip(),
}
}Now we can convert back and forth from User to userDetails easily, with our helper functions; we use the Get<fieldname> method to extract the values from our User type.
Next, we define a server struct that will satisfy the UserServiceServer interface generated by protoc:
type server struct {
protobuf.UserServiceServer
}Now, let’s define our methods for the server:
FetchUser() method
func (s *server) FetchUser(ctx context.Context, req *protobuf.FetchUserRequest) (*protobuf.FetchUserResponse, error) {
fmt.Println("Fetching User")
uid := req.GetUid()
for _, user := range users {
if user.Uid == uid {
return &protobuf.FetchUserResponse {
User: toUser(user),
}, nil
}
}
return nil, errors.New("User not found")
}The generated FetchUser() method accepts two arguments; a context and a FetchUserRequest and returns a FetchUserResponse and an error - so to implement it we pass it a context and FetchUserRequest as arguments.
In our implementation, we print a "Fetching User" message to the screen and use the GetUid() method generated by protoc in the .pb.go file to get the Uid from the FetchUserRequest. Then, we iterate through our list of users and return the user if found else we return nil and an error.
CreateUser() method
func (s *server) CreateUser(ctx context.Context, req *protobuf.CreateUserRequest) (*protobuf.CreateUserResponse, error) {
fmt.Println("Creating User")
user := req.GetUser()
data := fromUser(user)
users = append(users, data)
return &protobuf.CreateUserResponse {
User: toUser(data),
}, nil
}Here, we read the user from the CreateUserRequest using the GetUser() method generated by the compiler - to get the User from the request; next, we convert the user to a userDetails type using our fromUser helper function, then append the user to our list of users and send back the created user as a response.
UpdateUser() method
func (s *server) UpdateUser(ctx context.Context, req *protobuf.UpdateUserRequest) (*protobuf.UpdateUserResponse, error) {
fmt.Println("Updating User")
user := req.GetUser()
data := fromUser(user)
for i, user := range users {
if user.Uid == data.Uid {
users[i] = data
return &protobuf.UpdateUserResponse {
User: toUser(data),
}, nil
}
}
return nil, errors.New("Couldn't update user")
}Next, we implement our UpdateUser method; first, we get the user from the UpdateUserRequest by using the GetUser() method, and iterate through our list of users searching for the corresponding Uid, if found - we update the user, and return the updated user as a response, else we return nil and an error.
DeleteUser() method
func (s *server) DeleteUser(ctx context.Context, req *protobuf.DeleteUserRequest) (*protobuf.DeleteUserResponse, error) {
fmt.Println("Deleting User")
uid := req.GetUid()
var tmpUsers []userDetails
for i, user := range users {
if user.Uid == uid {
tmpUsers = append(users[:i], users[i+1:]...)
users = tmpUsers
fmt.Printf("User with id %d has been deleted.\n", uid)
return &protobuf.DeleteUserResponse {
Uid: uid,
}, nil
}
}
return nil, errors.New("User does not exist")
}We implement the DeleteUser() method, by fetching the Uid() from the request, and iterating through our list of users, looking for the matching Uid; if found, we delete the user from the list and return the deleted Uid, else we return nil and error if the corresponding user with the Uid was not found.
Creating our main function
Now that we have implemented our server struct which satisfies the interface by implementing all our methods, we can proceed with creating the main code to host the server:
func main() {
lis, err := net.Listen("tcp", "127.0.0.1:5000")
if err != nil {
log.Fatalf("Failed to listen: %v", err)
}
opts := []grpc.ServerOption{
grpc.MaxRecvMsgSize(math.MaxInt64),
grpc.KeepaliveParams(
keepalive.ServerParameters{
Timeout: 5 * time.Second,
},
),
}
s := grpc.NewServer(opts...)
protobuf.RegisterUserServiceServer(s, &server{})
fmt.Println("Starting server...")
fmt.Printf("Hosting server on: %s\n", lis.Addr().String())
if err := s.Serve(lis); err != nil {
log.Fatalf("Failed to serve: %v", err)
}
}First, we create a listener on 127.0.0.1:5000 using:
lis, err := net.Listen(tcp”, 127.0.0.1:5000”)We handle any errors creating the listener by calling log.Fatalf, which logs the error and calls os.Exit().
Using []grpc.ServerOptions, we create a slice of options for our server and set the maximum receive message size and server timeout.
We then create our gRPC server and pass our slice of ServerOptions as a parameter:
s := grpc.NewServer(opts…)After creating our gRPC server, we have to register it for our service:
protobuf.RegisterUserServiceServer(s, &server{})The RegisterUserServiceServer was generated by protoc, and it takes two arguments - a grpc.NewServer() instance, and a server struct that implements all the RPC methods added to our service; we then pass our implemented server struct as the second argument. This binds our gRPC server to our UserService.
Afterward, we host the server by calling s.Serve() with our listener as a parameter, and add proper error handling.
Putting it all together, our server.go script should look like this:
package main
import (
"context"
"fmt"
"log"
"net"
"errors"
"math"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/keepalive"
"grpc-api/protobuf"
)
type userDetails struct {
Uid int32
Name string
Nationality string
Zip int32
}
var users = []userDetails{
{
Uid: 1,
Name: "Josh Winters",
Nationality: "American",
Zip: 10111,
},
{
Uid: 2,
Name: "Brian Stone",
Nationality: "British",
Zip: 20212,
},
}
func toUser(data userDetails) *protobuf.User {
return &protobuf.User {
Uid: data.Uid,
Name: data.Name,
Nationality: data.Nationality,
Zip: data.Zip,
}
}
func fromUser(user *protobuf.User) userDetails {
return userDetails {
Uid: user.GetUid(),
Name: user.GetName(),
Nationality: user.GetNationality(),
Zip: user.GetZip(),
}
}
type server struct {
protobuf.UserServiceServer
}
func (s *server) FetchUser(ctx context.Context, req *protobuf.FetchUserRequest) (*protobuf.FetchUserResponse, error) {
fmt.Println("Fetching User")
uid := req.GetUid()
for _, user := range users {
if user.Uid == uid {
return &protobuf.FetchUserResponse {
User: toUser(user),
}, nil
}
}
return nil, errors.New("User not found")
}
func (s *server) CreateUser(ctx context.Context, req *protobuf.CreateUserRequest) (*protobuf.CreateUserResponse, error) {
fmt.Println("Creating User")
user := req.GetUser()
data := fromUser(user)
users = append(users, data)
return &protobuf.CreateUserResponse {
User: toUser(data),
}, nil
}
func (s *server) UpdateUser(ctx context.Context, req *protobuf.UpdateUserRequest) (*protobuf.UpdateUserResponse, error) {
fmt.Println("Updating User")
user := req.GetUser()
data := fromUser(user)
for i, user := range users {
if user.Uid == data.Uid {
users[i] = data
return &protobuf.UpdateUserResponse {
User: toUser(data),
}, nil
}
}
return nil, errors.New("Couldn't update user")
}
func (s *server) DeleteUser(ctx context.Context, req *protobuf.DeleteUserRequest) (*protobuf.DeleteUserResponse, error) {
fmt.Println("Deleting User")
uid := req.GetUid()
var tmpUsers []userDetails
for i, user := range users {
if user.Uid == uid {
tmpUsers = append(users[:i], users[i+1:]...)
users = tmpUsers
fmt.Printf("User with id %d has been deleted.\n", uid)
return &protobuf.DeleteUserResponse {
Uid: uid,
}, nil
}
}
return nil, errors.New("User does not exist")
}
func main() {
lis, err := net.Listen("tcp", "127.0.0.1:5000")
if err != nil {
log.Fatalf("Failed to listen: %v", err)
}
opts := []grpc.ServerOption{
grpc.MaxRecvMsgSize(math.MaxInt64),
grpc.KeepaliveParams(
keepalive.ServerParameters{
Timeout: 5 * time.Second,
},
),
}
s := grpc.NewServer(opts...)
protobuf.RegisterUserServiceServer(s, &server{})
fmt.Println("Starting server...")
fmt.Printf("Hosting server on: %s\n", lis.Addr().String())
if err := s.Serve(lis); err != nil {
log.Fatalf("Failed to serve: %v", err)
}
}Now that we've built our server, let's build our client.
Creating the Client
In an RPC system, client calls are executed on the server like local calls; so the client is simply going to call our methods, and the server will process them.
Let's create our client.go file:
$ touch client.goNow, open up our newly created file in your favorite text editor, and add the following lines:
package main
import (
"context"
"fmt"
"log"
"google.golang.org/grpc"
"grpc-api/protobuf"
)Now we’ve imported the required packages, let’s create our main function:
func main() {
fmt.Println("Starting Client\n")
cc, err := grpc.Dial("127.0.0.1:5000", grpc.WithInsecure())
if err != nil {
log.Fatalf("Error connecting: %v", err)
}
// Close connection before exiting application
defer cc.Close()
c := protobuf.NewUserServiceClient(cc)First, we create a connection to our server address at "127.0.0.1:5000" using grpc.Dial(). grpc.Dial() takes transport security as a second parameter. Since we're connecting on the same host machine, we don't need any layer of additional security set, so we simply use grpc.WithInsecure(). Then we perform error handling and defer closing the connection.
Next, we register our connection as a client for our service, using:
c := protobuf.NewUserServiceClient(cc)To test our CreateUser RPC method, we simply create a new User message, and call the corresponding CreateUser() method:
Calling CreateUser()
// Create a user
fmt.Println("Calling CreateUser()")
user := &protobuf.User {
Uid: 3,
Name: "Sarah Connors",
Nationality: "Canadian",
Zip: 45015,
}
createUserResponse, err := c.CreateUser(context.Background(), protobuf.CreateUserRequest{User: user})
if err != nil {
log.Fatalf("An error occurred: %v", err)
}
fmt.Printf("User has been created: %v\n\n", createUserResponse)We create a new protobuf.User object, and call the CreateUser() method with a context.Background() and our newly created User as a CreateUserRequest.
Remember that our CreateUser method takes a CreateUserRequest as an argument, so we pass our declared user as a value for the User field in the CreateUserRequest body.
Then we get back a CreateUserResponse, and print it to the screen.
Now, let's call our other methods to make sure they are all working as expected:
Calling UpdateUser()
// Update a user
updateUser := &protobuf.User {
Uid: 1,
Name: "Mandy Williams",
Nationality: "American",
Zip: 10111,
}
fmt.Println("Updating a user")
updateUserResponse, err := c.UpdateUser(context.Background(), &protobuf.UpdateUserRequest{User: updateUser})
if err != nil {
log.Fatalf("Error occurred updating user: %v", err)
}
fmt.Printf("A user has been updated: %v\n\n", updateUserResponse)Above, we framed a User message with the changes that we want made to that Uid, and called the UpdateUser method. Now the other fields of the User with Uid 1, will be changed on our server.
Calling DeleteUser()
fmt.Println("Deleting user")
var delUid int32 = 2
deleteUserResponse, err := c.DeleteUser(context.Background(), &protobuf.DeleteUserRequest{Uid: delUid})
if err != nil {
log.Fatalf("Error occurred deleting user: %v", err)
}
fmt.Printf("User with ID: %d has been deleted.\n\n", deleteUserResponse.GetUid())Here we called DeleteUser() on the User with a Uid of 2.
Calling FetchUser()
// Get a user
fmt.Println("Fetching a user")
var getUid int32 = 1
fetchUserResponse, err := c.FetchUser(context.Background(), &protobuf.FetchUserRequest{Uid: getUid})
if err != nil {
log.Fatalf("User not found error: %v", getUid)
}
fmt.Printf("User: %v\n", fetchUserResponse)
}Now, to make sure our update worked, we will call FetchUser on the user with a Uid of 1, to see if the details were changed successfully to our framed request earlier.
Our client.go file should now look like this:
package main
import (
"context"
"fmt"
"log"
"google.golang.org/grpc"
"grpc-api/protobuf"
)
func main() {
fmt.Println("Starting Client\n")
cc, err := grpc.Dial("localhost:5000", grpc.WithInsecure())
if err != nil {
log.Fatalf("Error connecting: %v", err)
}
// Close connection before exiting app
defer cc.Close()
c := protobuf.NewUserServiceClient(cc)
// Create a user
fmt.Println("Calling CreateUser()")
user := &protobuf.User {
Uid: 3,
Name: "Sarah Connors",
Nationality: "Canadian",
Zip: 45015,
}
createUserResponse, err := c.CreateUser(context.Background(), &protobuf.CreateUserRequest{User: user})
if err != nil {
log.Fatalf("An error occurred: %v", err)
}
fmt.Printf("User has been created: %v\n\n", createUserResponse)
// userID := createUserResponse.GetUser().GetId()
// Update a user
updateUser := &protobuf.User {
Uid: 1,
Name: "Mandy Williams",
Nationality: "American",
Zip: 10111,
}
fmt.Println("Updating a user")
updateUserResponse, err := c.UpdateUser(context.Background(), &protobuf.UpdateUserRequest{User: updateUser})
if err != nil {
log.Fatalf("Error occurred updating user: %v", err)
}
fmt.Printf("A user has been updated: %v\n\n", updateUserResponse)
fmt.Println("Deleting user")
var delUid int32 = 2
deleteUserResponse, err := c.DeleteUser(context.Background(), &protobuf.DeleteUserRequest{Uid: delUid})
if err != nil {
log.Fatalf("Error occurred deleting user: %v", err)
}
fmt.Printf("User with ID: %d has been deleted.\n\n", deleteUserResponse.GetUid())
// Get a user
fmt.Println("Fetching a user")
var getUid int32 = 1
fetchUserResponse, err := c.FetchUser(context.Background(), &protobuf.FetchUserRequest{Uid: getUid})
if err != nil {
log.Fatalf("User not found error: %v", getUid)
}
fmt.Printf("User: %v\n", fetchUserResponse)
}Running the Server and Client
We have successfully built our gRPC server and client, now to test out both of them.
Open up a terminal in our project directory and run the server first:
$ go run server.goWe'll get back our printed message:
Starting server…
Hosting server on 127.0.0.1:5000Now, open another terminal in our project directory and run the client:
$ go run client.goYou should get a response on the client side like this:
Starting Client
Calling CreateUser()
User has been created: user:<uid:3 name:"Sarah Connors" nationality:"Canadian" zip:45015>
Updating a user
A user has been updated: user:<uid:1 name:"Mandy Williams" nationality:"American" zip:10111>
Deleting user
User with ID: 2 has been deleted.
Fetching a user
User: user:<uid:1 name:"Mandy Williams" nationality:"American" zip:10111>From the output, we can see that our client ran successfully and all RPC calls worked as expected.
While on the server side:
Starting server...
Hosting server on: 127.0.0.1:5000
Create User
Updating User
Deleting User
User with id 2 has been deleted.
Fetching UserWe can see that all print calls inside our RPC methods implementation were made on the server-side, which shows all methods were executed on the server through the calls, and responses received on the client-side.
We have successfully built a CRUD web API using gRPC and protocol buffers.
Securing Communications
In gRPC, the client and server talk over HTTP/2 with binary data (i.e. Protocol buffers); but gRPC also offers SSL/TLS integration which can be used to authenticate the server and encrypt the message exchange.
Conclusion
In this article, we have learned how to build web APIs utilizing gRPC and protocol buffers in Go.