Automated Microservices Management with Open Source Packman

Matan Golan
December 1st, 2020

The world of micro-services changed the way developers think, code and manage their workspaces, suddenly new services popup on a weekly basis. When I started working at SecureNative the roadmap was pretty clear, we’ll need to manage thousands of different micro-services, sprint after sprint I was doing the same job for every new micro-service. So I started looking for tools to automate this tedious process, some had a steep learning curve, some didn’t work for us (feature-wise) and some were very opinionated.

So we did what developers does best, write a piece of code to automate it. Two weeks later packman was born, packman is a cli-tool built with extensibility and simpleness in mind, we use it intensively at SecureNative for the past 8 months and we have a wide range of templates that solves different problem.

In this article I’ll try to explain to you what is packman and even more important how it works… we’ll do it Tarantino-style, breaking it into unrelated parts hoping that you get a closure at the end of this article :)

Part 1: The Template Engine

Apparently GO has a pretty great template engine in its standard library, it’s so simple and powerful that I better start with examples of how to use it:

package example

import (
  "text/template"
)

func template(text string, data map[string]interface{}) string { 
  // Get a new instance of the template engine
  t := template.New("template")

  // parse the template text:
  tree, _ := t.Parse(text)

  // Execute the template engine on the provided data
  var out bytes.Buffer
  tree.Execute(&out, data)

  return out.String()
}

This function will template the input text for a given data map, the map acts as the template data and we can use this data to create smarter templates.

In this article I’ll cover only three of GO’s template engine features but knowing them should give you a great starting point.

Feature 1 — Variable Substitution:

The most elementary part of any template engine is to substitute variables, for example, for the following template file:

Hello {{{ .Name }}},
This is a template which prints your name, hurray!
and the following data:
{
  "Name": "Matan Golan"
}

the template engine will return:

Hello Matan Golan,
This is a template which prints your name, hurray!

Easy stuff right? wait it’s getting interesting…

Feature 2 — Condition Blocks:

Substituting variables is nice but what if we have to add code only if certain condition is met?

Hello {{if .Name}} {{.Name}} {{else}} there {{end}},
This is a template which prints your name, hurray!

now the template engine will print the name only if the name exists, otherwise it will just print Hello there,

Feature 3 — Ranges:

Now let’s add a shopping cart to the template, we’ll start from the data schema this time:

{
  "Name": "Matan Golan",
  "Cart": [ 
    {
       "Name": "Sunglasses", 
       "Price": 199 
    },
    {
       "Name": "Jetpack", 
       "Price": 599
    }]
}

and we’ll modify the above template to look like this:

Hello {{if .Name}} {{.Name}} {{else}} there {{end}},
This is your shopping cart:
{{{ range .Cart }}}
- {{{ .Name }}} ({{{ .Price }}}$) 
{{{ end}}

which will result in:

Hello Matan Golan,
This is your shopping cart:
- Sunglasses (199$)
- Jetpack (599$)

Cool right? and believe me it’s just the tip of the iceberg, if your’e interested in other features of GO’s template engine, please check this excellent cheatsheet.

As you might have guessed packman uses this template engine since we aren’t in the mood of re-inventing the wheel :)

Part 2: The Script

If you’ve read the last part you probably know that we need a way to feed the template engine with data. When designing packman we had only one thing in mind, we don’t want you to learn anything new! Your’e a developer, you know how to code and we want you to use what you’ve already know.

The script is written by you, the template owner, and it has only one purpose, to generate the data model. Later on packman will use this data model to feed the template engine.

To avoid from choosing a specific scripting language, packman is (in theory) able to call any kind of process that can write a json file; that’s right, packman and your script will communicate through temporary files.

When executing your script packman will add 2 additional cli-arguments: the first one is the path to a file containing the input flags of your script and the second one is a path which packman expects you to write the data model file to. Packman will parse your reply and pass it to the template engine.

One example is worth a thousand words:

// This data model, this will be used by the template engine
type ReplyModel struct { 
  Flags map[string]string 
  Timestamp time.Time
} 

func main() { 
  // Read the input parameters:
  flags := packman.ReadFlags()  
  /* Custom Logic */
  // Build the data model:
  reply := ReplyModel{  
             Flags: flags,  
             Timestamp: time.Now(), 
           }  
   // Write the data model as json to the file system:
   packman.WriteReply(reply)
}

func ReadFlags() map[string]string {
   var out map[string]string
   path := os.Args[1]
   flagsContent, _ := ioutil.ReadFile(path)
   json.Unmarshal(flagsContent, &out)
   return out
}

func WriteReply(model interface{}) {
   path := os.Args[2]
   bytes, _ := json.Marshal(model)
   ioutil.WriteFile(path, bytes, os.ModePerm)  
}

Part 3 — Where do templates come from?

Ok, so we have templates and scripts where do we put them? the answer is simple: git! why git? because every developer knows how to work with git, you get full version control, branches, pull request and eventually there is no difference between developing a template and developing any other project.

So your templates will be cloned from any git hosting service (Github, Gitlab, BitBucket, etc …) allowing you to share it easily, making it private, make code reviews or use any kind of work methodology you are comfortable with.

Part 4 — The Packman

What have we learned so far?

  1. Packman

  2. The template engine needs to be fed with data, the template script will generate the data based on your needs, it could parse a protobuf/swagger files, it can be static configuration or even check the weather :)

  3. The template files are stored on any git host.

Now, it’s time to see how all parts gets together to compose packman, again a picture worth a thousand words:

Packman architecture

packman has several commands in its toolchain but the most important one is unpack. for example the following command:

packman unpack \
    https://github.com/securenative/packman-init \
    myProject \
    -author matan \
    -company SecureNative

will clone the template project from “https://github.com/securenative/packman-init” to a local folder called “myProject” passing two flags to the template script (author and company). You can treat these flags as the input of your template and you can have as many flags as you want.

These flags are being written as a json file to a temporary location on your hard-drive and meant to be read by your script.

Then, packman will run your script with two arguments the path to the flags file and the path to save the reply to. Your script runs and suppose to write its data-model to the provided path so packman can read it and feed the template engine with it.

Now packman simply walks on your project file tree, applying the template engine on each one of your files and voila, your’e left with the generated project. Amazing isn’t it?

Packman sample project that just emits the input flags on a README file

Part 5 — What’s now?

First of all, we hope that you liked packman and you’ll give it a try; Rest assured that we will continue to develop packman giving it more capabilities and features that makes your workflow simpler.

If you want a real world example you can check out our protobuf example which takes a protobuf file and generates all the boilerplate needed from it.

And as always, you can give us a hand! packman is an open-source project and we will be more than happy to get any kind of feedback on it.

Happy Templating!