All of us in tech understand that there are deeper margins to be accessed on the performance side of things. No matter how well the product we might have built, we always strive for more speed, efficiency, and reliability. Less system utilization means lesser infrastructure bills to pay, which translates to more profits.

We develop applications to reach maximum users, and the most fundamental part of this is the communication over the network. Overtime, the apps have become highly interconnected, leveraging various networking protocols, architecture patterns, and cutting edge technologies to improve user experience and deliver great results.

Applications typically talk to the “backend server“ via APIs. APIs have been developed on the backend using various programming languages, which implement HTTP protocol support in their source code or libraries. As developers we don’t always think about it in deeper sense. For us, working API means using the http package. At times, we don’t even bother about the HTTP version - 1.1, 2, or 3.

I think it is important to understand the Hypertext Transfer Protocol (HTTP) a bit better and the improvements that it has undergone in the last decade.

HTTP/1.1 - introduced in 1997, laid the foundation as a text-based protocol relying on TCP for communication. When a request is made using HTTP/1.1, it is processed sequentially, often leading to delays when multiple resources are needed, a problem worsened by its lack of header compression and limited ability to handle parallel requests efficiently. Despite these constraints, its simplicity and widespread adoption have kept it relevant.

HTTP/2, finalized in 2015, marked a significant improvement where it adopted a binary format and introduced multiplexing. HTTP/2 allows multiple requests and responses to travel over a single connection simultaneously, reducing latency. It also compresses headers to cut bandwidth use and offers server push, enabling proactive resource delivery. However, its reliance on TCP can still cause delays if packets are lost, as the protocol waits for retransmission, which led to HTTP/3.

HTTP/3, which was standardized in 2022, builds on HTTP/2’s strengths but uses to QUIC (Quick UDP Internet Connections) protocol that runs over UDP. This change removes the bottlenecks introduced by TCP such as - head-of-line blocking, by allowing independent streams to proceed without delay. QUIC also integrates, and in fact mandates, TLS 1.3 encryption. This results in sped up secure-by-default connections with zero-round-trip resuming, making HTTP/3 ideal for modern, high-latency environments like mobile networks. While retaining HTTP/2’s multiplexing and compression, it adds resilience to network changes, such as switching from Wi-Fi to cellular.

This information leads us to ask questions like - how does it matter? what does it take for our applications to make use of HTTP/2 or HTTP/3? Or are they already using it? Does this involve code refactoring? or turning on the infrastructure support is enough? Does it increase or decrease cost? Is the current infrastructure supported?

We can approach these questions in 2 parts - application and infrastructure.

1. Application - also represent the programming effort and what changes we may need.

   1. Optimization for multiplexing - When working with HTTP/1.1, we often bundle static assets to cater to sequential nature of communication. With HTTP/2 and HTTP3, we can leverage multiplexing and parallel delivery.

   2. Server push - HTTP/2, you can take advantage of server push function to send certain critical data to client even before they request for it. Think notifications.

   3. Fallback mechanisms - if you decide to build support for HTTP/3, consider building fallback mechanisms for HTTP/2 and HTTP/1.1. Mainly because, HTTP/3 is still quite new and end user devices might lack support.

   4. Minimize header overhead - HTTP/2 and HTTP/3 improves performance by compressing headers. This opens 2 ways - minimize headers and further improve performance (should do anyway), or add new header to drastically imrove UX/feature without worrying much about overhead.

   5. Handle blocking operations - HTTP/3 leverages UDP, which is great for unreliable networks. But this can also mean that tightly coupled code can misbehave.

2. Infrastructure - servers, functions, clusters, gateways, etc.

   1. Protocol support on servers - Some cloud providers might need you to explicitly enable support for HTTP/2 and HTTP/3 protocols. If you are using reverse proxy like NGINX, make sure to configure the same.

   2. TLS Configuration - HTTP/2 need TLS1.2 and HTTP/3 mandates TLS1.3. Using strong ciphers, configuring load balancers, and creating custom mechanism where required might be needed.

   3. Network config - Use various tools to monitor network performance for throughput, latency. You might need to change firewall rules to enable UDP protocol, and open up secure ports.

   4. CDN with HTTP/3 support - Offloading the protocol complexity (handshakes, etc.) to CDN reduces the processing workload on the application. CDNs can also help in routing traffic to appropriate backend that supports appropriate protocols.

Keeping an eye on the latest updates on the development and adoption of these improvements will always be helpful.

Blog updates

This week I published 2 more articles around gRPC.

gRPC Communication Patterns

One of the advantages gRPC offers over REST based services is the streaming bi-directional communication. Traditional implementations which depend on REST APIs often implement Web Sockets to enable real bi-directional streaming of data packets. I said real because it is still possible to simulate streaming behavior using REST, which of course is not performant, and makes little sense.

gRPC in Kubernetes

I have been experimenting with gRPC for some time now. I wrote some articles to cover the basics like What is gRPC? SSL/TLS Auth in gRPC, and communication patterns used in gRPC. In these topics I went through some of the advantages of gRPC over traditional REST API for inter-service communication – especially in a distributed architecture which led me to wonder about how gRPC works in Kubernetes environment. The crux is – gRPC offers great performance using Protobuf and natively supports uni and bi directional streaming.

The last time I sent out the LDT News issue was a year and ten days ago, and since then I never uttered a word, until today. The reason being - rethinking that involves a lot of overthinking. Rethinking about the definite path to build Let’s Do Tech. I have been addressing questions like - whether it should be a solo peaceful venture or an investment marathon? what does Let’s Do Tech mean to me? What do I want to do with it? etc.

The good news is, I have some clarity and made a few promises to myself around this. I may not be 100% sure of what I want to do, but whatever 20% is fixed is quite concrete. And if you are a fan of 80:20 rule - you get the jig. IMO, it is difficult to make such promises, and one has to be careful. Carelessness is the other way, and it works, but not my cup of tea.

This does not mean that I did not do anything apart from thinking. I just chose to shut my mouth until I reached a point where my position made sense to myself. I have not published anything, anywhere, not even on my personal social media pages. But trust me, I have worked on quite a few things in the meanwhile and I am excited to share them with you, in LDT News series.

As far as this newsletter is concerned - I want to send out issues, but only when I have something substantial to say. That doesn’t mean I will be silent for the next one year. I will be sending out blog post updates where I often write about product dev, system design, architecture, and perhaps starting this year - AI. I would also shed some light on certain decisions I made to run LDT in a specific way, so that others can benefit from it in some way.

Product launch: Budge - for the traveller in you

I launched Budge (v0.1.1, now v0.1.2) in January this year. It is a basic travel app to help you to cut down on your research time for the things to do at the destination you are travelling to. You can also maintain your Budge profile which caters to the question we all get - Where all have you been? You can find more information about the app on the website, or in this blog post.

This is the first time ever I have done something like this. First time building the mobile app, first time releasing it to the app stores under the LDT banner, etc. Am I happy with the app? No. I made a lot of mistakes, and there is still a long way to go. But I learnt a lot of things while doing this. I realise that Budge is a very large project in terms of “could be features”. For now, try downloading the app and feel free to let me know any feedback you may have.

Backstory

I have been a backend guy for all of the 14 years of my career. Even though I know a lot of cool stuff there, I always had to struggle with frontend simply because I could not afford to spend more time. This also kept me from launching my own products. I know - it doesn’t make sense to do everything by myself, more over its not even scalable in the long run. But more on that later.

Blogs

I have started publishing again.

And I never stopped learning. Somewhere in the past I made a post about “Why learning K8s is important“. Amongst other things, I followed my own words and got certified in the October of the last year. Continuing the tradition, this year I plan to learn and work on “AI/ML stuff”. Apologies, but as a newbie I don’t have better words to choose. But I will of course keep publishing blogs on the Cloud and DevOps. Here are some blogs I published recently.

Intro to gRPC and Protocol Buffers using Go

gRPC had been on my radar since a long time, and honestly I never got to explore it fully. The blog post above provides a quick introduction to gRPC and the example will help you realise what it would be like working with gRPC and get you up to speed.

Implementing SSL/TLS Auth in gRPC

Ever since I dealt with certificate based auth mechanism, I have been a huge fan. One can never say not to secure communication these days. SSL/TLS based auth IMO is the “easiest best” way to secure machine-to-machine gRPC comms. It supports other auth mechanisms as well, which I will cover in later blog posts.

One of the most used open source tools to assist container orchestration in the CNCF landscape is Argo. It offers 4 different offerings - Workflows, Rollouts, Events, and CD (Continuous Deployment). I got some time to play around with Argo Workflows and the same is discussed in this post.

Automation is the king today in almost all paths of life. It facilitates convenience and abstracts away the redundant and menial tasks, and lets us focus on better things. Almost in all the automation solutions, and especially in DevOps, automation often involves a series of steps and decisions undertaken by the system to achieve a certain result.

These series of steps and decisions are known as workflows. Argo Workflows mean the same in the context of containers. With Argo Workflows, it is possible to program a workflow to be executed by multiple containers either in sequence or in parallel.

Pre-requisites

Apart from the kubectl, Docker, and minikube installation, working with Argo Workflows requires the Argo CLI to be installed as well. As far as configuration files are concerned, it follows the format similar to, but not same as Kubernetes resources. Argo introduces CRDs for K8s, and Workflows is an example of the same. We will see how later in this post.

Why do we need workflows?

Imagine a scenario where we have to process an image (picture) in a series of steps to apply multiple effects followed by changing the dimensions of the photograph, etc. Let us assume that every modification is performed by an individual programs which are containerised. If we know the sequence of the transformations, then we can use Argo Workflows to spin up corresponding container images, pass the image as input for processing, and then pass the processed image to the next container, and so on.

Basic Argo Workflow example

You might have used Docker to run single containers, or kubectl to run an instance of a container in a pod within Kubernetes. The example below shows how to achieve the same using Argo Workflow. Note that this example is just to familiarise you with how Argo templates are used to create container instances.

apiVersion: argoproj.io/v1alpha1
kind: Workflow                  
metadata:
  generateName: image-inverter-    
spec:
  entrypoint: invertimage          
  templates:
    - name: invertimage              
      container:
        image: docker/invertimage
        command: [ invert ]
        args: [ "picture.jpg" ]

Here, we specify the apiVersion as argoproj.io/v1alpha1 and kind as Workflow. This lets K8s know where to get the resource definitions from and what kind of resources will be created in the K8s environment. We also supply a generateName attribute under metadata which helps generate a unique name by appending a string of random alphanumeric characters.

The spec section defines an entry point. Every workflow requires an entry point, where the execution begins. The templates attribute which follows, defines the entrypoint further with respect to its container image repository location, command to run in the container and some arguments.

Apply this yaml template by running the Argo CLI command as below.

argo submit <filename>.yaml

This will create a container instance of invertimage image, run the invert command with args specified and shutdown once the invert process is complete.

High-level image processing example

The example above is a very basic introduction to YAMLs to be written to use Argo capabilities. Lets assume that we want to process the image in below sequence of activities:

1. Invert operation - containerised as invertimage image

2. Increase the brightness operation - containerised as brightness image

3. Highlight the edges operation - containerised as highlight image

4. Compress - containerised as compress image

To achieve this, we make use of Steps in Argo Workflow, as shown below.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: image-processing-
spec:
  entrypoint: image-processor
  templates:
    - name: invertimage
        inputs:
          parameters:
            - name: imagename
        container:
          image: docker/invertimage
          command: [ invert ]
          args: [ "{{inputs.parameters.imagename}}" ]
    - name: brightness
        inputs:
          parameters:
            - name: imagename
        container:
          image: docker/brightness
          command: [ invert ]
          args: [ "{{inputs.parameters.imagename}}" ]
    - name: highlight
        inputs:
          parameters:
            - name: imagename
        container:
          image: docker/highlight
          command: [ invert ]
          args: [ "{{inputs.parameters.imagename}}" ]
    - name: compress
        inputs:
          parameters:
            - name: imagename
        container:
          image: docker/compress
          command: [ invert ]
          args: [ "{{inputs.parameters.imagename}}" ]
    - name: image-processor
      steps:
        - - name: invert
            template: invertimage
            arguments:
              parameters:
              - name: imagename
                value: "picture.jpg"
        - - name: brighten
            template: brightness
            arguments:
              parameters:
              - name: imagename
                value: "picture.jpg"
        - - name: highlight
            template: highlight
            arguments:
              parameters:
              - name: imagename
                value: "picture.jpg"
        - - name: compress
            template: compress
            arguments:
              parameters:
              - name: imagename
                value: "picture.jpg"

Note that the example above does NOT work - this is only for representational purpose to understand how is it possible to configure Argo Workflow.

Here, after the entrypoint spec, we define a few templates. The template name specified in the entrypoint “image-processor“ depends on more templates, which constitute individual steps in the workflow. The individual steps are also called as leaf nodes and all of them are specified with the corresponding image names based on the operation they perform.

In the image-processor template, we specify steps referring to the leaf node templates and pass the parameter argument indicating the filename of the image that needs to be processed. The sequence is defined using ‘-‘. ‘ - -‘ means that the workflow needs to wait for the previous step to complete. Whereas, ‘ -’ would mean the next job can run in parallel to the current step.

Learn more about Argo Workflows here: https://argo-workflows.readthedocs.io/en/latest/

Hey!

I hope it’s not too late for wishing you a very Happy New Year 2024! Apologies for the late wishes though. My vacations extended for way to long than intended. I would not call them vacations since I wasn’t really having a “leisure” time. But I did take time to address certain things on personal front to put them behind closed doors once and for all.

Anyway, as promised, I am back with some more cool stuff around Cloud and DevOps. While I was away I could think of many ideas with which I can deliver more value to you, starting with this newsletter. Instead of delivering a tech-capsule in each email, I plan to publish more during the week and send a consolidated newsletter at the beginning of the week.

This will also help me in being more conversational with you in my newsletter and perhaps will give more points for you to discuss them with me. As I have always mentioned, I always look forward to your comments and opinions on everything, especially the feedback that would make this newsletter more worthwhile.

This year, I plan for major changes with regards to my career. It finally looks like the sky is clear to take off towards a full-time independent role. Going indie full time for me means, I will write more, explore more, build more, and develop more. I am keen to share this journey with you so that you can do this too!

Having said that, here are a couple of blog posts I wrote this week.

Why Should You Take Kubernetes Seriously In 2024.

New year new resolutions. When it comes to learning a new tech, we often end up in a situation where we learn it “as and when required”. I did the same with Kubernetes since a few years - only to realise its worth towards the end of 2023. It was then, when I realised that mastering Kubernetes is a much needed virtue in Cloud and DevOps career. This post describes why we should take up Kubernetes as priority ASAP, i.e. 2024.

Managing PostgreSQL instance on RDS using Pulumi and Go

Keeping in tune with the New Year theme, I picked up Pulumi to explore more. Well, I didn’t exactly start this year, but I have been doing it since a couple of months while travelling. In this post, I talk about how to manage - i.e. provision and configure - a relational database in AWS. We should know by now that provisioning databases in cloud is straightforward, but configuring them using IaC can be a bit challenging. Here I present an example of provisioning a PostgreSQL database instance, and configuring a few parameters, using Pulumi.

Sumeet N.

I have been dabbling with Pulumi since a couple of months, which is an Infrastructure as Code tool. I would not call myself an expert at this stage, but having closely worked with IaC it was easy for me to learn Pulumi. This is the first ever post I have written on Pulumi, and in this post I discuss how easy it is to spin up a PostgreSQL database instance in RDS.

Provisioning

Creating a relational database in RDS is easy. But the database experts would disagree with this statement. Data is important, and thus how we configure the database for security and efficiency matters a lot. Spinning a basic PostgreSQL instance is a matter of few seconds, whether you do it with web console or any API calls. But to configure the same, there are hundreds of parameters available.

The reason I tried to create and configure RDS database using Pulumi is that I really wanted to experience for myself - how exactly can we configure the database instance parameters. It was as expected - straightforward.

The code below provisions a basic PostgreSQL instance. I have used Pulumi’s Golang SDK, and the same is achievable using TS, Java, Python, C#, and YAML.

package main

import (
	"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/rds"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {

		// Create the database instance
		db, err := rds.NewInstance(ctx, "myrds", &rds.InstanceArgs{
			AllocatedStorage: pulumi.Int(10),
			DbName:           pulumi.String("mypgdb"),
			Engine:           pulumi.String("postgres"),
			EngineVersion:    pulumi.String("15.4"),
			InstanceClass:    pulumi.String("db.t3.micro"),
			Password:          pulumi.String("pass1234"),
			SkipFinalSnapshot: pulumi.Bool(true),
			Username:          pulumi.String("bob"),
		})
		if err != nil {
			return err
		}

	})
}

Since this post focuses on managing PG Database using Pulumi, I will not go through the details of Pulumi here. However, the code above is fairly readable. The IaC code in Golang to create RDS PostgreSQL instance is wrapped in the pulumi.Run() function.

rds.NewInstance() function defines several parameters like allocated storage, database name, engine and it’s version, etc. Of course, you can specify more arguments, and here is the link to Pulumi docs for reference. The parameters specified here are basic - enough to create a database instance.

Let’s try to provision this database using Pulumi’s up command, and observe the output below.

pulumi up
Previewing update (aws-rds)

View in Browser (Ctrl+O): <URL to Pulumi dashboard>

     Type                 Name             Plan       
 +   pulumi:pulumi:Stack  aws-rds-aws-rds  create     
 +   └─ aws:rds:Instance  myrds            create     

Resources:
    + 2 to create

Do you want to perform this update? yes
Updating (aws-rds)

View in Browser (Ctrl+O): <URL to Pulumi dashboard>

     Type                 Name             Status             
 +   pulumi:pulumi:Stack  aws-rds-aws-rds  created (210s)     
 +   └─ aws:rds:Instance  myrds            created (205s)     


Resources:
    + 2 created

Duration: 3m33s

Note: Since I am using Pulumi’s backend to manage state and stack information, I have removed the URL information from above output.

Running pulumi up command has confirmed the creation of RDS instance myrds within a stack named aws-rds-aws-rds. Pulumi creates a stack to manage resources being provisioned in the current directory/project. It counts the stack as one resource, due to which we see that 2 resources are created in the duration of 3m33s.

Fetching the database info after provisioning

If we check the AWS console, we can find the newly created database. It is also possible to configure output variables to print certain information of the newly created. Some information like the database endpoint is only available after the provisioning process. It is convenient to have relevant information available in the console output for various purposes.

Let us update the code above by adding ctx.Export function as shown below. Place this function before the return of Run() function.

//Get database connection string
ctx.Export("DB endpoint", db.Endpoint)

Here, we are accessing the database endpoint output of the Pulumi run. The Export() function saves this output in the state managed on Pulumi’s dashboard, as well as outputs the same in console. The screenshot below confirms the same.

Also observe the console output. You will find the same information there.

...
Outputs:
    DB endpoint: "myrdsf487fca.cosay9dfuboe.eu-central-1.rds.amazonaws.com:5432"

Resources:
    + 2 created

Duration: 3m33s

Configuring PostgreSQL database parameters

As discussed previously, various types of relational database have their specific set of configuration parameters. These configuration parameters help database administrators to apply more settings as per their needs. In AWS RDS, this is achieved by creating a Parameter Group and associating the same with relevant database instance.

The arguments we used in the NewInstance() functions apply to all kinds of databases to be provisioned in AWS RDS. However, since we have provisioned Postgres database and would like to further configure the same, we need to use another Pulumi resource to create Parameter Groups. In the code below, the function rds.NewParameterGroup() provisions a parameter group. Here we have specified a couple of parameters - autovaccum and deadlock_timeout - which are specific to Postgres.

Note: Refer to this link to find reference to more parameters.

If you are following along, place this function before rds.NewInstance() function.

//Parameter group for PostgreSQL
pgParameterGroup, err := rds.NewParameterGroup(ctx, "pg-param-group", &rds.ParameterGroupArgs{
	Family: pulumi.String("postgres15"),
	Parameters: rds.ParameterGroupParameterArray{
		&rds.ParameterGroupParameterArgs{
			ApplyMethod: pulumi.String("immediate"),
			Name:        pulumi.String("autovacuum"),
			Value:       pulumi.String("1"),
		},
		&rds.ParameterGroupParameterArgs{
			ApplyMethod: pulumi.String("immediate"),
			Name:        pulumi.String("deadlock_timeout"),
			Value:       pulumi.String("10"),
		},

		// ... more Parameters
	},
})
if err != nil {
	return err
}

The code above will independently provision a parameter group with above values, but will not associate the same to with our Postgres database instance. To do the same, add one argument named “ParameterGroupName” to the NewInstance() function we previously wrote and associate it with the parameter group (pgParameterGroup) we created in above function, and as shown below.

// Create the database instance
db, err := rds.NewInstance(ctx, "myrds", &rds.InstanceArgs{
	AllocatedStorage:   pulumi.Int(10),
	DbName:             pulumi.String("mypgdb"),
	Engine:             pulumi.String("postgres"),
	EngineVersion:      pulumi.String("15.4"),
	InstanceClass:      pulumi.String("db.t3.micro"),
	Password:           pulumi.String("pass1234"),
	SkipFinalSnapshot:  pulumi.Bool(true),
	Username:           pulumi.String("bob"),
	ParameterGroupName: pgParameterGroup.Name,
})
if err != nil {
	return err
}

This time when we run “pulumi up” the output confirms that we are creating an additional resource - Parameter Group.

pulumi up     
Previewing update (aws-rds)

View in Browser (Ctrl+O): <URL to Pulumi dashboard>

     Type                       Name             Plan       
 +   pulumi:pulumi:Stack        aws-rds-aws-rds  create     
 +   ├─ aws:rds:ParameterGroup  pg-param-group   create     
 +   └─ aws:rds:Instance        myrds            create     

Outputs:
    DB endpoint: output<string>

Resources:
    + 3 to create

Do you want to perform this update? yes
Updating (aws-rds)

View in Browser (Ctrl+O): <URL to Pulumi dashboard>

     Type                       Name             Status             
 +   pulumi:pulumi:Stack        aws-rds-aws-rds  created (307s)     
 +   ├─ aws:rds:ParameterGroup  pg-param-group   created (2s)       
 +   └─ aws:rds:Instance        myrds            created (299s)     

Outputs:
    DB endpoint: "myrds26be326.cosay9dfuboe.eu-central-1.rds.amazonaws.com:5432"

Resources:
    + 3 created

Duration: 5m10s

After a successful Pulumi execution, login to AWS console and verify if the parameter group is associated with the Postgres database instance.

Conclusion

This was a small demonstration of how can we provision and configure RDS Postgres database using Pulumi. If you would like me to write more similar articles, feel free to subscribe and send me a note.

I have always believed in the power of Kubernetes when it comes to managing large workloads for large organisations. However, I am a natural critic, and this virtue drove me toward other directions where I started searching for “options”. And there indeed are a few other ways to do what Kubernetes can do. So after travelling the world, I have come to the conclusion that Kubernetes is here to stay and it shines different. Here, I elaborate on a few factors of my personal analysis on why you should take Kubernetes seriously if you haven’t yet.

How to skip Kubernetes and the guilt.

I have always been fond of exploring new technologies. Thus, when I came across K8s in 2016 for the first time, I did attempt to explore it then. Soon I found out there were so many concepts, features, and strategies to learn. Moreover there were things beyond documentation, which we could only grasp with experience. I liked the technology then, but it did not make sense for me to invest time dedicated only for Kubernetes.

In the following years, I did work on few more projects involving Kubernetes but the same attitude was carried forward. During this time the serverless domain had started to heat up. The buzz around “how cost-effective the serverless technology is” had just begun. There came a point where I thought why can’t we do the deployments the serverless way? Why do we have to be good at K8s?

Being good at Kubernetes is not a joke. It does require dedication. You may work full time on provisioning and managing Kubernetes clusters for a couple of years, and still feel lost.

Or even better, if money is not a problem, then why not stick to the basics? Let’s spin up auto-scaling capabilities and buy a few reserved EC2 instances with load balancers, and we are good to go right? So with such thoughts, I only learnt Kubernetes “when required” in the projects, and focused more on alternatives instead.

It was only in the 2nd half of 2023, I could conclude that nothing beats Kubernetes and had to dedicate my time to learning the same in greater details and more focus. At the same time, I have been thankful to all the “when required” learnings, which in my opinion are significant and unorganised.

What’s wrong with non-Kubernetes methods?

Generally, a lot depends on the kind and scale of the workload you want to run. Based on this information there are a couple of options as mentioned above.

Deploying the workloads on EC2 instances in an auto-scaling group, behind a load balancer is a traditional way to do it. But it comes with it’s own complexities with respect to provisioning of EC2 instances, making them part of the auto-scaling group, placing them in appropriate subnets, paying for the unused capacity, and not being able to go multi-cloud.

Agreed, that Kubernetes has a steep learning curve and setting up a cluster can be a bit of a complex task. Additionally adopting containerisation and the associated DevOps practices can add up to this initial effort.

But once done, its abstraction offers flexibility and scalability with minimum efforts. It abstracts away the cognitive load required while thinking about managing new workloads with varied resource requirements.

On the other hand, serverless sounds quite promising in theory. You may even feel the same way for initial few weeks when traffic is limited. But the cost-effectiveness fades away during peak traffic, when compared to Kubernetes. Serverless also has other not-so-minor issues like cold starts, and there has hardly been any ideal solution around for them.

Writing web applications for serverless architecture also involves utilizing multiple other services, which require learning. More services also mean more elaborate pricing model. If not given enough attention, it can cause unintended cost implications.

Why does Kubernetes make sense?

To answer this simply - Kubernetes provides the much needed optimisation on all fronts at higher scale.

Working with traditional methods or serverless is completely fine, but with conditions. In terms of cost, if the incoming traffic is not very high, a serverless setup can be very very cost-effective and efficient. The same amount of traffic if served with EC2 instance will accumulate a lot of unnecessary EC2 instance costs.

On the other hand, serving high traffic on serverless deployments will soon reach the billing limits and will tend to uncover hidden costs in an unpleasant way. One should remember - refactoring serverless apps to containers or executables, and vice versa also involves some effort. Auto-scaling group might sound a better option for high traffic, but it is not the most optimum solution either.

With the experiences I have had, I can say for sure that all the capabilities offered by Kubernetes make the most sense in these contexts. Kubernetes might not be the right choice at the beginning, but I will always choose the same for it’s long term management and maintability aspects.

When does it make sense to use Kubernetes?

The previous sections highlight the fact that it might not be right for small applications and might not financially make sense at the beginning. Kubernetes is definitely not a solution if you are looking at a couple of USD 1000 in monthly revenue. In such cases, I would still stick to which ever non-K8s approach I have been using till now.

It is important to understand the relationship between the revenue and the costs associated with running a Kubernetes cluster. With this understanding we get a better clarity of the pivot point. Now, depending on how you run your business, you may have this clarity from day 0, and if thats the case and you are sure you will generate enough revenue that overshadows the costs to run K8s - then by all means, go for K8s from day 1.

However, for smaller applications, applications with lesser traffic, non-critical applications, there is either no revenue model attached, or has a very small revenue. Then K8s might not be the right choice here.

Lets assume that you are not sure about how a specific application/business would grow, and you decide to go the non-Kubernetes way. First, this is completely fine. Second, keep an eye on economics related to Kubernetes. Periodically ask this specific question to yourself - How much would it cost to run all of this on K8s? vs. How much is it costing you right now? That would give you an idea about when to make a switch with a sufficient runway.

Industry demand and influencing factors

Lastly, I am not the authority to preach all of this. Looking at the industry, the job openings, the fact that every major and minor cloud platforms offer Kubernetes as a service - it is clear that Kubernetes has proven its importance time and again.

Some of the crucial factors to think about Kubernetes success are listed below. Feel free to read further on these topics.

1. Strong community support

2. Rich ecosystem of K8s related products

3. Benefits offered by containerisation

4. CI/CD flexibility

5. Microservices architecture

6. Ease of scaling

7. Reduced operational overhead

8. Vendor lock-in

Conclusion

So 2024 indeed is the 2nd best time to dedicate some time to learn Kubernetes and add this valuable skill to your Cloud and DevOps resume.

Sumeet N.

Hey!

Have you been able to achieve the goals you set at the start of this year?

In the last 2 months I have been writing about Golang in Let’s Do Tech’s newsletter, and have delivered several critical aspects around Go Theme. Thank you for getting back to me with queries and appreciation texts. I hope all of you got some value out of it.

Having said that, I am here to provide some updates to this newsletter going ahead. As mentioned in one of the previous posts, I will be writing a series of newsletters around a specific topic related to cloud-native. For example, in Sept and Oct, it was about Go Theme.

The break

The Themes will continue, but perhaps after a long pause. In the month of November, I will be travelling a lot and may not be able to keep up with regular newsletter deliveries. The newsletters will be delivered with reduced frequency for November and may be even in December - if the travel plans extend. But starting January onwards, I will be back with regular frequency i.e. every Monday.

I will be most active on Twitter @letsdotech_dev though! Do follow me there to get the latest updates.

Format update

I took some time in the last 2 months to reflect on the value being provided in current newsletters. To be fair, I would not say I am communicating everything I publish on the internet. So I have decided to change the format to make it better by linking to the articles I wrote and were published on various platforms.

It will not just be theory anymore. Instead, it will have links to the latest published technical articles written by me. These articles encompass multiple “categories” - IaC, Productivity, Architecture, etc.

Themes will still be there, but instead of delivering the theory in a lengthy newsletter, I will shorten it and provide the link to the details. Some of the upcoming themes are on the lines of DevOps, Cloud-Native, and Platform Engineering concepts.

Path forward for LDT

I started LDT to primarily share my solo journey in tech business. I have made some progress this year, but I am still not at that point where I want to be. But where do I want to be? Will it be about consulting, or product dev, or farming?

To cover the same, I will be starting a new member-only newsletter series (in January, if things go well). I plan to share lessons from tech, business, marketing, and if theres anything else’s perspectives. I also plan to share numbers!

So the path forward for LDT will be revealed in more details in the new series!

That’s it for now! I will see you around soon.

Sumeet N.

The Docker-Kubernetes' containerization of Go applications and orchestration capabilities has embarked us in a new era of streamlined application deployment. The robustness of Go, coupled with the portability and scalability advantages of Docker and Kubernetes, offers us a potent toolkit to conquer the challenges of contemporary software deployment.

As the demand for scalable and fault-tolerant applications skyrockets, so does the need for insights into best practices that optimize resource utilization, ensure seamless scaling, and strengthens the application's resilience. By combining Go's speed and concurrency with Docker's containerization and Kubernetes' orchestration prowess, we gain the ability to architect applications that transcend traditional deployment hurdles.

In this post we will cover 10 best practices to achieve the same.

Containerized Application Design

Design the Go application with containerization in mind. Keep components loosely coupled and microservices-oriented, allowing them to function as independent containers that can be easily scaled and updated.

Optimized Docker Images

Craft minimalistic Docker images for the Go applications by utilizing multi-stage builds. Begin with a base image, compile the Go application, and then copy only the necessary artifacts into the final image. This reduces image size and improves security.

Efficient Resource Utilization

Tune the application's resource allocation to prevent over-provisioning. Monitor the application's memory and CPU usage to right-size Kubernetes pods, ensuring efficient resource utilization and minimizing costs.

Horizontal Pod Autoscaling

Leverage Kubernetes' Horizontal Pod Autoscaler to automatically adjust the number of replicas based on traffic. Set up custom metrics and define scaling thresholds to ensure Go application handles varying loads while minimizing downtime.

Graceful Shutdown and Startup

Implement graceful shutdown and startup mechanisms in the Go application to ensure that it handles connections, completes ongoing tasks, and properly cleans up resources during pod scaling or maintenance events.

Health Probes and Readiness Checks

Utilize Kubernetes' liveness and readiness probes to monitor the health of the Go application. These checks help Kubernetes decide whether a pod is ready to serve traffic and when to restart it.

Secrets Management

Store sensitive information like API keys, passwords, and configuration details outside the application code. Use Kubernetes Secrets or other secure methods to inject these secrets into the Go application at runtime.

Logging and Monitoring

Integrate centralized logging and monitoring solutions to gain insights into the Go application's performance and health. Tools like Prometheus for monitoring and Grafana for visualization can help us identify and mitigate issues promptly.

Deployment Automation

Implement CI/CD pipelines to automate the deployment process. This ensures consistent and reliable application updates across environments while reducing the risk of human error.

Error Handling and Recovery

Develop robust error-handling mechanisms in the Go application to gracefully handle failures and recover from errors. Implement retries, circuit breakers, and fallback strategies to maintain application stability.

By adhering to these best practices, we can deploy Go applications with Docker and Kubernetes in a manner that optimizes performance, scalability, and reliability while fostering a streamlined development and deployment workflow.

Sumeet N.

In this post, we go through various aspects of data serialization and deserialization. As developers, we often find ourselves working with data in various formats, ranging from simple text files to complex data structures. Serialization involves transforming data into a format that can be easily stored or transmitted, while deserialization is the process of reconstructing that data into its original form.

These processes play a crucial role in modern software development, where data interchange between different systems is a common requirement. We will explore how to perform data serialization and deserialization in the Go programming language, uncovering techniques, best practices, and the libraries that make these tasks efficient and manageable using Go programming language.

What is Data Serialization?

Data serialization is a fundamental concept in computer science, and Go programming. Serialization enables seamless communication and data interchange, especially when dealing with diverse technologies, languages, or platforms. In Go, understanding data serialization is essential for building robust and scalable applications that interact with external services, databases, or even for implementing efficient communication between different components of a single application.

Go offers a variety of techniques to accomplish data serialization, each catering to different scenarios and requirements. One common approach is using the encoding/json package, which provides a straightforward way to serialize Go data structures into JSON format, a widely used standard for data interchange.

JSON's human-readable and lightweight nature makes it a popular choice for web APIs and configuration files. However, while encoding/json is great for simple use cases, more complex scenarios might demand additional features or performance optimizations.

For scenarios where performance is critical, Go supports binary serialization through the encoding/gob package. Gob serialization is more efficient than JSON serialization in terms of both size and speed, making it a suitable choice for high-performance applications or systems with limited bandwidth.

Serializing and Deserializing JSON, XML, and Gob in Go

JSON (JavaScript Object Notation) is one of the most used formats for data interchange due to its simplicity, human-readable structure, and compatibility with a wide range of programming languages. Go makes JSON serialization and deserialization a breeze with its built-in `encoding/json` package.

This package allows us to effortlessly encode Go data structures into JSON and decode JSON data back into Go objects. Whether we are building RESTful APIs, handling configuration files, or communicating between different microservices, JSON serialization is a versatile tool at your disposal.

XML (eXtensible Markup Language) is another popular format for data serialization, particularly in scenarios where hierarchical data structures need to be represented. While XML might not be as lightweight as JSON, it offers more expressive power for representing complex data relationships and metadata.

In Go, XML serialization and deserialization are supported by the `encoding/xml` package. This package empowers us to marshal Go structs into XML and unmarshal XML data into Go objects, while preserving the hierarchical and attribute-based nature of XML.

For prioritising performance, Go offers an alternative in the form of binary serialization with the `encoding/gob` package. Gob serialization is a Go-specific format that leverages compact size and efficient encoding/decoding speeds. It is particularly well-suited for inter-process communication, local storage, or network communication between Go applications.

Versioning in Go’s data Serialization

Versioning and Compatibility are crucial aspects of data serialization that we must address to ensure seamless communication and maintainability in evolving software systems. When data structures evolve over time due to feature enhancements or bug fixes, it is essential to manage these changes in a way that doesn't disrupt existing functionalities. This is where versioning comes into play.

When serializing and deserializing data, it's important to be prepared for changes in the structure, such as adding new fields or modifying existing ones. Without proper versioning, changes can lead to deserialization errors, data loss, or even system failures.

To handle changes in data structures without breaking deserialization, Go provides a powerful tool: struct tags. Struct tags are metadata annotations that can be attached to struct fields, providing additional information to the encoding and decoding processes. By using tags, we can specify how a field should be serialized and deserialized, even across different versions of a data structure. This allows for mapping fields to different names, ignoring fields that are no longer needed, or even implementing custom serialization logic for specific cases. Example below.

type OldUserProfile struct {
  ID          int    `json:"user_id"`        // Field Renaming
  Username    string `json:"username"`       // Field Renaming
  Email       string `json:"-"`              // Field Omission
  CreatedTime int64  `json:"created_timestamp"` // Custom Encoding Logic
}

type NewUserProfile struct {
  ID          int       `json:"id"`                 // Field Renaming
  Username    string    `json:"username"`           // Field Renaming
  Email       string    `json:"email,omitempty"`    // Field Omission
  CreatedTime time.Time `json:"created_time"`   // Custom Encoding Logic
  NewField    string    `json:"new_field,omitempty"` // Optional Fields
}

This version identifier informs the deserialization process about which version of the data structure is being used, enabling it to apply the appropriate parsing logic. Deprecated fields are handled by marking them as deprecated in the code and providing clear migration paths for users to update their data. The use of explicit versioning also empowers us to communicate changes effectively, making it easier for users to adapt their systems accordingly.

Security considerations

Improper handling of serialized data can lead to severe vulnerabilities and compromise the integrity and confidentiality of applications.

Serialization attacks: Attackers can manipulate serialized data to execute malicious code during deserialization, potentially leading to remote code execution, denial of service, or data leaks. These attacks exploit weaknesses in the deserialization process, making it crucial for us to implement stringent security measures.

To prevent security risks during deserialization, several best practices should be followed.

1. It's essential to minimize the use of third-party serialization libraries and prefer using well-vetted, built-in serialization mechanisms such as the standard encoding packages in Go.

2. Limiting the deserialization of untrusted data is crucial, as it significantly reduces the attack surface.

3. Input validation plays a pivotal role in resisting attacks – implement strict input validation and sanitize incoming data to ensure it adheres to expected formats.

4. Applying the principle of least privilege by restricting the permissions of deserialized objects can also mitigate potential damage.

5. Struct tags, used for both serialization and validation, can help enforce constraints on deserialized data.

6. Sanitization involves removing or neutralizing potentially harmful content from data before processing. This is particularly important for fields like strings that might contain special characters or injection attempts.

7. Utilizing regular expressions, predefined whitelists, or blacklists can aid in effective sanitization.

Sumeet N.

From injection attacks to cross-site scripting and everything in between, there is a lot to discuss as for as securing our codebase is concerned. In this post, we delve into the world of secure coding practices in Go, shedding light on the common vulnerabilities that can undermine the integrity of your software.

Common Vulnerabilities, and Go

In software development, vulnerabilities are like hidden traps, waiting to be exploited by malicious actors. Injection attacks, Cross-Site Scripting (XSS), Cross-Site Request Forgery (CSRF), and other common vulnerabilities have become common in many applications.

Injection attacks, such as SQL injection, occur when untrusted data is fed into an interpreter as part of a query, causing unintended and potentially harmful actions. XSS involves malicious scripts injected into web pages viewed by users, compromising their data and privacy.

CSRF manipulates a user's trust in a specific website, leading them to unknowingly perform actions they did not intend. Recognizing these vulnerabilities is the first step towards fortifying our software against cyber threats.

Go is not immune to these vulnerabilities. Injection attacks can infiltrate Go applications through unsanitized user inputs, exploiting data validation. XSS can manifest in Go web applications when output is not properly sanitized, opening a gateway for malicious scripts. CSRF attacks can exploit Go applications if they lack mechanisms to verify the origin of incoming requests. Understanding how these vulnerabilities can materialize within the Go environment is crucial for us aiming to build robust and secure applications.

Avoiding Memory Corruption and Buffer Overflows

Memory corruption and buffer overflows arise from improper manipulation of memory spaces, leading to data corruption, crashes, and even unauthorized access. Memory corruption occurs when a program writes data outside the bounds of allocated memory, often due to unchecked buffers or inadequate input validation. These vulnerabilities can grant malicious actors unintended control over a program's execution.

One of Go's standout features is its ability to mitigate the risks of buffer overflows and memory corruption through a combination of features and practices. Go employs a garbage collector that automatically manages memory, deallocating unused memory blocks and reducing the likelihood of memory leaks.

Moreover, Go enforces strict array and slice bounds checking, ensuring that memory writes stay within their designated spaces. This fundamental shift in memory management inherently reduces the prevalence of buffer overflow vulnerabilities, paving the way for more robust and secure applications.

Go slices and arrays play an important role in preventing buffer overflow incidents. Slices are dynamic structures that offer a safer alternative to arrays, as they handle their own memory management. The built-in append() function, for instance, ensures that underlying arrays are resized as needed, eliminating the risk of buffer overflows caused by manual memory management. This fundamentally shifts the onus of memory management away from developers, reducing the likelihood of accidental vulnerabilities.

Go does allow the use of pointers and unsafe operations, but it's crucial to wield this power responsibly. The "unsafe" package sidesteps some of Go's safety mechanisms, making it a potential breeding ground for vulnerabilities if not used judiciously. We should exercise extreme caution when resorting to unsafe operations, using them only when absolutely necessary and with a thorough understanding of the risks involved.

Securing Data in Transit

Secure communication begins with encryption, and Go offers a rich set of cryptographic libraries that cover a wide range of encryption algorithms, from AES to RSA. The `crypto` package in Go provides easy-to-use APIs for encrypting and decrypting data, as well as generating and managing keys. This package is an invaluable resource for us looking to implement encryption standards within their applications, ensuring that data transmitted over networks remains confidential and tamper-proof.

Transport Layer Security (TLS) is the cornerstone of secure communication over the internet. Go simplifies the implementation of TLS through the `crypto/tls` package, enabling us to establish encrypted connections with ease. This package allows for the configuration of SSL/TLS certificates, ciphersuites, and mutual authentication, ensuring that data exchanged between clients and servers remains private and untampered.

Go's standard library includes the `net/http` package, which seamlessly integrates with TLS to provide a secure foundation for building web servers. With just a few lines of code, we can enable HTTPS, encrypting data exchanged between users and servers. This is particularly crucial for protecting sensitive user information, such as login credentials and personal data, from falling into the wrong hands.

Securing Data at Rest

Go's standard library includes the `crypto` package, which provides encryption and decryption functions to secure data before it's stored. Using symmetric or asymmetric encryption algorithms, we can encode sensitive information in a way that only authorized parties with the appropriate keys can decode.

Go's `crypto` package assists developers in generating and managing cryptographic keys securely. We can employ techniques like key derivation and key wrapping to ensure that keys are well-protected, preventing unauthorized access to the encrypted data.

Modern applications heavily rely on databases to store and manage vast amounts of data. Go’s capabilities extend to database security, enabling developers to encrypt data before storing it in databases like PostgreSQL and MySQL. By encrypting data at the application level, we retain control over the encryption process, mitigating the risk of database breaches compromising sensitive information.

Code Reviews and Analysis

Ensuring code quality is essential to building reliable and secure applications. Code reviews and static analysis stand as two vital pillars of this quality assurance process. These practices take on even greater significance, as they help identify issues early, promote best practices, and contribute to the creation of maintainable and efficient codebases.

Code reviews are a collaborative practice where developers scrutinize each other's code to identify defects, improve readability, and enforce coding standards. The approach to code reviews is guided by the language's simplicity and readability.

With its strict formatting requirements and conventions, Go's codebase becomes predictably readable. Code reviews in Go emphasize clean code and adherence to best practices, fostering a culture of quality that translates into more maintainable software.

Static analysis tools are essential tools to scrutinize source code without executing it, identifying potential issues and vulnerabilities before they manifest. Go's static analysis tools, such as `go vet` and `golint`, provide us with insights into issues like code correctness, style violations, and potential performance bottlenecks.

The synergy between code reviews and static analysis is of great value. Code reviews provide human insight, fostering an environment of knowledge sharing and mentorship among developers. Static analysis tools, on the other hand, offer a systematic and unbiased perspective that can catch issues that might escape human eyes. Together, they create a robust defense against bugs, vulnerabilities, and suboptimal code.

Resources

Go Security: The official Go website provides a dedicated page on security best practices, covering topics like authentication, encryption, and secure coding guidelines.

Go Crypto: The standard `crypto` package in Go offers a comprehensive set of cryptographic functions and libraries for encryption, decryption, hashing, and more.

Web Application Security in Go: This GitHub repository contains a curated list of resources and libraries for building secure web applications in Go.

Awesome Go Security: This GitHub repository gathers a collection of resources, libraries, and tools related to security in Go programming.

Sumeet N.

Microservices architecture has gained immense popularity in recent years, while organizations on the other hand, have chosen to move away from monolithic software design. Microservices is a design approach where a complex application is broken down into smaller, loosely coupled services, each responsible for a specific piece of functionality.

These services are developed, deployed, and scaled independently, fostering agility and adaptability in the face of evolving business needs. By decoupling components, microservices allow teams to work on individual services concurrently, accelerating development cycles and enabling rapid innovation.

In this post, we dive into the world of microservices architecture and explore how Go, with its speed, simplicity, and concurrency support, can be the perfect choice for developing and deploying microservices.

Go for Microservices

Using the Go programming language for building microservices offers a range of distinct advantages that align perfectly with the architectural principles of microservices.

1. Go's exceptional performance and efficiency stand out. With its compiled nature and lightweight concurrency model, Go enables microservices to handle high traffic loads and concurrent requests with remarkable speed. This is crucial in a microservices ecosystem where responsiveness and low latency are paramount, allowing applications to efficiently manage numerous simultaneous interactions.

2. Go's simplicity and clean syntax contribute to rapid development and ease of maintenance. Microservices projects often involve multiple services, each with its own codebase. Go's straightforward syntax reduces the cognitive load on us, making it easier to write and understand code.

3. Go's built-in testing and profiling tools add another layer of convenience, enabling us to ensure the reliability and performance of microservices throughout their lifecycle.

4. Go's native support for concurrency through goroutines and channels is a game-changer for microservices architecture. It’s concurrency primitives allow us to elegantly manage these tasks without the complexities of traditional threading. This results in applications that are both efficient and scalable.

5. Finally, the compact and self-contained nature of Go binaries simplifies the deployment of microservices. Go programs compile to standalone executables that include all their dependencies, eliminating the need to manage complex runtime environments. As a result, deploying, scaling, and managing individual microservices becomes smoother, reducing potential conflicts and streamlining the overall system architecture.

Overall, Go's performance, simplicity, concurrency, and deployment characteristics makes it a great choice for architects and developers seeking to develop robust and responsive microservices systems.

Communication Between Microservices in Go

When it comes to communication between microservices in the Go programming language, several strategies and tools are employed to ensure seamless interaction and data flow.

1. HTTP: Go's native support for HTTP is a natural fit for microservices communication. Leveraging the standard library's `net/http` package, we can effortlessly create HTTP-based APIs that facilitate the exchange of data between services. With frameworks like Gorilla Mux, we can easily build sophisticated HTTP routing and middleware, streamlining the development of RESTful APIs.

2. gRPC: A powerful communication mechanism in the Go ecosystem is the use of gRPC. Built on top of HTTP/2, gRPC offers efficient and low-latency communication by employing protocol buffers for serialization and deserialization. This approach is particularly advantageous in scenarios where high-performance, real-time communication is required.

3. Event-driven: When aiming for event-driven communication or asynchronous messaging between microservices, Go's channels and goroutines are most useful. While not a dedicated messaging framework, Go's concurrency primitives provide a lightweight and intuitive way to establish communication patterns like publish-subscribe or request-reply queues. Libraries such as NATS or RabbitMQ are employed to extend Go's capabilities in event-driven communication scenarios.

Go provides a robust foundation for creating effective communication pathways between microservices. Whether through traditional HTTP, gRPC, event-driven channels, or the assistance of orchestration tools, Go empowers us to build microservices systems that communicate seamlessly and reliably, fostering the growth and adaptability of modern software architectures.

Securing Microservices using Go

Securing microservices is of paramount importance in today's interconnected and distributed software landscape. The Go programming language offers a range of features and libraries that can be leveraged to fortify the security of microservices-based systems.

1. Authentication and authorization: Go's standard library provides tools for implementing authentication mechanisms such as JWT (JSON Web Tokens) and OAuth2. These protocols enable services to validate the identity of users and grant them access based on predefined roles and permissions.

2. Simplicity and readability: The clear and concise syntax of Go makes it easier to write secure code by minimising the potential for common programming errors that could lead to vulnerabilities like injection attacks. Additionally, Go's type safety and memory management features help prevent buffer overflows and other memory-related vulnerabilities that can be exploited by attackers.

3. Secure communication: Go's support for HTTPS through the `net/http` package enables services to establish encrypted connections using SSL/TLS protocols, safeguarding data in transit from eavesdropping and tampering. The open-source package "crypto" in the Go standard library provides a comprehensive set of cryptographic functions that can be used to implement hashing, encryption, and other security measures.

Conclusion

The fusion of microservices principles and the Go programming language has unveiled a powerful synergy, offering architects and developers the tools to create systems that adapt, scale, and withstand the challenges of today's complex digital world. From dissecting the intricacies of communication to ensuring security, monitoring, and debugging, we've delved into the core aspects that underpin the success of microservices architecture using Go.

The realm of microservices is a space of both boundless opportunities and formidable challenges. With the proper understanding of service decomposition, communication patterns, fault tolerance, and the array of tools Go brings to the table, I hope you are at a bit of comfort while exploring the avenues of building microservices using Go.

Sumeet N.

The foundation of any robust software project depends not just on how optimally the code is written, but also on how seamlessly it integrates with external libraries, frameworks, and tools. This is where dependency management steps into the spotlight, which ensures our project's stability, scalability, and maintainability. In this post we explore dependency management in the context of the Go programming language.

Go modules are constructs, introduced as the official method for managing dependencies. They define the way we handle external packages. With Go modules, the process of acquiring, updating, and organizing dependencies is streamlined, making it easier than ever to maintain a clear and predictable project structure.

Modules in Go

Dependency management in Go is all about orchestrating the external libraries, frameworks, and tools that your project relies on. Effective dependency management ensures our project's stability, security, and longevity. As the Go ecosystem evolves and grows, managing dependencies becomes increasingly complex, necessitating a reliable solution that can navigate these intricacies seamlessly.

Before the introduction of Go modules, managing dependencies in Go projects was far from straightforward. The Go community initially relied on the GOPATH environment variable to establish a unified directory structure for all Go code. However, this approach had its limitations. It posed challenges when different projects required different versions of the same library, leading to version conflicts and often leaving developers in dependency hell.

With the official release of Go 1.11 in August 2018, the Go community introduced Go modules as the definitive solution to the once-difficult task of dependency management. Go modules are designed to streamline the way developers handle external packages, providing a systematic approach to versioning, compatibility, and collaboration.

A Go module is a collection of related Go packages. It encapsulates not only the source code but also crucial metadata that defines the module's dependencies, version constraints, and other essential information. The main file of the Go module system is the `go.mod` file, a declarative file that outlines the module's structure and its dependencies' specifications. This simple yet powerful file transforms the landscape of Go development by offering easy control over dependency management.

Managing dependencies in Go

The dependencies in Go modules are managed by the `go get` command. This command not only fetches the desired package from the Go module repository but also updates the project's `go.mod` file with the necessary information about the dependency. It eliminates the need for manual updates to dependency lists and version information.

With Go modules, we can define version constraints using a combination of operators and version numbers, making sure that only compatible versions of dependencies are used. This helps in mitigating the risk of unexpected breaking changes due to dependency updates. Further, Go modules adhere to semantic versioning principles, ensuring that version changes are communicated effectively and consistently.

As the development landscape continues to evolve, Go modules provide a robust framework for not only adding and managing dependencies but also for adapting to changing requirements and maintaining the integrity of projects over time. This dynamic toolset empowers us to navigate the complex web of dependencies with confidence, fostering a collaborative and efficient environment for building remarkable Go applications.

Updating and Upgrading Go Modules

When it comes to managing updates in Go modules, the process is designed to be both intuitive and efficient. The `go get` command serves as our friend to the latest versions of our dependencies. By invoking this command with specific version constraints, we can ensure that only compatible updates are pulled, preserving the stability of your project.

Once an update is fetched, Go modules automatically update our `go.mod` file, reflecting the change and keeping track of version information. This smart integration simplifies the process and allows us to remain focused on building, without getting tangled in dependency management complexities.

Navigating the realm of upgrades demands an understanding of semantic versioning. By adhering to the rules of semantic versioning, we can confidently decide when to perform a major version upgrades. Go modules' version constraints facilitate this process, allowing us to incrementally upgrade dependencies without triggering breaking changes in our project.

As Go modules manage our updates, they also bring attention to indirect dependencies, an often-overlooked aspect of the dependency ecosystem. These are the dependencies that our direct dependencies rely on. Go modules automatically track and manage these indirect dependencies, ensuring that they're not just compatible with our project but also aligned with each other.

By adhering to version constraints, semantic versioning, and automated dependency tracking, we can confidently navigate the process of keeping their projects current, secure, and ready to embrace new functionalities while maintaining the stability that is important to any successful software process.

Vendor Directory and Dependency Isolation

The vendor directory, a feature of Go modules, is a designated repository that holds all the dependencies required by a project. It acts as a shield against external changes, creating a self-sufficient environment where the project's dependencies are insulated from changes in the global package space.

This isolation is a key factor in mitigating compatibility issues that can arise when different projects rely on different versions of the same dependency. By containing dependencies within the vendor directory, Go modules prevent unintended interactions and version collisions.

The beauty of the vendor directory lies in its simplicity. When we use the `go get` command to fetch dependencies, Go modules not only update the `go.mod` file but also populate the vendor directory with the corresponding package code. This means that the project can be built and run independently, regardless of the state of the user's global Go environment.

Additionally, Go modules prioritize the contents of the vendor directory over any globally installed packages during compilation, ensuring that our project remains isolated from external changes and that its dependencies are consistently utilized.

Conclusion

The importance of effective dependency management cannot be overstated, as it directly impacts the stability, maintainability, and collaboration potential of projects. Go modules are necessary since they address the limitations of the traditional GOPATH approach and align seamlessly with the needs of modern development.

Go modules not only simplify the process of handling dependencies but also enhance version compatibility, reproducibility, and security in our projects. From creating new projects to updating dependencies, supporting team collaborations, and migrating legacy projects, Go modules offer a comprehensive toolkit that empowers us to overcome challenges and build reliable software.

Additional Resources

For further exploration and learning, consider these additional resources:

1. The Go Blog: Using Go Modules - A comprehensive guide from the official Go blog on using Go modules, including information on the vendor directory and dependency management.

2. Go Modules Wiki - The official Go wiki page on Go modules, providing detailed information about the vendor directory, dependency isolation, and best practices.

3. Go Modules by Example - A collection of practical examples and use cases for Go modules, including insights into how the vendor directory contributes to dependency isolation.

Sumeet N.

Error handling is a critical aspect of software development in Go, playing an important role in creating robust, reliable, and maintainable programs. In Go, errors are considered a first-class citizen rather than an afterthought, emphasizing the importance of gracefully handling unexpected situations. This approach encourages us to confront potential issues head-on, leading to more resilient codebases.

One of the main reasons error handling is crucial in Go is that it promotes program stability. By explicitly addressing errors, we can prevent unhandled exceptions that might lead to program crashes or unpredictable behavior.

Go's emphasis on checking and handling errors at the point of occurrence encourages programmers to anticipate failure scenarios and handle them gracefully, ensuring that a single faulty operation doesn't jeopardize the entire application.

Furthermore, effective error handling contributes to code readability and maintainability. Clear and concise error messages facilitate troubleshooting and debugging. When errors are properly handled and reported, it becomes easier to diagnose issues during development and in production environments, reducing the time spent on identifying and rectifying problems.

Additionally, comprehensive error handling allows us to make informed decisions about how to proceed when things go wrong, whether that involves retrying an operation, falling back to an alternative approach, or alerting administrators about critical failures.

By dealing with errors proactively, we ensure that our applications are more predictable, reliable, and user-friendly, ultimately leading to higher quality software products.

Approaches To Handle Go Errors

As mentioned earlier, error handling is a first-class citizen, and the language provides several approaches to handle errors effectively and gracefully. Here are some common approaches to handling errors in Go:

1. Return Error Values: This is the most straightforward approach, where functions return both their usual result and an error value. If the function executes successfully, the error is typically nil; otherwise, an error value containing relevant information is returned. The calling code can then check the error value and take appropriate action.

2. Panic and Recover: While not recommended for routine error handling, we can use panic to stop normal execution of a function and initiate a panic, and recover to capture and handle this panic, allowing the program to continue running. This approach is more suitable for catastrophic errors.

3. Custom Error Types: Go allows us to define custom error types by implementing the error interface. This enables us to create more informative error messages or group related errors together. This is particularly useful when we need to distinguish between different types of errors.

4. Error Wrapping and Propagation: Sometimes, we might need to wrap errors to provide additional context about where the error occurred. The errors package in Go provides the Wrap function to add context to an error and Unwrap function to retrieve the original error. This helps to preserve the error chain while enriching it with more information.

These approaches provide various levels of granularity and control over error handling in Go. Choosing the appropriate approach depends on the nature of the error and the context in which it's being handled.

Error Types And Assertions

Error types and assertions are mechanisms used for managing and processing errors in a more structured and informative manner.

Error Types

In Go, an error is not just a simple string but a value of an interface type called error. The error interface has a single method:

type error interface {
    Error() string
}

This means that any type that implements a method named Error() that returns a string can be used as an error. This provides the flexibility to create custom error types that carry additional information beyond a simple error message. By defining custom error types, we can include more context about the error, making it easier to identify the source and nature of the problem.

Here's an example of defining and using a custom error type:

type MyError struct {
    Message string
}

func (e MyError) Error() string {
    return e.Message
}

func someFunction() error {
    return MyError{"This is a custom error."}
}

func main() {
    err := someFunction()
    if err != nil {
        fmt.Println("Error:", err)
    }
}

Assertions

Also known as type assertions, are used to extract the underlying value of an interface and check its concrete type. This is especially useful when we work with interface values like error types and need to access the methods or properties of the concrete type. Assertions allow us to safely convert an interface value to its concrete type.

Assertions are performed using the syntax (value).(Type). If the assertion is successful, the value is converted to the specified type; otherwise, a runtime panic occurs.

Here's an example of using assertions with error types:

func main() {
    err := someFunction()
    
    // Check if the error is of type MyError
    if myErr, ok := err.(MyError); ok {
        fmt.Println("Custom error:", myErr.Message)
    } else {
        fmt.Println("Generic error:", err)
    }
}

In this example, the code attempts to assert the error returned by someFunction() into a MyError type. If the assertion is successful, it prints the custom error message; otherwise, it treats the error as a generic error.

Both custom error types and assertions contribute to clearer error handling by allowing you to encapsulate error-specific information and safely work with interface values. This leads to more informative error messages and better debugging capabilities in our Go programs.

Error Management Strategies

Error management strategies are employed to handle unexpected situations, errors, and exceptions that can occur during the execution of a program. These strategies are essential for creating reliable, robust, and maintainable software. Some of the error management strategies used while programming in Go are:

1. Return Errors Explicitly: Go encourages functions to return both the primary result and an error. Errors are returned as a separate return value, often the last one, allowing callers to check for errors directly.

2. Use Named Return Values: Named return values in Go functions allow us to initialise variables along with the return statement. This simplifies error handling by reducing the need to create new variables for return values and errors.

3. Check Errors Immediately: Errors should be checked and handled as close to their origin as possible. This prevents errors from propagating through multiple layers of code and makes error handling more explicit.

4. Wrap Errors: The errors package provides the fmt.Errorf function, which allows us to wrap errors with additional context. This helps to provide more meaningful error messages without losing the original error information.

5. Custom Error Types: Go allows us to define our own error types by implementing the error interface. This is useful when we want to categorize or differentiate between different types of errors.

6. Defer and Clean-Up: The defer statement is used to ensure that certain clean-up actions, such as closing files or releasing resources, are performed even if an error occurs.

7. Logging and Debugging: Go's standard library provides a robust logging package (log) that is used to log errors and other relevant information. Debuggers and profilers are also utilised for diagnosing errors during development.

8. Graceful Degradation: Designing systems to handle errors gracefully and continue functioning with degraded features is an important strategy in Go, especially for distributed systems.

Go's error handling philosophy centers around explicit error checking, simplicity, and transparency. By following these strategies, we can create reliable, maintainable, and robust code that handles errors effectively while maintaining a clear and readable code structure.

Sumeet N.

An interface is a type that specifies a set of method signatures that any type implementing the interface must provide. It is a core concept in Go's type system and is a building block used to achieve polymorphism and abstraction. Following pointers help us explain interfaces better.

1. An interface defines a contract by specifying a list of method signatures (function prototypes) without any implementation details. Any type that implements all the methods listed in the interface is said to satisfy or implement that interface.

2. Unlike some other programming languages, Go's interfaces are implemented implicitly. If a type has methods with the exact method signatures defined in an interface, it is automatically considered to implement that interface.

3. Go's approach to interfaces is sometimes called "structural typing" or "duck typing." This means that a type is considered to implement an interface based on the methods it has, rather than being explicitly declared to implement the interface.

Consider an example of a “shape” interface in Go that implements an “area()” method. Further, there are a couple of more types - “circle” and “rectangle” which implement a method each, with same name - area(). The circle and rectangle types are said to implicitly implement the Shape interface.

Interfaces play a significant role in Go's philosophy of simplicity and composition, enabling code to be more modular, testable, and adaptable.

Why Do We Need Interfaces?

Interfaces in Go have several important uses that contribute to the language's design principles and capabilities.

1. Polymorphism and Abstraction: Interfaces enable polymorphism, which means we can write functions and methods that can work with different types that implement the same interface. This promotes code reuse and allows us to write more generic and flexible code.

2. Decoupling: Interfaces help decouple different parts of our codebase. When we write code that depends on interfaces rather than concrete types, we create a separation between the implementation details and the parts of the code that use those implementations.

3. Composition: Interfaces encourage composition over inheritance. Instead of building deep inheritance hierarchies, we can compose types by combining smaller interfaces. This approach is often more flexible and easier to manage.

4. Testability: Interfaces make it easier to write unit tests and mock implementations. We can create mock implementations of interfaces to isolate and test specific components of our code without relying on real implementations.

5. Flexibility and Future-Proofing: Interfaces allow us to write code that's more adaptable to changes. If we later need to introduce a new type that satisfies an existing interface, we can seamlessly integrate it without modifying existing code.

6. Third-Party Libraries: Interfaces facilitate the integration of third-party libraries. If a library defines interfaces, we can implement those interfaces to customize or extend the library's functionality.

7. Dynamic Behavior: Interfaces provide a way to achieve dynamic behavior in Go. This is particularly useful when working with unknown types at runtime, as in scenarios involving reflection.

8. Code Contracts: Interfaces serve as contracts that define what behavior a type must provide. This makes it clear to developers what methods a type should implement to satisfy a particular interface.

9. API Design: Interfaces play a crucial role in designing clean and usable APIs. They allow us to define the core behaviours that our types should provide, promoting consistency and ease of use.

10. Avoiding Tight Coupling: By programming interfaces rather than concrete types, we avoid tightly coupling different parts of our application. This makes our codebase more modular and maintainable.

Interfaces in Go contribute to the language's simplicity, flexibility, and focus on clean code design. They encourage practices that lead to better software engineering principles, such as separation of concerns, testability, and maintainability.

Interfaces vs. Concrete Types

Interfaces and concrete types represent different aspects of data and behavior abstraction, and understanding their distinctions is important for designing effective and maintainable software.

Concrete types are the building blocks of data in a program. They define the structure and attributes of a specific object or entity. A concrete type provides the blueprint for creating instances with specific data and methods. They types are used to represent tangible entities and encapsulate their properties and behaviors.

Interfaces define a contract for behavior. It specifies a set of method signatures that any type can choose to implement. It serves as a way to guarantee that certain methods will be available on a type, irrespective of its concrete nature. Interfaces are used to establish a common ground for various unrelated types to interact with the rest of the program in a consistent manner- aka Polymorphism.

Concrete types provide the specific implementation details and encapsulate data, while interfaces establish a shared language for communication between different parts of the application. This separation promotes loose coupling, allowing different components to work together without needing to know the intricate details of each other.

Interfaces are implemented implicitly, meaning there's no need to declare that a type explicitly implements an interface. As long as a type defines all the methods in an interface, it automatically satisfies that interface. This design encourages a cleaner codebase, making it easy to add new implementations without modifying existing code.

Empty interfaces {}

Empty interfaces, often denoted as `interface{}`, are a unique and powerful feature in Go. Unlike regular interfaces that define a set of required methods, an empty interface doesn't have any method requirements. This means that any value in Go can be assigned to an empty interface.

The flexibility of empty interfaces allows us to work with values of unknown types, making them a powerful tool for handling dynamic and heterogeneous data. They are commonly used in scenarios where we need to create generic functions that can work with a wide range of data types.

Another important use case for empty interfaces is in reflection, a mechanism that enables programs to inspect their own structure and data at runtime. When combined with reflection, empty interfaces allow us to dynamically examine the type and structure of an unknown value. This is useful when writing code that needs to handle arbitrary data coming from various sources.

However, it's important to use empty interfaces judiciously. While they offer flexibility, they can also lead to less type safety, as type information is lost when a value is assigned to an empty interface. This leads to runtime errors if the actual types don't match the expected behavior.

Empty interfaces provide a dynamic and versatile way to work with values of varying types, making them valuable in scenarios that require handling heterogeneous data or using reflection. However, care should be taken to ensure proper type checks and assertions to avoid runtime errors.

Sumeet N.

The lightning-fast speeds and reduced latency of 5G have ushered in a new era of real-time data exchange, prompting APIs to evolve accordingly. APIs now need to support the seamless and instantaneous transfer of larger data volumes, catering to the increased demand for high-quality multimedia content and dynamic interactions.

This demands a reevaluation of API design, necessitating the creation of endpoints that can handle the surge in data traffic without sacrificing performance. Modern APIs must prioritize efficiency, scalability, and low latency, ensuring that applications can leverage the technology's capabilities to their fullest extent.

In this post, we will explore how to build lightning fast APIs using Go programming language.

Factors contributing to the API Performance

API performance is a multidimensional concept that encompasses factors like throughput, request/response times, and latency benchmarks. Developers must consider these factors while designing and optimizing APIs to ensure they can handle the demands of modern applications, deliver a seamless user experience, and leverage the capabilities of technologies like 5G to their fullest extent.

API performance directly impacts user experience, application responsiveness, and overall system efficiency. Several factors come into play when considering API performance, and understanding these factors is essential for creating responsive and reliable applications.

1. Throughput: refers to the number of requests an API can handle within a given time frame. High throughput indicates that the API can efficiently process numerous requests concurrently. It is especially crucial in scenarios where the API is handling a substantial number of simultaneous connections, for example during peak usage periods.

2. Request-Response Time: Request time is the duration it takes for a client to send a request to the API, while response time is the duration it takes for the API to process the request and send a response back to the client. Low request and response times are essential for delivering a seamless user experience, especially in interactive applications where users expect quick results.

3. Latency: This refers to the time it takes for data to travel from the client to the server and back. With the advent of technologies like 5G, where low latency is a hallmark, APIs must strive to minimize latency to provide real-time and interactive experiences.

Several factors can influence API performance, including the hardware and infrastructure on which the API runs, the efficiency of the code, the complexity of the database queries, and the network conditions, scalability, caching mechanisms, data compression, and optimized algorithms can contribute to improved performance.

Designing Efficient APIs in Go

Designing efficient APIs in Go requires a deep understanding of the language's features and its concurrency model. By leveraging features like strong typing, composition, concurrency, and efficient memory management, we can create APIs that leverage Go's strengths for optimal performance. Here are some key principles and practices to keep in mind:

1. Use Strong Typing and Structs: Go's strong typing and struct support allow you to define well-structured data models. Design your API endpoints to work with well-defined structs, making data handling more efficient and reducing the risk of type-related errors.

2. Favor Composition over Inheritance: Go does not support traditional class-based inheritance. Instead, it promotes composition through embedding structs. This approach encourages clean and modular code, which can lead to more efficient APIs by minimizing unnecessary overhead.

3. Concurrency with Goroutines: Go's concurrency model is centered around goroutines and channels. Utilize goroutines to handle concurrent tasks efficiently. For example, processing incoming requests concurrently, enabling better utilization of resources and improved response times.

4. Optimize for the Heap: Go's garbage collector can impact performance. Minimize unnecessary memory allocation by reusing objects and using object pooling when appropriate. This reduces the load on the garbage collector and improves overall throughput.

5. Keep Dependencies Minimal: Go's philosophy encourages minimal dependencies. Only import packages that are essential for the API's functionality. Excessive dependencies can bloat the codebase and increase startup times.

6. Use Benchmarking: The `testing` package in Go includes benchmarking tools. Regularly run benchmarks to identify performance bottlenecks and track improvements. This helps us make data-driven decisions to optimize our API.

7. Profiling: Go's built-in profiling tools (like the pprof package) allows us to analyze our code's performance. Profiling helps pinpoint hotspots and bottlenecks, guiding our optimization efforts effectively.

Architectural Considerations For Improving API Performance

Any discussion about system performance is incomplete without discussing the architectural details. Some considerations:

1. Minimalist Design: Keep your API design simple and focused. Avoid unnecessary endpoints and minimize data transferred in each request. A minimalist design reduces processing time and response payload size.

2. Caching Strategies: Utilize caching for frequently requested data. Employ edge caching, in-memory caching (Redis), or content delivery networks (CDNs) to serve cached content quickly and reduce the load on the server.

3. Content Compression: Compress response payloads using techniques like Gzip or Brotli. This significantly reduces data transfer time and enhances API response speed, especially for clients with limited bandwidth.

4. Efficient Data Transfer Formats: Use lightweight data interchange formats like JSON or Protocol Buffers. Minimize unnecessary fields and nested structures to decrease serialization and deserialization times.

5. Optimized Database Queries: Optimize database queries by using appropriate indexes, avoiding N+1 query issues, and employing database caching. Well-structured and efficient queries enhance response times.

6. Asynchronous Processing: Offload non-critical tasks to asynchronous processing to free up the API to handle incoming requests promptly. Utilize message queues or event-driven architectures for efficient background processing.

7. Microservices with Service Segmentation: Employ microservices architecture to segment functionality into discrete services. This allows each microservice to be optimized individually, leading to better performance and scalability.

8. Connection Pooling: Use connection pooling for databases and external services. Reusing established connections reduces the overhead of creating new connections for each request.

9. Load Balancing: Distribute traffic across multiple server instances using load balancers. This prevents overloading a single server and ensures even resource utilization.

10. API Gateway for Aggregation: Implement an API gateway for aggregating requests to multiple microservices. This reduces the number of client-server round trips and minimizes latency.

Announcement:

I have enabled the paywall on this publication, and to access posts older than 2 months, the same can be temporarily broken for free by referring this newsletter to more friends. Check out the options below.

Refer a friend

Sumeet N.

One of the most in-demand skills in the cloud-native community is the ability to write concurrent programs to leverage all the multi-core processor power today’s hardware has to offer. Golang was developed keeping concurrency in mind.

Digital transformation does not just mean moving to cloud platforms. In fact, digital transformation is indeed a continuous process undertaken by organisations to constantly optimize their IT spend. Well, not just organizations, solo-businesses, and startups alike - who does not want to save money?

In this post, we will understand the concepts related to Golang’s concurrency, the benefits in the context of cloud architecture, and also explore a few patterns to use concurrency in Golang.

Goroutines vs. Threads

Before we move ahead, it is important to understand the difference between threads and goroutines.

Threads are managed by OS, while Goroutines are managed by Go runtime environment. Threads are single units of execution within a process - multiple threads belong to a process - and all these threads share the same resources. While Goroutines are independently scheduled functions and do not share resources.

Due to the sharing aspect described above, threads tend to be more prone to deadlocks and race conditions. While Goroutines are better immune to the same. Since Goroutines are managed by Go runtime environment, it offers higher level of abstraction - thus easy to implement by developers. Threads on the other hand offer lower level of abstraction.

This managed nature of Goroutines is of great advantage over threads when automatically scheduling them on multiple OS threads. As developers, it reduces the cognitive load of worrying about shared resources, and also makes them lightweight so that instantiation and context switching becomes faster.

What should we know to understand Golang concurrency?

Goroutines and channels are fundamental concurrency constructs in the Go programming language, designed to simplify and enhance the development of concurrent and parallel applications. They allow us to achieve concurrency by enabling multiple tasks to be executed concurrently without the need for creating separate threads or managing complex synchronization mechanisms.

Channels, provide a safe and structured way for goroutines to communicate and synchronize their actions. Channels act as pipes for the exchange of data between goroutines. They ensure proper synchronization and avoid race conditions by enforcing a model where data is sent on a channel by one goroutine and received by another in a coordinated manner, resulting in clean and clear communication between concurrent tasks. Channels are used for data sharing and signaling, allowing goroutines to coordinate their actions and operate in a synchronized manner.

The combination of goroutines and channels enables us to write concurrent programs that are both efficient and comprehensible, fostering easier maintenance and debugging of complex parallel applications.

Top 5 Benefits of Goroutines in cloud architecture

Backend applications built using Golang have greater impact on reliablility, and resource optimization - in a good way!

Efficient Concurrency

Goroutines are designed to be lightweight and have a lower memory footprint compared to traditional threads. This efficiency makes it easier to handle a large number of concurrent tasks in cloud applications without consuming excessive resources.

Scalability

Cloud architectures often require the ability to scale resources up or down dynamically based on demand. Goroutines can help distribute workloads efficiently, enabling cloud applications to handle increased traffic and workload reducing the need to provision or manage additional VM instances.

Parallelism

Goroutines allow for easy parallelism by executing tasks concurrently. This leads to improved performance for tasks that are divided into smaller subtasks, such as data processing, image manipulation, or network requests. In a cloud environment, this leads to faster response times and optimized resource utilization.

Cost Optimization

Cloud services are billed based on resource usage. By utilizing goroutines, applications can make better use of available resources, optimizing the utilization of CPU cores and memory. This efficiency can result in cost savings in cloud deployments.

Resource Pooling

In cloud architectures, resources like database connections or network sockets need to be managed efficiently. Goroutines are used to manage resource pooling, allowing multiple tasks to share limited resources effectively.

Top 3 Frequently used concurrency patterns in Golang

As a very basic example, if you make a function call with a `go` keyword at the beginning, it will automatically be executed in parallel. Let us take a look at some of the advanced patterns below.

Producer-Consumer

As the name suggests, the producer function is where input data is generated and passed to the calling function via channels to consumer function. The consumer function processes any data that arrives on the channel in concurrent manner. There can me multiple producers, and a single consumer can take care of the input data.

Worker Pools

In this pattern, the logic to process the input/data is wrapped in a separate function called “worker” function. The calling function

1. divides the input data into multiple batches

2. creates input channel

3. calls the worker function with each batch of input data

Fan-Out, Fan-In

Similar to Worker Pools pattern, the worker function is present here as well. The difference here is that the calling function waits for aggregating the results from all the workers (fan-in). The calling function

1. divides the input data into multiple batches

2. creates input channel and results channel

3. passes each batch of data to the worker function using go keyword to induce concurrent execution

4. waits for all the workers to return results on results channel

Refer to this video for more information on Golang Concurrency.

Sumeet N.

This is the last newsletter of this month, and I would like to take this opportunity to provide you guys with some updates about the plans regarding the same in upcoming months. I have been doing a bit of organising and thinking about how can I provide more value in your email inbox.

Themes

To being with, I have realised the importance of having a schedule for these emails, and to give that schedule a meaning - I am introducing Themes.

As you may have realised by now, my emails usually land in the morning hours every Monday. That remains the same. However, starting from September I will be sending out emails related to a particular topic. This topic will be discussed in progression throughout the period of a month or two.

Themes define the topic of the month, wherein I will share curated and crisp insights about frequently used and most relevant aspects of the Theme.

For example, I will begin discussing various aspects of Go Programming Language with respect to various architectural usecases, patterns, frameworks, etc. I found so many important topics for Go, that I decided to run this theme for 2 months.

These newletters will be code-less, but architecture and business oriented - enough to give food for thought for the rest of the week.

It is not just about programming languages, I plan to explore various topics in Cloud, DevSecOps, and Serverless domains. Additionally, I will also be running themes covering upcoming projects in CNCF landscape. I am also open for suggestions from YOU - if you want me to cover a specific Theme topic, I would be glad to do so. Next sections will point at a way to get in touch with me for the same.

Social Media

This is a brand new addition. I have decided to create short-form content on various social media platforms - all links here. These short-form content will be multiple mini-deep-dives in the topic being discussed in that particular week. There will NOT be any overlap in the information shared in the newsletter and the short-form content.

The intension here is to spend the week’s time in understanding various aspects of the topic at hand in that particular week. Feel free to follow along if you wish and invite a few friends!

Chat

I’m also announcing a brand new addition to my Substack publication: the Let's Do Tech chat.

This is a conversation space in the Substack app that I set up exclusively for you all — kind of like a group chat or live hangout. I’ll post short prompts, thoughts, and updates that come my way, and you can jump into the discussion.

To join this chat, you’ll need to download the Substack app, now available for both iOS and Android. Chats are sent via the app, not email, so turn on push notifications so you don’t miss conversation as it happens.

Imp! If the chat gets too busy, I will be restricting the ability to start the threads only to paid subscribers. All of us can still continue to contribute then.

How to get started

1. Download the app by clicking this link or the button below. Substack Chat is now available on both iOS and Android.

Get app

2. Open the app and tap the Chat icon. It looks like two bubbles in the bottom bar, and you’ll see a row for my chat inside.

3. That’s it! Jump into my thread to say hi, and if you have any issues, check out Substack’s FAQ.

Join chat

More…?

That is about it for now. There will be more updates in upcoming weeks. The format of the newsletter would be:

1. Announcements and updates (if any)

2. The Theme topic post

3. Curated list of top 3 news in Cloud, DevSecOps, and Serverless (in progress)

Poll

I love exploring tech. I have explored a lot since 10+ years. So, naturally I have come to find a new topic to explore. Given the demand, I plan to explore ML and AI space in coming months.

I am a complete newbie in this space, and would love to share my mistakes, lessons, things-I-could-have-done-better, and overall learning experience so that it helps you learn as well. How about we do it together?

Sumeet N.

I pledged my support to opentf foundation, and here are the details about why I did. I have tried to be as clear as possible from the information I have read.

Context

Hashicorp’s Terraform has been a leading open source IaC tool for almost a decade now. Terraform has helped startups and organizations to streamline their cloud infrastructure operations - saving them a fortune in the digital transformation journeys.

Given their open source commitment, and the success described above, an entire ecosystem of partners, vendors, and community was born providing additional features with freemium pricing model. As far as the pricing model comparison between Hashicorp and most of the other vendors is concerned, they are similar.

What happened?

Hashicorp recently adopted Business Source License v1.1 (BSL or BUSL) which was previously based on Mozilla Public License v2.0 (MPL v2.0). In summary, now they do support open source and community efforts in terms of “copy, modify, and redistribute the code for all non-commercial and commercial use, except where providing a competitive offering to HashiCorp“.

This restriction is specifically targeted towards vendors who are using Hashicorp’s products to build their commercial offerings which provide additional features and benefits in varied flavours.

The main concern expressed by Hashicorp is that the offerings provided by these vendors are either fundamentally or substantially based on Hashicorp’s core products, while not contributing enough to their open source commitment.

How does this affect?

As an individual developer, it does not affect you and me. It also does not affect if your organisation’s commercial offerings are completely different than those of Hashicorp’s. I.e. you can still use Hashicorp’s products in production (if I understand the words correctly).

But if you are a vendor (or work for a vendor) who creates a platform to offer competing features which are similar to what Hashicorp’s platform has to offer, then that is not allowed after a specific Terraform version - v1.5.5.

Overall, I think this is a bit vague. Besides, the cause I am rooting for is greater than this.

Opinion

I truly appreciate and admire what Hashicorp has achieved and contributed in the space of cloud operations. They are indeed an authority in the space of IaC. I am sure this feeling of gratitude is shared by that specific set of vendors too.

It is not that I never had this question about the obvious concern they have expressed. I have always admired Hashicorp to play a fatherly figure in this way. This also makes me understand where they are coming from with regards to this issue - for all the right reasons.

I understand Hashicorp’s decision and the fact that open source projects do undergo this license evolution phase. But at the same time, this always leaves an impact in upcoming days - especially at this stage. Apart from the specific set of vendors and partners, this also sends ripples of uncertainty across the community and customers - even if each of them fully understand what BSL means.

It is not about what the current or previous license means - it is more about the conscious action taken in the light of this situation, which somewhere impacts the confidence of community. That goes on to say that the customers - not the targeted vendors and partners - may now consider other options with lesser licensing complexities - which is also another headache.

To look at it retrospectively with an absurd angle - and I apologise beforehand for this - I question myself - was this all part of the plan by Hashicorp? Why did they wait 10 years for this to happen? In hindsight, Hashicorp could have leveraged this authority to contribute to the healthy competition - with an upper hand - to their vendors. Its not that all the infrastructure pain points are addressed.

I do understand that the pioneers like Hashicorp have a better view about this situation, more so who else would know their business better than themselves. Hashicorp deserves full credit to what they have built, and they are very well aware of what impact this decision would cause. But perhaps, they had tough choices to make from business survival perspective.

The contradiction I am unable to figure out for Hashicorp is - did they really understand the meaning of OSS back then? It is easy to talk about charity, and hard to do so. If business was the corner-stone of their operations, then they should have not touted the “free and open source” slogan in the first place. At least an intimation or a hint would have worked too. As I said before, I understand this decision of theirs, but if that is the case then it all sounds like a misuse of OSS model and community. Almost like gaining popularity and hard-but-free great contributions from thousands in the community, and then licensing it.

I want to but I am not in a position to imagine what would happen if all the OSS decides to adopt BSL/close-source hence forth.

It can happen that I did not understand the license fully, but I am skeptical for 2 reasons:

1. I am not a legal expert, I am just a developer who loves Terraform. I am really not sure about when to use and when not to use it for production.

2. I am not sure if and when will they change their minds again.

I signed the opentf pledge to highlight the impact of this decision on their own ecosystem, more importantly I believe they can do better.

Rest is described in the OpenTF Manifesto.

Sumeet N.

This will be a short one, since I don’t really have a good explanation of the behaviour posed by API Gateway. I have been struggling with this issue all throughout last week. But I do want to appreciate the Auth aspect offered by Cognito and API Gateway.

There are mainly 3 ways to do implement Authentication and Authorization logic for the private API calls made by users and service acccounts.

1. Do it from scratch: This means

   1. Developing user signup, signin, signout workflows from scratch

   2. There are more workflows to be handled - forgot password, reset password

   3. The complexity increases multifold with MFA and email confirmation logic

   4. Store the user credentials securely, etc.

   5. … this is a lot of work

2. Use Cognito with Lambda authorizers: here, all the workflows are handled by Cognito, along with the support for Federated auth with enough room for customizations. The auth logic (both authentication and RBAC) is to be developed in the Lambda functions. Special lambda functions are created to handle these incoming requests. There is indeed some coding involved.

3. Use Cognito with API Gateway authorizers: To think of it, if we categorize incoming requests targeted towards, they are mainly of 2 types - user initiated, and service accounts initiated (for automation purposes). It makes sense for API Gateway to incorporate the Auth handling by leveraging Cognito. By simply setting some parameters like scopes and headers, various use cases are satisfied. Additionally, this is not mutually exclusive to Lambda authorizers. For specific use cases it is possible to incorporate both. For example, for basic authentication, API Gateway authorizers work well, for further authorization Lambda authorizers can be used.

Implementing a secure system is a basic - all pervading - task, but at the same time it causes a lot of deliberation in terms of security. For small teams, Cognito abstracts away this crucial aspect and lets them focus on the core business.

Having followed the 3rd route, when I try to test the logic using Postman, it throws weird error - “not a valid key=value pair (missing equal-sign) in Authorization header”. I think there is either something minor setting that I have not been able to find since a week, or there is an entire universe in auth world that I might be missing. Fingers crossed.

Sumeet N.

Working with Golang can be a pain sometimes, especially when you are new and don’t yet appreciate its beauty. A trivial task like processing a JSON response object - which is way easier in JS, to a point where you don’t even have to care about using variables to hold those values - in Golang, becomes too complicated.

But trusting the process and the wise words spoken around, if you be patient and just continue with the tedious process, there will come a point where you would love doing it in Golang. In this post, I help relieve some of that pain.

What’s the deal with JSON and Golang?

JSONs are the top-most universal standards defined for communication of text based information across tools, interfaces, and SaaS-based services. It enjoys inherent support in all the major programming languages today - arguably except Golang.

This caused headaches for me for a few weeks, until the requirements forced it down my throat - a hard pill to swallow. Especially, coming from a JS background, I found it absolutely useless to do so much of wiring just to process a simple JSON - forget about the nested ones.

How we do it in JS?

Its a joke - and I say this with a sense of appreciation. It is way too easy to access attributes or loop over a JSON object for processing. Let us consider a simple JSON object that looks like below.

{
  "name": "value",
  "env" : "production"
}

We can simply access name and env values by doing this:

var jsonObj = <object above>
console.log(jsonObj.name)
console.log(jsonObj.env)

This saves up a lot of development time.

In Golang…

Like otherwise, even in Golang you need to know the structure of the JSON object before running the executable. We take advantage of this to explicitly define a struct for the JSON object being processed. Golang is a bit orthodox that way. To process the JSON object above, we first define a struct as below.

type JSONObj struct {
  Name string `json:"name"`
  Env  string `json:"env"`
}

The code block above just defines the struct in Golang to process the corresponding instance of JSON object. As we can see, we explicitly define the type (as string) for each of the attributes which would be processed by the functions. Additionally, `json:”name”` and `json:”env”` tells the compiler that the incoming JSON object would have attributes named (keys) with these values.

Further, to actually access these values the incoming JSON object - let us call it a response - has to be Unmarshalled into this struct, and vice versa - Marshalled to be converted back in byte format if it has to be returned. To do the same, we have to use the json module of Golang as shown below.

// Assuming we get this response in response.Body
response, err := ioutil.ReadAll(response.Body)
var jsonResp JSONObj
err := json.Unmarshall(response, &jsonResp)

// Access the attributes now -
fmt.println(jsonResp.name, jsonResp.env)

Think of this like an essential transformation of the incoming JSON object so that it can be processed normally in the Golang environment. For the specific understanding of the Golang syntax above, if you are not familiar, I suggest you to check out the documentation or simply ChatGPT it. ;)

What about complex JSONs?

Yes, you guessed it right - things get even worse in Golang. In cases where we have multi-nested JSON objects and for the sake of keeping the code readable, we make use of multiple structs. Consider nested JSON example below.

{
  "name": "John Doe",
  "age": 30,
  "contact": {
    "email": "john@example.com",
    "phone": "123-456-7890"
  },
  "address": {
    "street": "123 Main St",
    "city": "New York",
    "zipcode": "10001",
    "country": "USA"
  }
}

To consume and process such a JSON response, the structs below have to be created, before we can Unmarshall or Marshall the same.

type Person struct {
	Name    string  `json:"name"`
	Age     int     `json:"age"`
	Contact Contact `json:"contact"`
	Address Address `json:"address"`
}

type Contact struct {
	Email string `json:"email"`
	Phone string `json:"phone"`
}

type Address struct {
	Street  string `json:"street"`
	City    string `json:"city"`
	Zipcode string `json:"zipcode"`
	Country string `json:"country"`
}

Notice how the structs for nested JSONs - Address and Contact - are created, and then used in the “parent” struct named Person. Similar pattern is followed if we have 3-level, 4-level, so-on-and-so-forth level of nested JSON.

What have you decided?

Before we talk about the ChatGPT part, perhaps you are thinking this to yourself if you are not a Golang dev already. Not that ChatGPT should be the reason for you to choose this language. Golang has its merits which far outweigh than this generic trick.

No doubt, writing Golang code is tedious. But if you ask any cloud architect worried about compute costs - they will any day prefer Golang.

ChatGPT? Relieve pain?

Now straight to the point - since we already know what kind of JSON is going to be processed, we can simply ask ChatGPT to generate the corresponding structs and it does so in seconds - perfectly with the `json:”<attribute_name>”` notations.

I am shameless in saying this because:

1. We cannot ignore the reality of ChatGPT. When space travel has become affordable, why would you still buy vacations in Hawaii?

2. Manually typing structs is really a tedious and slow process, prone to human error - thus slowing down the development.

That was the tip with some insight and background in Golang. I highly encourage all the web devs to learn Golang for all the right reasons. By no way I meant other languages are not good, each language has its merits.

Sumeet N.