Challenges Encountered

The common functionality of package management tools is to download the modules on which a project depends in some way and then refer to these dependent modules when compiling/running the project code. This is a basic function.

However, under this basic functionality, there are more advanced requirements, such as dependency resolution. Our project may only depend on module A, but module A may depend on modules B and C. This means that our project indirectly depends on projects B and C, and this work is usually done by package management tools. There are also features like version management. We often receive notifications that a certain module has vulnerabilities and needs to be updated. One common way is that modules have versions, and when updating, the updated version of the module will be used.

Diamond Dependency

Because package management tools automate so much work for us, there are some implicit issues that we might encounter. The most common problem is the diamond dependency problem:

Suppose our project is A, which directly depends on two modules, B and C.
Both module B and module C depend on module D.
However, module B depends on version v1.1.0 of module D, while module C depends on version v1.2.0 of module D.

In this scenario, some package management tools I’ve used handle it differently. For example, using pip, project A can specify the version of D in requirements.txt. However, this can lead to two different scenarios:

On the surface, the first approach seems to solve the diamond problem. However, in many cases, this does not actually solve the problem because versions 1.1.0 and 1.2.0 of D may have changed the code’s interface and structure. Therefore, project B may not run properly.

To avoid this, Go introduces the concept of semantic versioning to restrict the range of version changes, allowing dependency tools to choose the most suitable module version effectively.

Semantic Versioning (SemVer)

Semantic Versioning (SemVer) is a versioning standard proposed and developed by individuals such as Tom Preston-Werner, David Heinemeier Hansson, and Guido van Rossum. The latest version of this standard is SemVer 2.0.0, which defines the specification of version numbers and how to handle increases, changes, and fixes in software versions.

SemVer does not have a formal specification but is managed and maintained through a public repository on GitHub. The repository address is Semantic Versioning Specification. This repository contains detailed explanations, history, and discussions about SemVer. Community members can participate in the discussion and improvement of the standard by raising issues and submitting pull requests.

A brief overview of SemVer:

v0: Unstable Versions

In the convention of Go modules, v0 versions do not guarantee forward or backward compatibility for patches. They are considered unstable versions, allowing for API redefinitions to provide a stable API in subsequent formal versions.

v1: First Formal Version

The v1 version is the first formal version, and it must ensure the stability of the provided interface. Iterations of versions must maintain backward compatibility according to the requirements of SemVer.

v2: Incompatible New Versions

Principle:

If an old package and a new package have the same import path,
the new package must be backwards compatible with the old package.

So when developing a new module that is incompatible with the old version, you need to explicitly add a version suffix in go.mod. For example:

  1. v1: github.com/liuliqiang/gomod/pkg
  2. v2: github.com/liuliqiang/gomod/pkg/v2

The module name is placed on the first line of go.mod:

  1. module github.com/liuliqiang/gomod/pkg/v2

At this point, Go Module is different from other package management software. However, this is Go’s answer to the diamond dependency problem.

Exception:

gopkg.in

Before the introduction of Go Module, gopkg.in allowed modules to use the .vx suffix to differentiate version numbers. Therefore, Go Module allows this exception for gopkg.in to use the .vx suffix. However, for modules with other paths, it must be in the form /vx.

Sample:

  1. gopkg.in/yaml.v1
  2. gopkg.in/yaml.v2

Module Directory Example

For different versions of module code, the recommended organization of the code path is:

  1. github.com/googleapis/gax-go @ master branch
  2. /go.mod    → module github.com/googleapis/gax-go
  3. /v2/go.mod → module github.com/googleapis/gax-go/v2

Does Go Really Solve the Problem?

Burden or Wealth

When I was in middle school, I vividly remember a teacher asking whether traditional culture is a burden or wealth. I was chosen to answer this question, and at that age, I liked to talk nonsense. So, I answered that it is both a burden and wealth because wealth is contained in the burden (looking back, how did I feel like I had the potential to be a spokesperson at that time?).

Returning to the topic, similarly, the Go language started in 2006 (as inferred from Go’s Time Format) and was released in 2009. It was not until Go 1.11 (August 2018) that Go Module support was added as a subcommand of Go, and it became the default in Go 1.13 (September 2019). During this period, I used at least two module management tools: govendor and godep. So, when we started using Go Module, what should we do with libraries that were not released according to the SemVer specification?

This mainly involves two issues:

In fact, these two issues are mixed. For example, a library may not have a tag at all, so when it is depended on, it directly finds the latest master commit. However, the problem is that the library may not have considered compatibility issues at all, and each commit may have changed the interface. This makes package management tools very uncomfortable. From the perspective of package management tools, they cannot handle this situation. Common ways to deal with this include:

The most common approach I see is the third one because Go Module provides the replace feature. This way does not depend on the changes of third parties and can solve the problem within a controllable period. After all, the first two approaches depend on others to solve the problem, and the time is uncontrollable.

Unethical gRPC

Another ironic thing is that Google, as the official maintainer and promoter of Go, their own RPC framework, gRPC, does not follow SemVer. In a Minor Version, it made changes that are not backward compatible with the previous interface. This has been quite uncomfortable for a long time. I once wrote a document introducing this issue: Handling gRPC Go Connection Handshake Timeout. In short, after upgrading to version 1.19, the previous code no longer works. Additional parameters must be added to the code for it to work properly. This is a typical case of not being backward compatible and not conforming to the SemVer specification.

In this case, the simplest solution is certainly to modify our code to meet the requirements of the new version. However, things are often not that simple because our project will depend on other projects, and we cannot modify their code. Therefore, even if we modify our own code, our project may not run properly. In this case, the solution is to suppress the version of gRPC from upgrading. So, when you see that the version of gRPC in your project’s dependencies is suppressed to version 1.18 or 1.33, do not remove this dependency. It is likely that this is encountered a problem.

Should we keep gRPC from upgrading forever? Certainly not. The solution in this case is usually two parallel paths:

After some time, when the projects we depend on or most projects are compatible with the new version, we can upgrade and modify our version. For some dependencies that cannot be pushed or maintained, we can fork them and maintain them ourselves to support our upgrade.

Version Compatibility Strategies

Forward Compatibility Strategies

Add Default Parameters

The first common way to extend function parameters is to add a variable parameter list to existing functions. For example:

  1. func Run(name string)
  2. --->
  3. func Run(name string, size ...int)

This way, the Run function can add support for additional parameters. However, the new Run function needs to handle default values for the added parameters.

Create New Methods

A common way to extend conditions is when the old function does not have a Context, but the new function needs to add Context support. In this scenario, it is common to add a new function, and the new function name is suffixed with Context. The old function is then modified to use the new function:

  1. // Old function
  2. func (db *DB) Query(query string, args ...interface{}) (*Rows, error) {
  3.     return db.QueryContext(context.Background(), query, args...)
  4. }
  6. // New function
  7. func (db *DB) QueryContext(ctx context.Context, query string, args ...interface{}) (*Rows, error)

Conversely, you need to handle the default values when calling the old function.

Backward Compatibility Strategies

Pass Option Extension Parameters

For example, when developing for the first time, our function only supports certain options. However, we know that in the future, we will support more options. In this case, we can set an option parameter for the function. Common ways to implement this option parameter are:

Here is an example of the first approach:

  1. type Config struct {
  2.     // ... ...
  3. }
  4. func Dial(network, addr string, config *Config) (*Conn, error)

Conclusion

In this article, I briefly introduced the core problems that Go Module aims to solve and how it solves them. I also discussed potential issues and solutions. Finally, I introduced several common compatibility strategies (learned from official documents).

However, the mechanism of Go Module has spawned some clever operations, and these operations are applied in MonoRepo. I will dig into this later and introduce these clever operations when discussing MonoRepo.

References