My experience with Github Copilot

I've been keeping an eye on the Github Copilot project for the past 6 months as it has been rolling out, and I have finally had a chance to try it out for myself for the past month or so and I wanted to share my thoughts on it.

What is it?

The Github Copilot project is a collaborative effort between OpenAI and Microsoft to introduce AI into the developer experience of writing code. It is a derivative of GPT-3 called Codex, which has learned on a large set of open source code on Github to build a model.

The promise of Copilot is that it will autocomplete your code for you as you type..

My Experience

My approach to learning how Copilot could affect my work as a developer was to code as I usually would and see what Copilot would do. I had seen others stress testing it by giving it a coding challenge, but I was curious to see how it would affect more standard day to day type scenarios.

Boilerplate Code

When using certain libraries in my day to day programming, there is often a need to write some level of boilerplate code to get the library to work. I found that Copilot was very good at spotting a pattern in the boilerplate structure or common syntactical patterns.

I found that popular libraries would autocomplete the boilerplate code for me almost instantly, where as lesser known libraries would take a little longer to learn and spot a common pattern.

One example of the former I found worked well was when using the Sequelize library. When defining a Model in Sequelize there is a reasonable amount of boilerplate to define the attributes and the relationships between Models in a project.

As you can see in the above example, we first define the attributes as an interface and then define a class which will use those attributes. This is where Copilot was able to spot the pattern of how classes had been defined in other files in the project and made a good assumption on what needed to be written for the Magazine definition.

For the Magazine class file, it was the fourth model definition in that project. On the first couple of definitions Copilot did make good autocomplete suggestions right away as there must be a good collection of open source projects that have used the Sequelize library and Copilot had learned from them. In my experience the initial suggestions from Copilot were good but needed a little tweeking on some of the specifics, but by the time I had reached the Magazine definition the suggestions were spot on as it could use the previous 3 files as a reference to mold its existing understanding.

Writing tests

One area where Copilot pleasantly surprised me was when writing tests.

Typically when writing tests I try to write the descriptions in a way that can be read by any other developer (or my future self) and understand from the written description what that test is meant to be testing.

Some test descriptions might be:

• 'it should return a subtotal of zero if there are no items'
• 'it should return a subtotal of the sum of the unit price times the quantity of each item'

As you can see in the above examples, they are written in a plain English manner and are easy to read and this is an area that Copilot really excels at.

Copilot's Downsides

Creativity

The promise of ai in general is that it should be able to act creatively, but at this stage Copilot isn't quite there yet.

In my experience if there was a piece of code that required some truly creative thinking, Copilot wasn't really able to suggest anything that I would consider usable.

Verbosity is king

For the ai to glean information from the code, it helps to write fairly verbose comments and variable names.

This is a balancing act of giving Copilot enough context to suggest a good autocomplete suggestion and not make the code overly verbose.

Conclusion

In my short time using Copilot I have found it to be a great tool to aid the development experience.

In the big picture of a given software project, there tends to be 10-20% of the code that is the truly novel that requires creativity to solve a problem or provide a solution.

The other 80-90% of the code tends to be plumbing code that is utilizing a collection of libraries or services and providing some way to communicate it with an api of some kind. This type of code is where Copilot really excels at as the plumbing type of code tends to be more boilerplate driven or have quite repetitive patterns to enable to flow of data.

If Copilot can help to reduce the amount of time spent of the plumbing type of code, it will free up more time to focus on the creative side of a project which is an attractive proposition.

I'm excited to see how Copilot will evolve and how it can be used to help developers in the future.