AI Coding assistants in the business (Part 1)

Introduction

Artificial Intelligence (AI) has become an increasingly important part of the software development process, with AI coding assistants like Github Copilot and Tabnine offering coders a more efficient and productive way to write code. Despite their growing popularity, there is a lack of published information and data available about the impact of these tools in a commercial environment. In this blog post, we aim to address this gap by exploring the current state of AI coding assistants and their practical applications in the workplace.

With so little information available on this topic, this blog post offers a valuable resource for anyone interested in understanding the impact of AI coding assistants on the software development industry.

Benefits for your business (as Marketed)

No more woodchopping

One of the main benefits of AI coding assistants is that they can take over repetitive tasks, allowing coders to focus on more complex and creative aspects of software development. For example, Github Copilot can suggest code snippets based on the developer’s past work, reducing the time and effort required to write similar code in the future. This leads to increased productivity, as coders are able to complete tasks faster and with less fatigue.

Better code completion

Another key benefit of AI coding assistants is that they can help coders work more efficiently and effectively. For example, Tabnine uses machine learning algorithms to provide coders with code suggestions that are context-aware and personalized, which can speed up the development process. This is particularly useful when working on large and complex projects, where it can be time-consuming to search for the right code snippets.

Best practices for everyone

Moreover, AI coding assistants can help improve code quality by providing suggestions that are optimized for performance and maintainability. For example, Github Copilot can suggest code snippets that follow best practices and are optimized for performance, reducing the risk of bugs and errors in the final product.

Drop me a few test cases!

AI coding assistants can also be useful for generating good test data. For example, Github Copilot can suggest test cases based on the code being written, reducing the time and effort required to manually create test data. This can lead to more comprehensive and reliable testing, as well as faster development times.

In summary, the marketed benefits of AI coding assistants include taking over repetitive tasks, increased productivity, faster development, and reduced fatigue. These tools have the potential to revolutionize the software development industry, providing coders with a more efficient and effective way to write code.

Possible drawbacks

While AI coding assistants offer many benefits, there are also potential drawbacks that should be considered.

Sending your code over the wire and privacy

One of the main concerns is the privacy and security of code snippets, as many AI coding assistants require sending code snippets to the cloud for processing. Not all clients may be comfortable sending snippets of their software to companies such as Microsoft for processing, but that really depends on the circumstances as you might already host your code in GitHub or Azure DevOps in which case your code is already with Microsoft. Tabnine is different in this sense as it is capable of running both in the cloud and locally.

Tabnine places a strong emphasis on privacy. When speaking with their salesperson, the first thing that was mentioned was their privacy-focused approach, which sets them apart from their competitor, Copilot.

On the other hand, Copilot’s current privacy policy states that they collect telemetry data and use code for training purposes even in their business plan. However, when asked about this, the Copilot salesperson stated that this is not correct as they do collect telemetry and use your code for training in their $10 individual plan, but not in their $19 business plan. The reason for the difference in pricing is the fact itself that the business plan does not collect or use customer data for telemetry and training purposes.

It should be noted that to use Copilot in a business setting, you need to be on a managed enterprise account with yearly payment. Alternatively, Copilot licenses can be purchased on a monthly rolling basis, based on usage, completely separately from an enterprise account and is not limited to your number of seats in GitHub enterprise.

Privacy Policies:

• Github Copilot for business – https://docs.github.com/en/site-policy/privacy-policies/github-copilot-for-business-privacy-statement
• Tabnine – https://www.tabnine.com/code-privacy

Juniors being mislead

Another drawback of AI coding assistants is that they may not be suitable for junior developers, as they may have a hard time identifying if a suggestion is incorrect or incomplete while still looking syntactically correct. This can lead to code that is not optimized for performance, is prone to bugs and errors, or does not follow best practices. Because of this code reviews are essential to be done correctly, so a senior developer can verify the code being produced this way.

Baseline survey

To determine the effectiveness of AI coding assistants in a commercial environment, we conducted a survey with two teams within our company. The purpose of the survey was to establish a baseline that we could use to compare improvements after deploying AI coding tools and measure their impact. The survey consisted of the following questions:

1. What coding tasks take up most of your work time?
2. What coding tasks are the most repetitive for you?
3. What coding tasks are the most boring for you?
4. What coding tasks frustrate you the most?
5. Is there any tool that could help you with your work that we don’t have a license for, but should?

The results of the survey provided a baseline of the current coding challenges and pain points faced by the teams. This information was then used to determine the impact of deploying AI coding tools and measure any improvements in efficiency, productivity, and job satisfaction.

Results of the survey

The results of the survey conducted showed that code analysis, tests, and build time take up most of the development time. However, the most repetition, boredom, and frustration came from topics around automated tests, according to almost all developers.

Deployment

Based on the results of the survey, we have decided to deploy both Github Copilot and Tabnine, each for two different teams on an internal project. This will allow us to measure the impact of AI coding assistants on the productivity and satisfaction of the teams both objectively, based on story points, and subjectively, based on their perception of productivity and satisfaction compared to the baseline research.

Objectively, we will measure the productivity of the teams by tracking the number of story points completed before and after the deployment of the AI coding tools. This will give us a clear picture of any improvements in efficiency and output.

Subjectively, we will conduct another survey after the deployment of the AI coding tools to gauge the perception of the teams regarding their productivity and satisfaction. This will provide insights into how the teams feel about their work after using the AI coding tools, and if they have seen any improvements in their daily tasks.

The results

The results will be published in the second follow-up part of this post.

Conclusion

We hope to determine the effectiveness of AI coding assistants in a commercial environment. This information will be valuable for other companies looking to improve their software development processes and increase efficiency and productivity.