AI: Is More Really More? 

The Impact of AI Assisted Coding on Engineering Productivity  

It is hard to imagine a time not long ago where AI has not been front and centre of our everyday news, let alone in the software engineering world? The advent of LLMs coupled with the existing compute power catapulted the use of AI in our everyday lives and in particular so in the life of a software engineer. This article breaks down some of the use cases of AI in software engineering and suggests a path to investigate the key question: Did we actually become more productive?

The Meteoric Rise of AI Assisted Coding 

It has only been a few years since the inception of GitHub Copilot in 2021. Since then, AI assisted coding tools have had a significant impact on software engineering practices. As of 2024 it is estimated that 75% of developers use some kind of AI tool. Often these tools are not fully rolled out in organisation and used on the side. However, Gartner estimates that  we will reach 90% enterprise adoption by 2028. 

Today there are dozens of tools that do or claim can help software engineers in their daily lives. Besides GitHub Copilot, ChatGTP, and Google Gemini, common tools include GitLab Duo, Claude, Jetbrains AI, Cody, Bolt, Cursor, and AWS CodeWhisperer. New advances are reported almost daily leading to new and advanced solutions.

AI in Software Engineering: What is Changing? 

Looking at the use cases inside engineering organisation, we can identify a number of key purposes: 

1. Building proof of concepts and scaffolding quickly for new products. Engineers use AI-based solutions that leverage intrinsic knowledge about frameworks for generating initial blueprints and solutions. Solutions include Bolt, v0 and similar.
   
   
2. Writing new code and iterating existing code and using AI as a perceived productivity assistant. The purpose is to quickly iterate on existing solutions,  have an AI-supported “knowledge base” and assistance. This type of AI not only produces code, but to a degree replaces expert knowledge forums and  sites such as Stack Overflow. This is the space where we have seen the most success with solutions being embedded on the IDE, connected to repositories and tightly integrated into the software development process. 
   ‍

3. Automating engineering processes through Agentic AI. The latest approach is increasing the level of automation on niche tasks as well as connecting across tasks and development silos. Besides automating more mundane tasks, Agentic AI shapes up to be helpful in creating test cases, optimizing build pipelines and managing the whole planning-to-release process and is an area of much ongoing development. 

For the purpose of this article let us focus on the most mature technology, AI assisted coding solutions. Besides all the progress and the increasing adoption of AI, the main question remains: 

Are we any more productive?

Productivity means getting done what needs to be done with a particular benefit in mind. Producing more code can be a step in the right direction, but it might also have unintended consequences of producing low-quality code, code that works, but does not meet the intention, or where junior developers might blindly accept code leading to issues down the road. 

Obviously, a lot depends on the skill of the prompt engineer (asking the right question), the ability to iterate on the AI generated code (the expertise and experience of the developer) and of course on the maturity of the AI technology. 

Let us dive into the productivity aspect in more detail.

‍

AI and Productivity: The Big Unknown 

One of the key questions in rolling out AI tools across the engineering organisation is judging its productivity impact. How do we know if and when AI assisted coding really helps our organisation to be more productive? What are good indicators and what might be good metrics to measure and track productivity over time? 

Firstly, as mentioned above, productivity does not mean simply writing more code. More code is just more code. It does not mean it necessarily does anything useful or adds something to a product that is actually needed. Nonetheless, more code produced quickly is helpful if it solves a business problem. Internal indicators for this can be that feature tickets get resolved quicker, code reviews are accepted (quickly) and security and quality criteria are met. Either through higher pre-release pass rates, or lower incidence tickets post release. 

As such, some common indicators for productivity are 

• The throughput of your accepted coding activities as for instance defined by the number of PRs you get approved and merged in a week. 
• The number of feature tickets or tasks that can be resolved in a sprint, for instance measured by the number of planning tickets you can complete. 
• The quality and security standard of your coding activities. For instance, does AI coding assistance generate less security issues, do quality tests pass more often, or do code reviews take less time and less cycles?
• The time it takes to get any of the above done and a release out of the door. Do you release more often? Are your release pipelines more reliable? 

All things being equal, in a productive AI assisted coding organisation we would expect that you would be able to ship more or ship faster – ideally both. 

ROI:  Measuring the Impact of AI 

The best time to measure your engineering productivity is today. Productivity is never a single number and the trend is important.  Having a baseline to measure the current state against future organisational and process improvements is crucial to evaluate to gauge any productivity gains.  

If you haven’t invested heavenly into AI tooling yet but planning to, it is a good time to establish a baseline. If you have invested in AI, it is essential to track ongoing changes over time. You can do this with manual investigation at certain points in time, or automatically and continuously with software engineering intelligence platforms such as Logilica, which not only track your ongoing metrics, but also enable you to forensically look into the past and project future states. 

There are a number of key metrics we suggest tracking and see if your AI investment pays off. We suggest centring them around the following aspects: 
‍

• Speed of delivery. Are you able to deliver faster than before? This means, are you able to respond to customer needs and market demand quicker and more flexibly? Indicators are your release cadence, your lead time for releases, lead time for customer and planning tickets and even cycle times for each individual code activities (PRs). 
  ‍

• Features shipped. Are you able to actually ship more? Not producing more code only, but finishing more planned tasks, approving and merging code activity (PRs), and are you able to have more or larger releases? Throughput metrics are important if they are balanced with time and quality metrics. 
  ‍

• Level of predictability. One main challenge with software engineering is having on-target delivery and not letting deadlines or scope slip. Do your AI initiatives help you with this? For instance, do you hit the target dates more reliably? Does your sprint planning improve, and conversely are you able to reduce your sprint overruns? Does your effort more reliably align with the business expectation, e.g., do you track if new features increase and bug fixing/technical debt decreases?
  ‍
  
• Security/quality expectations. Does your downstream release pipeline improve with less build failures? Do you hit your testing and security scanning criteria? Do you see less support tickets since the introduction of AI? Is there a change in user sentiment that supports your ongoing investment?
  
  
• Developer team health. Lastly, does the introduction of AI positively impact your developer team health, lead to less overload and to happier teams? This is a big one and much less clear cut than one might expect. While AI assisted development can produce more code quicker, it is unclear if it does not create more burden elsewhere. For instance, more code means more code reviews, easily making humans a bottle neck again, which leads to frustration and burn out. Also, AI generated code might be larger, leading to larger PR where the actual submitter has less confidence in his own AI-assisted code. QA/security might feel the extra burden and customers report more bugs that take longer to resolve.

Overall, it is essential to track engineering processes and key metrics from multiple dimensions at the same time for ensuring that your AI investment actually delivers positive, measurable productivity gains. 

‍

‍

Conclusion 

AI assisted development has arrived. It is a new reality that will rapidly permeate all parts of the software development lifecycle. As such, it is critical to build up the expertise and strategies to use that technology in the most beneficial way. Ensuring success requires the right level of visibility into the software engineering processes to provide the essential observability for decision makers. Those decisions are two-fold: Justifying the investment to executive teams with data-driven evidence, and being able to set the right guardrails for productivity improvements and process goals. 

There is the inevitable hype cycle around AI assisted coding. To look beyond the hype it is important to measure the positive impact and steer its adoptions into the right direction, to ensure a positive business impact. 

Software Engineering Intelligence platforms connect with your engineering data and give you the visibility into your processes and bottlenecks to get answers to the above questions. These platforms automate the measuring and analytics process for you to focus on the data-driven division making.  

In future parts of this series we will dive into details of how predictive models can be applied to your engineering processes, how you can use AI to monitor your software engineering AI and how Software Engineering Intelligence platforms can help you to build high-performance engineering organisations. 

Want to learn more?

You can register now for our free AI meets Engineering Productivity webinar.

‍

About Logilica

Logilica is a leading software engineering intelligence platform. With insights across the software lifecycle including Git, Jira and CI/CD, Logilica makes it easy to empower data-driven software management. Logilica gives unrivalled visibility to your velocity, engineering bottlenecks and team health risks. Move from gut-feel to data transparency with our value stream analytics solution. Loved by engineering and platform teams who need to move fast and deliver predictably.

Ship faster with higher confidence by increasing engineering efficiency.

Book a demo | Contact Us

‍