Automation of software development using AI agents raises important governance questions, including:
- How do AI agents operationalise human values such as privacy and fairness in their processes and outputs? Human values are typically expressed using high-level principles that must be ’translated’ into practice in a given context. Given the significant challenge for humans in undertaking such ’translation’ in the design and evaluation of systems, it is unclear how (or even whether) AI agents could undertake and justify such ’translation’ with sufficient rigour for systems to be trustworthy with respect to these values
- Who is responsible for the process and outputs of these AI agents? AI agents can not only generate code, but also generate the code that tests the code. Humans move further away from not only the code itself, but the assurance of the code with respect to functional and non-functional properties. Additionally, as the scope of AI agent work increases, the volume and complexity of code (and tests) generated may exceed that for which one or more humans have the capability or capacity to comprehend and reasonably oversee. Individuals within organisations may be asked to assume legal and moral responsiblity for processes and outputs that they cannot or do not comprehend, and for which their participation in assurance processes may be limited
- How is the risk of supply chain attacks mitigated? AI agents may generate code that introduces dependencies on components from external package repositories. AI agents therefore require criteria on which specific repositories and packages within these repositories can be robustly evaluated to determine whether or not they should be trusted. In the absence of robust and rigorous evaluation processes, AI agents may introduce dependencies on components that are not trustworthy and place organisations at risk of supply chain attacks. Reputational and financial impacts of such attacks would negate some of the economic benefits on which use of AI agents for software development may be predicated
Read the Guardian article at: https://www.theguardian.com/technology/2026/may/07/your-craft-is-obsolete-wisetech-staff-in-limbo-as-ai-touted-as-better-than-humans