A bank in Mumbai signs up an AI vendor to flag suspicious transactions in real time. The vendor won’t explain how the model was trained; it calls that a trade secret. Months later, the model eventually and internally retrains itself and starts flagging a different set of transactions, and no one at the bank is told why. An insurer licensing an underwriting model, or a manufacturer buying a predictive-maintenance system, would recognise the same pattern but under different labels. None of this is hypothetical but rather, it is close to the ordinary experience of Indian enterprises buying AI tools today. It points to a governance problem that has gone largely unnoticed, because of how the contracts bringing them into the business are written in the first place and not because AI systems are misbehaving once deployed. (The “black box” of the title, in this context thus simply means a system whose internal workings the buyer cannot inspect or fully explain.)

India’s enterprises are well past the pilot stage. Deloitte’s State of AI in the Enterprise survey finds Indian organisations ahead of global peers in deploying AI at scale across business functions. EY’s AIdea of India survey tells a similar story about a shift “from pilots to performance,” with multiple AI use cases already running in production. Progress has been made on paper as well. The IndiaAI Mission is providing the compute and data infrastructure necessary for the adoption, and the India AI Governance Guidelines by MeitY, launched on 5 November 2025, have laid out a risk-proportionate approach based on soft law provisions from the IT Act and the Digital Personal Data Protection Act. While all of this may be useful, it provides guidelines on the way the AI systems need to operate after they have come into being. Little has been done on how such systems can be procured by businesses in the first place and similarly about the commercial transactions involved in bringing in these systems. It is this gap that this piece tries to fill.

Algorithmic accountability, data protection and liability have been quite well-researched, and there is a recent trend of literature, including Ben Dor and Coglianese piece, which views public procurement as a governance tool in itself. What is still lacking, however, is research in this area from the perspective of private enterprises through which banks, insurers, and manufacturers bring AI into their operations.

A Contract Problem Hiding as a Technology Problem

“Enterprise AI procurement” in this context only refers to enterprises purchasing or licensing technology products that incorporate AI features for their own use, whether it be a transaction monitoring tool of a bank, an underwriting algorithm by an insurance firm, or a predictive maintenance tool at a manufacturing plant. The majority of these transactions are done using regular software or SaaS contracts: service level agreements, limitation of liability clauses, and data processing schedules, which cater to stable and predictable products. Commentary on technology sourcing also confirms that this remains the case in India. However, such a contract model does not always suit AI tools for four main reasons.

The first is the asymmetry of information. There is typically no way for a buyer to know how the algorithm was trained and why the model fails in certain instances, particularly where the seller claims trade secret status on this very fact. As Scholarship on contracting for algorithmic accountability demonstrates, even a government purchaser with considerably more negotiating power and leverage than an ordinary company finds it difficult to compel this information from private vendors.

The second is the probabilistic nature of decision-support results. Conventional software responds identically when presented with identical inputs, whereas an AI tool does not. While the fraud detection algorithm can be quite accurate on average, it will inevitably fail to classify a few highly valuable transactions in unexpected ways. Given these circumstances, vendors cannot possibly warranty the results, and a statistic such as uptime offers little information about the quality of a model’s decision-making. Liability provisions applicable to deterministic software may not be entirely appropriate in this case. Indian commentary on AI vendor contracts already notes this gap in practice.

Third, model drift. AI products are continually re-trained and re-tuned post-production by their upstream dependencies (foundation models, cloud APIs), which neither the vendor nor the buyer entirely control. Consent, notice, or retesting does not naturally fit within the context of a contract focused on a singular, one-off deliverable, in light of the fact that the underlying model itself evolves on its own accord, often without any intentional change on anyone’s part.

Fourth, switching costs. AI products are heavily integrated into the buyer’s data processes, which means that switching to another vendor is an expensive process involving retraining, re-integration, and validation of the whole output pipeline. OECD research on AI in public procurement warns that, if left unaddressed, this can eventually lock buyers into a small pool of vendors and blunt competition. Enterprise buyers face the same risk governments do.

None of this means AI contracts are broadly defective, or that private negotiation has failed, but rather that the template most parties reach for first was built for a different kind of product than the one now being sold through it.

From Isolated Clauses to a Governance Question

Public-procurement scholars have already made a similar argument in the context of government contracting. In “Procurement as AI Governance”, Lavi Ben Dor and Cary Coglianese argue that because agencies depend on private vendors who invoke trade-secret protection, governments have to use the procurement stage itself to secure the transparency they will need later for oversight and accountability. Procurement contracts, on their account, become a form of AI “soft law.” A similar dynamic can show up in the enterprise context once many organisations run into the same four uncertainties across many separate deals. Whereas, at first glance, the challenge may appear to be a drafting issue for any one company, once this process repeats on a scale, with rising transaction costs and adoption being slowed down to the extent that becomes relevant to India’s broader economic strategy, regardless of the absence of a regulatory gap, this issue becomes a systemic problem.

This is important because procurement, where the main decisions are being made regarding who the vendor is, what risks will be shared, what audit rights are granted, and what will happen upon exit, is made way before deployment. Renegotiation of terms at a later stage proves to be more difficult, especially for small organizations that lack the requisite legal and technical muscle to undertake the due diligence of a large incumbent. In case every organization has to go through this process independently, the time involved increases and there will be no uniformity of procurement safeguards across all agreements. SMEs are most disadvantaged by such an approach, being essential to India’s AI adoption strategy.

Why Soft Law, Not a Statute

Considering that there is a very fast pace at which the use cases evolve, a prescriptive procurement law is bound to be premature and it does not seem to be the path that India has been pursuing so far. The MeitY guidelines as well as NITI Aayog’s Responsible AI for All report advocate an agile, principled and cooperative approach over a rules-based one, just as the OECD too prefers standards and guidance over regulation. Four non – coercive and non-prescriptive ways of doing the same for enterprise procurement:

1. Procurement Framework from MeitY: As convenor of the IndiaAI Mission, MeitY is well-placed to issue non-binding guidance on intended use, lifecycle, limitations, and audit rights. It would give common ground for buyers and sellers without prescribing contract terms.
2. Industry convergence on basic provisions: Law firms, sectoral organizations, and standards bodies could collaborate to develop standard language with respect to portability, updates, and auditing rights, like how commercial practice converged on SLAs and liability caps for regular software products. This would drive down costs of drafting for newcomers and SMEs without compelling anybody to follow suit.
3. 3.Voluntary disclosure template: NASSCOM and CII could draw upon the World Economic Forum’s AI Procurement in a Box project to prepare a common RFP schedule covering planned usage, high-level categories of training data, update practices, and limitations. This would help narrow down the information asymmetry mentioned above.
4. Sector-specific expectations regarding AI applications in areas of regulation: RBI, SEBI, and IRDAI have already begun interacting with algorithmic risks; RBI’s own FREE-AI Committee report serves as an example. Application of such principle-based standards for supervision in terms of vendor assessment and responsibility allocation in credit scoring, fraud detection, and trading surveillance would integrate procurement with sectoral risks, without compromising on MeitY’s cross-sectoral mandate.

Each of these complements the negotiations rather than displacing them. The idea is to reduce transaction costs and build confidence as more and more firms adopt AI, rather than pre-empting the negotiation-based contractual solutions that contract law does quite well already.

A Caveat

A sceptic might say private contracting is already adapting on its own. Sophisticated buyers negotiate bespoke AI clauses today, and market pressure may eventually standardise good practice without any coordinated intervention. That is a fair point, as far as it goes. Large banks and insurers with strong in-house legal teams may genuinely not need this kind of help. The four proposals above are aimed less at them than at the mid-sized enterprises and SMEs that India’s AI strategy needs to bring along, and who don’t have the leverage to extract the same disclosures on their own. Where sophisticated bargaining already works, nothing here is meant to displace it.

The skeptic can argue that the negotiation process for private contracts will evolve by itself. More sophisticated buyers will negotiate individual clauses in AI contracts and market incentives will eventually standardize the good practices even without any intervention. This is a valid point up to a certain extent. For sophisticated players with strong legal departments, such measures will be unnecessary. The four recommendations listed above are more likely intended for the medium and small enterprises who will form the bulk of India’s AI Strategy. These enterprises lack the power to extract such disclosures from vendors on their own.

Conclusion

The Indian debate on the governance of AI has been rightly preoccupied with how the AI systems operate; their safety, fairness, and adherence to the laws regarding data protection. This piece has made the case for a parallel concern with how the systems are procured. If AI is indeed becoming everyday business infrastructure, the governance of AI cannot be limited to that of the technology alone. It must extend to the legal frameworks governing the procurement process by which businesses get hold of, and integrate, such AI technology.

Author Bio

Arijit Gogoi is a third-year B.A. LL.B. (Hons.) student at Gujarat National Law University (GNLU), Gandhinagar. His academic interests lie in commercial arbitration, insolvency and bankruptcy law, and commercial dispute resolution. He has interned with Senior Advocates at the Delhi and Gauhati High Courts, where he worked on arbitration, insolvency, corporate, constitutional, and civil litigation matters, and has also served as a Judicial Intern at the District & Sessions Court, Jorhat. Arijit is a Student Researcher at the GNLU Centre for Alternative Dispute Resolution and serves as the Treasurer of the GNLU Department of Alumni Relations. He was also a Semi-Finalist at the 11th Symbiosis International Criminal Trial Advocacy Competition.