Evaluating GenAI on Forms Management: The Ongoing Need for Human Insight

The age of artificial intelligence (AI) has advanced rapidly with the release of tools like ChatGPT becoming available to the end user. These tools can compile information from various sources to answer questions on a wide range of topics. But how accurately can they comprehend truly complex disciplines that require years of specialized expertise?  

To test the knowledge base and discover the limitations of these tools in forms management, a series of questions were posed. The responses revealed gaps in the generative AI’s knowledge.

First, ChatGPT was asked to “describe the origin and history of forms management.” While it provided a reasonable timeline, a closer look revealed misconceptions about the scope of this discipline. For example, it focused only on forms development rather than the comprehensive management of forms over their lifespan.

Examples of these difference are below, the word or words at issue are noted in red italics while the human comments/corrections are noted in bold green.

In further review, beyond misunderstanding the subtleties of the field, Ray Killam, president of Essociates Group a forms management consulting company, identified three pivotal areas in the history of forms management not mentioned in the output. These areas are:

1. Products - Form products have evolved extensively from the days of large volume, multiple part continuous forms, to single-ply page printed forms, to desktop electronic forms, to internet forms, to apps.
2. Tools – The tools used have changed substantially from index cards and manual data entry to software and electronic forms.
3. Processes - The processes used to manage forms have not changed all that dramatically. We still use design standards, forms control standards, privacy regulations, management processes for regulated forms, etc. We just use different tools to accomplish these processes.

ChatGPT was then also asked to “describe the benefits to organizations from a well-run forms management program. Include statistics, such as cost savings, etc.” Overall, the generated response was able to successfully highlight many reasonable advantages. However, as noted with previously generated responses, the AI confused streamlining general business processes with optimizing forms management itself as well as provided unverified statistics.

Examples of the benefits highlighted in the response and the word or words at issue are noted in red italics while the human comments/corrections are noted in bold green.

In responding to questions on the specialty of forms management, the generative AI was able to compile informational snippets and to answer the questions it was asked. However, the responses required significant human oversight to review and correct inaccuracies and build upon missing details. The AI content, if taken as unequivocally accurate by the uninformed, is generally misleading and incomplete.

ChatGPT doesn’t really “get” forms management. For example, forms management is a business function, a discipline, a field of knowledge.  Equating the function of forms management with one of its sub-processes such as forms development or forms completion is just wrong. AI content, if taken as gospel by the uninformed, is misleading. 

The benefits to quickly access a wealth of information through AI’s web scraping capabilities shows immense promise, this review of forms management responses reveal that the accuracy of such information varies. At times, the content was wide of the mark (forms history) and sometimes more on target (benefits of a well-run forms management program. As the technology progresses, generative AI may someday provide accurate overviews of multifaceted disciplines such as forms management. For now, AI cannot replace specialized knowledge accrued over years of forms management experience.