AI Text Analysis: From Experiment to Structured Data

A few years ago, I started a fairly ambitious project: building a database of words and multi-word terms that could be used to analyse texts automatically. The idea was to gather enough syntactic, lexical and semantic information to dissect a text and, from that data, study its structure, style and content.

Using traditional techniques, I managed to get quite far. From the dictionary of the Royal Spanish Academy, I could extract words, meanings, classifications, idioms and usage examples. It was also possible to generate plural and feminine forms automatically, conjugate verbs, and incorporate other sources for given names and surnames.

The problem appeared when I tried to use all that information to analyse real texts.

An isolated word is relatively easy to classify. A complete sentence is not. The meaning and function of each element depend on the others, multi-word expressions appear, syntactic structures overlap, ambiguities arise, and many relationships are extremely difficult to resolve using conventional algorithms.

Large language models change the situation considerably.

Instead of trying to build an enormous collection of rules capable of handling every case, we can use AI only in the steps where language interpretation is really needed, while the application takes care of everything else: dividing the process, storing results, controlling context, running tasks in parallel and validating the information obtained.

The experiment

I have built a first text-analysis workflow divided into several steps.

The text is first split into sentences. Each sentence is then divided into individual elements — words, numbers and punctuation marks — while groups of several words that may have an identity of their own, such as compound names, idioms or syntactic phrases, are also identified.

From there, different syntactic and secondary classifications are applied to both words and multi-word terms. Finally, the model can add free-form semantic, thematic and emotional tags.

The result is not simply an answer generated by an AI model. It is a collection of structured data stored in a database and linked to the positions occupied by each element in the original document.

This means that many different questions can later be asked about the text without having to analyse it again from scratch.

A first test, not the final result

These initial trials have mainly been useful for building and refining the process.

Some tasks have proved extremely reliable, such as sentence segmentation or tokenisation. Others, especially the identification and classification of multi-word units, are much more difficult and will require further experimentation.

That is precisely one of the interesting parts of the project. The goal is not simply to ask a model to “analyse a text”, but to determine which tasks it can perform well, how they should be divided, what information the model needs in each step, and which results are worth preserving.

The process also records every model operation: the context used, function calls, results, errors, token consumption and estimated cost. This means the workflow itself can also be analysed.

In the accompanying video, I explain in much more detail how the experiment is structured, the different steps involved, and the results obtained from four separate trials. I also show how context management, caching, output tokens, model selection and apparently minor design decisions can significantly affect the process.

This is only the starting point. Once we have sufficiently reliable structured data describing the elements that make up a text, the next step becomes much more interesting: finding out what we can discover from it.