The use of topic modeling to optimize the process of searching for relevant historical documents (on the example of the stock exchange press of the early 20th century)

Galushko Il'ya Nikolaevich Graduate Student, Historical Information Science Department, History Faculty, Lomonosov Moscow State University 119234, Russia, g. Moscow, ul. Lomonosovskii Prospekt, 27, korp.4 i.galushko15@gmail.com Other publications by this author

The key objective of the presented article is to test the methodology for analyzing the information potential of a collection of historical sources using thematic modeling. Some modern collections of digitized historical materials number tens of thousands of documents (as, for example, the "Electronic Library of Historical Documents" created by the Russian Historical Society (RIO) contains 294 thousand recognized historical documents ^[1] – and at the level of an individual researcher, it is difficult to cover all available heritage. Following a number of researchers ^[2], we assume that thematic modeling can become a convenient tool for preliminary assessment of the content of a collection of historical documents; a tool for selecting only those documents that contain information relevant to the research tasks set.

         Our research, for which the methodology described in the article was developed, is devoted to the study of the profitability of securities on the St. Petersburg Stock Exchange at the beginning of the XX century from the perspective of behavioral finance. We were interested in the principles of investment evaluation of public companies – how acceptable or insufficient levels of capitalization were determined; how securities were determined that were a good choice for placing capital, how widely these methods (if they existed) were used in the practice of exchange trading. The newspaper "Birzhevye Vedomosti" was chosen as one of the main sources, in the daily issues of which there was an exchange column, where the chronicler's comment was printed, which described the mood of the bidders and often provided a detailed analysis of the current situation in the economy of the Russian Empire. Analytical notes on the profitability of securities are often found in the columns of the "Exchange Statements": at what percentage the next issue of government debt securities is placed; at what level relative to the nominal value these securities are traded; whether the proposed percentage corresponds to the current statistics of the money market and how to explain the exchange rate dynamics of recent days. For the study, it was decided to collect a collection of such notes in order to identify stable analytical patterns characteristic of the stock exchange press in matters related to the profitability of securities. We used the materials of the digitized set of "Stock Exchange Statements" from the website of the Russian National Library (447 issues for 1905 and 1913, morning and evening issues). As part of a general study of the profitability of securities on the stock market of the Russian Empire, we were interested in a limited set of problems related to the behavioral and institutional aspects of the functioning of the stock market of the Russian Empire. On the contrary, the content of the "Exchange Statements" is diverse. And, except for the column of the stock chronicler, there are numbers that are completely devoid of the information we need. The content of such issues is filled with military news, theatrical and literary reviews, economic arguments of a non-exchange nature and other similar broad-profile articles (see Figures 1 and 2). And in this context, it is logical to turn to the possibilities of thematic modeling as an applied tool for automatic search of those numbers (pages) from our collection of digitized newspaper materials that contain information related to the functioning of the securities market.

Figure 1. An example of the organization of the page of the number of "Exchange statements".

Figure 2. Example of the text you are looking for. The "Lucky Girl" column. A story about how a certain woman found a bundle of shares in a store that banker Borisov had lost (Issue No. 13621 of June 28 (July 11)).

* * *

Thematic modeling is a method of machine learning without a teacher, used to determine the main topics of a collection of documents (or the topics of proposals of one document, which is considered in this case as a set of proposals) based on the allocation of topics (for the difference between the concepts of "topic" and "topic", see Appendix 1). As a rule, a topic is a probability-weighted list of words that together express the general content of the intended topic ^[2]. The higher the coefficient of a word, the more importance the model attaches to this word when forming a topic. So, Table 2 shows an example of two topics defined by our model for the third page of the issue of the "Stock Exchange Statements" dated April 29, 1913 (No. 13521):

Table 1. Examples of two topics defined by the model, represented by a set of keywords and their probabilities (see Appendix 2.1)

One of the most popular thematic modeling methods currently used is the latent Dirichlet distribution (LDA), which is a "generative probabilistic model for collections of discrete data, such as text corpora" ^[3]. This method is used within Digital Humanities to extract topics from a set of texts ^[4]. In this model, the document (in our case, a separate page of the newspaper issue of the Stock Exchange) is a mixture of topics, and the topic is a probability distribution according to the dictionary. A dictionary is understood as a list of all the words of the studied collection of documents – it is the dictionary that sets the word space in which the documents need to be distributed in such a way as to form the number of topics set by the researcher. An important feature of LDA is a predetermined number of topics, which is set before the start of training. A convenient function of LDA models is to set weights equal to 0.001 for words defining redundant topics. That is, if the researcher initially set the search for 5 topics for the model, and according to the results of thematic modeling for three topics, the model set different word weights varying in boundaries, for example, from 0.005 to 0.0021, and for the other two topics – the weight of 0.001 for all words – it means that LDA cannot distribute the text with the given settings more than 3 topics. In the course of our experiments, we found that a threshold of 5 topics is sufficient to solve our research problems.

The creation of a dictionary can be supplemented by the use of various methods of text preprocessing.  In our work, the lemmatization method, classic for computational linguistics, was used – bringing all words to the basic form. Another widely used method of preprocessing involves searching and adding to the dictionary of the bigram model – words with a double basis, distinguished in the text by the frequency of joint occurrence (for example, "joint-stock capital", "state rent"). We tested this approach initially when we planned to create an LDA model of the entire existing collection. With this approach, an entire issue of an exchange newspaper is considered as one document. However, the results turned out to be extremely impractical – due to the too wide spread of topics per issue of the newspaper, we were unable to create interpretable models. Adding filters (TF-IDF (see Appendix 1), changing the training parameters (for example, the size of the batch portion of data for the training iteration), increasing the number of topics (different options ranging from 5 to 30 topics) – could not improve the situation. Then we decided to create LDA models for individual issues of the newspaper, considering them as a collection of proposals. The results have improved, but still remain insufficient to use thematic modeling as a stable search algorithm. The last and most productive option turned out to be the construction of an LDA for each individual page in our collection of "Exchange Statements". This approach allowed us to create an algorithm for selecting materials relevant for research based on thematic modeling, but due to the limited vocabulary, we had to abandon the creation of bigrams and filtering by TF-IDF, which ultimately did not affect the quality of the final modeling, since we did not set the task of exhaustive semantic analysis; identification of topics based on single words it turned out to be sufficient to select all the issues of the newspaper that have information relevant to our research tasks. The Python module Gensim, which is one of the most popular NLP tool packages in the world of modern data science, was chosen as the main library for implementing the program code^[5].

Figure 3. Visualization of the LDA model created for the 3rd page of Issue No. 13521 of the Stock Exchange Gazette dated April 29, 1913.

Within the framework of thematic modeling, the expert himself must determine the name for the topic in such a way that it reflects the semantics of individual words forming the topic. For example, according to Figure 8, we can unequivocally state that the words from topic No. 1 clearly gravitate to the topic of urban loans, as indicated by words such as "urban", "loan", "bond", "capital". The architecture of existing LDA models allows you to quickly analyze the content of selected topics based on keyword search, taking into account their "weight". As an algorithm for searching for newspaper issues containing the information we need, we decided to use a search for words forming topics with a threshold value of word weights > 0.01. As semantic units marking the presence of stock information in the simulated document, we used the words listed in Table 1. A list of Fuzzy-match patterns and their corresponding word forms ^[6].

2411 LDA models were created for the entire selected collection of texts of the "Stock Market Statements". Of these, our search algorithm identified 457 in the group "containing stock information". Of course, to a large extent, these include the pages with the column of the stock chronicler. As a definite proof of the applicability of LDA models in such source-related problems, we note that the analysis of a random sample of 100 numbers of the "Stock Sheets" showed that not a single column of the chronicler was missed by the search algorithm – each of them was placed in a summary table. As an illustration, we will present a small sample from the fields of the final table, including only those pages and numbers in which it was possible to find stock information outside the column of the stock chronicler. There are 29 such models in total. Table 2 shows examples of sets of topics defined by the model. Table 3 contains a brief description of the materials found on the pages listed in Table 2. In the appendix we have placed photos of the pages presented in both tables.

Table 2. Examples of topics and keywords of LDA models for sampling numbers from the collection of "Stock Market Statements" for 1913.

Table 3. The contents of the selected examples with an index to Appendix 2 (corresponding scans of the pages of the "Exchange Statements").

***

The analysis of materials found using LDA models was included in the main content of our study devoted to the study of behavioral practices related to the analysis of securities profitability. Thus, an approach was implemented in which the substantive research tasks were solved by traditional historical methods, involving a detailed study of the source text and embedding a separate document into a single system with other available material – archival sources and writings of exchange practitioners ^[7].

At this stage, we can confirm that within the framework of our research, the use of thematic modeling turned out to be a productive solution for optimizing the process of searching for historical documents in a voluminous collection of digitized historical materials. At the same time, it should be emphasized that in our work thematic modeling was used exclusively as an applied tool to accelerate the search and initial assessment of the information potential of a collection of documents through the analysis of selected topics. Our experience has shown that, at least for "Stock Market Statements", thematic modeling using LDA from the Gensim library does not allow us to draw conclusions from the position of the methodology of content analysis used by us, which involves working with the internal content of the source at the level of analytical practices. The data of our models are too fragmentary, they can only be used for the primary assessment of the topics of the information contained in the source. Of course, we must take into account the possibility of further complication of models through the use of additive regularization ^[8] and the improvement of OCR recognition towards splitting the page into documents for each separate column, which can significantly increase the ability of the model to capture deeper semantics of newspaper text. Summing up, we would like to note the applied importance of continuing research on the practical applicability of thematic modeling in solving problems of source studies.

For interested readers, we leave a link to access the program code and a set of source data proposed by the author ^[9].

Appendix 1: Dictionary of Definitions

Thematic modeling is a technology of statistical analysis of texts for automatic identification of topics in large collections of documents. The thematic model determines which topics each document belongs to, and what words describe each topic. This does not require manual markup of texts, the model is trained without a teacher. This process can be compared to clustering, but thematic clustering is "soft" and allows the document to belong to several clusters-topics. Thematic modeling does not pretend to understand the meaning of the text, but it is able to answer the questions "what is this text about" or "what common themes do these texts have" ^[10].

Using thematic modeling, it is important to distinguish between topics and, in fact, topics. Topics are the result of statistical processing of a collection of documents and consist of words that, depending on the chosen statistical model (LDA, LSA, BertTopic, etc.), were assigned a certain significance, on the basis of which the model assumed that these topics (a cluster of significant words) form the topic of the document at the semantic level. However, it is important to emphasize that the procedure for recognizing that a certain topic really reflects the topic present in the text of the document is purely expert. The researcher determines how semantically relevant the set of topics formed by the model turned out to be to the text collection.

TF-IDF (from the English TF — term frequency, IDF — inverse document frequency) is a statistical measure used to assess the importance of a word in the context of a document that is part of a collection of documents or corpus. The weight of a certain word is proportional to the frequency of use of this word in the document and inversely proportional to the frequency of use of the word in all documents of the collection.

APPENDIX 2

 The pages of the newspaper "Birzhevye Vedomosti" identified by the automated search method for the contents of a set of LDA models

Figure 2. 1. Exchange statements. 1913, No. 13521 (April 29 (May 12)). Third page

 Figure 2. 2. Exchange statements. 1913, No. 13529 (May 3 (16)). Second page

Figure 2. 3. Stock exchange statements. 1913, No. 13543 (May 11 (24)). First page

Figure 2. 4. Stock exchange statements. 1913, No. 13553 (May 17 (30)). Second page

Figure 2. 5. Exchange statements. 1913, No. 13565 (May 25 (June 7)). Page seven

 Figure 2. 6. Exchange statements. 1913, No. 13587 (June 8 (21)). Second page

                  Figure 2. 7. Exchange statements. 1913, No. 13597 (June 14 (27)). Second page

           Figure 2. 8. Stock exchange statements. 1913, No. 13621 (June 28 (July 11)).  Third page

                      Figure 2. 9. Stock exchange statements. 1913, No. 13696 (12 (25) Aug.). 1 page