Knowledge Graph Question Answering with Generative Language Models

Abstract

A Knowledge Graph (KG) is a data structure that stores information about the world in the form of nodes and edges. The nodes represent people, places, things etc., while the edges store the relationships between the nodes. The nodes are also known as entities, while the edges are known as relations or predicates. Several popular search engines today make use of such KGs in the background. Some well-known and freely available KGs are DBpedia and Wikidata.

One way to access information from a KG is through Question Answering. For example, web-based search engines today give people the ability to type their questions and receive answers. Unfortunately, the current state of search engines leaves much to be desired in the complexity of the questions that a user may type. Current search engines work best when the search term is a keyword or a set of words. Processing complete sentences, with complex logical rules, is still an open problem.

One large step in the direction of language understanding has been the arrival of pre-trained Language Models, such as BERT. Such models have been trained on large amounts of text corpus, and surprisingly, some variants of these models, such as T5 and BART, develop a remarkable ability to generate text, the likes of which are difficult to distinguish from that produced by a human author. These models are also called generative Language Models and are a central focus of this thesis.

Given a question by a user, how does one fetch an answer from the KG? This task is commonly known as Knowledge Graph Question Answering (KGQA). One of the techniques is to convert the user's question to a logical form, or a structured query. One popular query language the reader might be familiar with is SQL. SQL, though, is appropriate for relational databases. In the KG world, the analogue would be a language called SPARQL. The task of converting the natural language text to a logical form is known as semantic parsing.

To be able to execute a SPARQL query on a KG, the SPARQL schema must be valid, e.g., it must be syntactically correct, and it should be logically correct, e.g., one can not expect the correct answer if AND is replaced with an OR. The other requirement is that the constants in the query, such as entity and relation IDs, have to be placed in the correct manner in the query. In this thesis, we explore the abilities of generative Language Models (LMs) in the task of KGQA, with a focus on the semantic parsing approach.

This dissertation hypothesizes that generative LMs can be used effectively for the task of KGQA. We form two research questions over this hypothesis, and to answer these research questions, a series of five topically interconnected publications are presented in a cumulative fashion. We first test two popular generative LMs on the task of semantic parsing. We compare the performance of these models to their non-pre-trained predecessors. To this end, we utilize some well-known and openly available datasets, which address well-known KGs. Later, we develop methods to improve the default performance of such models on this task.

This thesis shows that while generative LMs are not ideal for the semantic parsing task in their default mode, special text-handling mechanisms can be incorporated into the model to improve their performance considerably. With these adaptations, the models produce state-of-the-art performance across five datasets that address four different KGs.

This thesis provides researchers in this field with a set of tools and techniques to work with generative LMs and adapt them appropriately to the task of KGQA. We show in subsequent chapters, that our findings have been used by contemporary researchers to further advance the state-of-the-art. In the end, we produce a new KGQA dataset, built over a smaller domain-specific KG. On this dataset, a challenge was organized in which seven participating teams tried the latest methods, and further pushed the state-of-the-art for this task. All the code and data used and developed in this thesis have been released as open-source.