Our proposed work uses the European Union Railway Agency (ERA) KG governed by the European Union Railway Agency as a Knowledge Base (KB). As part of ERA KG, which consists of many ontologies and graphs, we have used public data for the European Register of Infrastructure (RINF) ontology as a target KB. RINF facilitates access to all national railway infrastructure data and information records, covers and designates all concepts, classes, instances, and relationships related to the data of national registers and all vehicles that facilitate infrastructure (Rojas et al, 2021). Our experiment employed 20 NL competency questions formulated based on RINF KG to test targeted LLMs capability to answer diverse question complexity illustrated by simple queries: single triple, complex queries: multiple triples amd/or aggregation and filtering functions. Table 1 illustrates a sample of NL questions used in the experiment with their corresponding use justifications. The ‘No.’ column identifies the NL question sequence, the ‘NL Question’ corresponds to the actual input text of the NL question, and finally the ‘Justification’ explains the reason behind using each question. A complete list of NL competency questions including their corresponding SPARQL query translation can be found at the GitHub repository.

To investigate the capabilities of LLM to generate SPARQL queries and retrieve information from a domain-specific KG such as RINF based on the NL question, we conducted an experiment using two scenarios with three different prompt structures against two different LLM models without any prior NL processing and in an unadorned fashion. The main difference between the two scenarios is that scenario 1 uses only lists of ontology vocabulary extracted from the reduced graph, while scenario 2 uses a reduced ontology graph serialized in turtle format. As shown in Fig. 1, the intention of the first scenario (Scenario 1) is to provide LLM with the necessary vocabulary information as a context to explore the degree to which LLM can answer the NL questions. The vocabulary information is composed of a list of classes, instances, and relations extracted from the RINF ontology, as described in Section 3.4. This scenario is mainly oriented to be used with LLMs that have a limited or relatively small token size window such as GPT-3.5. The second scenario uses a reduced graph version of the RINF ontology graph serialized in RDF turtle format to overcome the limitation of the LLM context window size and conform to the Gemini specification. Finally, the results between the two are compared in terms of the generated SPARQL correctness.

Fig. 1.

Proposed model design and evaluation Scenarios. RINF, Registers of Infrastructure; ERA, European Union Agency for Railways; SPARQL, SPARQL Protocol and RDF Query Language; RDF, Resource Description Framework.

Initially, our reduced graph generation starts by creating an empty graph. Parsing the entire ERA Vocabulary ontology from the RDF. The graph content will be used later to fetch triple information for our ontology vocabulary information list and reduced graph. Since LLMs are constrained by the size of the context window, reducing a graph becomes a crucial step. To create a reduced graph, as illustrated in Fig. 2, we initially retrieve all classes related to the RINF ontology from the ERA endpoint using the SPARQL query shown below in real time and save their corresponding URIs in a list called relevant classes. This will return 21 URIs classes that RINF employ. Obtaining the ontology information directly from the endpoint allows our model to count for any changes or updates that may occur on the KG.

Fig. 2.

Reduced RINF graph generation process.

select distinct ?class FROM ¡http://data.europa.eu/949/graph/rinf¿ where ?subj a ?class .

Then, we retrieve all properties used by entities using the SPARQL query below and save their corresponding URI in a list called relevant properties. The process will return 226 predicates URI from RINF.

select distinct ?pred FROM ¡http://data.europa.eu/949/graph/rinf¿ where ?subj ?pred ?obj .

The URI contents of the two lists will be used to generate the reduced graph. The results retrieved from both queries are in JavaScript Object Notation (JSON) format and contain URIs only. To create our reduced graph and populate it with triples corresponding to extracted URIs, we first extract the complete URIs from JSON list, for example, the URI (http://data.europa.eu/949/OperationalPoint) followed by extracting the last fragment of the URI which represents a resource name or a class, in this particular case is OperationalPoint class. To create our reduced RINF graph, we loop through all extracted fragments and retrieve their corresponding triples from the original ERA ontology.

To generate vocabulary information for our reduced RINF graph intended to be injected into our LLM prompt, three empty sets are created (main class, properties, and instance of class). A Python Set data structure is used instead of a list to prevent duplicated triples from being appended.

A method is created that iterates through all triples (s, p, o) in our RINF reduced ontology to extract classes, instance of a class and properties. The subject part can represent one of two concepts, a class or an instance of a class. The separation between the two concepts is based on the properties that they use within the triple. If the property is of RDF type and the object resource does not represent a type of rdfs class, then it can be concluded that the subject is an instance of a class and the object is a main ontology class, subsequently added to a list called instances of class. Otherwise, if the property is not of RDF type, then it is appended to the properties list. Finally, three sets are generated that represent the vocabulary of our reduced RINF graph and can be found in the GitHub repository.

Prompt engineering is a delicate step in calibrating LLM to generate relevant responses that align with the intended tasks (White et al, 2023; Marvin et al, 2023), in our case translating NL questions into SPARQL queries. Generally, our proposed LLM prompts consist mainly of three parts: LLM orientated instructions part, the original NL question part supplied by the user, and the knowledge base part. The prompts labelled P1, P2, and P3 as illustrated in Figs. 3,4,5, respectively, were formulated after a detailed analysis of the requirements of the task and outcomes format required ensuring the inclusion of all elements that could efficiently guide the LLM to produce the desired responses and enforce rules in terms of context and format. More specifically, to effectively guide the model and enforce rules that ensure specific qualities of the generated responses, we have experimented with various prompt contexts by altering the phrasing of the LLM instruction parts while maintaining the NL question and knowledge base unchanged, which allowed us to observe the impact of LLM orientated instructions modification on the outcome rather than the effect of the knowledge base information. It was very clear that the complexity and expressivity of the LLM oriented instruction part of the prompt is directly proportional to the accuracy of the generated results, which can be tuned depending on the content and format of the required results (He et al, 2024; White et al, 2023). Consequently, each prompt was evaluated according to its ability to produce accurate and consistent responses. The optimal prompt was then selected based on several criteria, such as task alignment, completeness of retrieved responses, and consistency of SPARQL over multiple executions.

Fig. 3.

P1, LLM prompt structure with ontology vocabulary information.

Fig. 4.

P2, Detailed LLM prompt structure with reduced RINF graph.

Fig. 5.

P3, An Abbreviated LLM prompt structure with reduced RINF graph.

In our proposed approach, three versions of prompts (P1, P2, and P3) are used against two LLM models. In addition to providing the NL question, knowledge information is passed in two different forms: as discrete ontology vocabulary lists or as a reduced graph serialized as RDF format. To be more specific, P1 as shown in Fig. 3, is used to provide LLM with only extracted discrete vocabulary information using three lists (main class, properties, and instance of class) generated as explained in Section 3.4, each of which contains vocabulary extracted from the RINF ontology KG, while P2 and P3 use the reduced RINF graph serialized in RDF generated as described in Sections 3.3. The main difference between the two is that P2 and P3 are more expressive than P1 in terms of knowledge utilization and semantics, which allowed us to compare LLM ability to translate NL to SPARQL between the two. The NL question is passed to the LLM without any pre-processing, as our intention is to evaluate LLM responses over different knowledge forms.

Even though the augmented KG information within the prompt significantly impact the correctness and the semantic accuracy of the generated SPARQL query, it is important to note that employing specific explicit directives within the prompt can influence structural consistency and correctness of the generated SPARQL queries. For example, the directive specified in P1: “Only use classes, properties and instances URIs defined in the ontology_information provided” reduced LLM hallucination of the instances and properties and forced LLM to map terms in the NL question with the correct vocabulary, eliminating this directive, especially with our proposed vocabulary method, can lead to incorrect mapping of instances and properties. On the other hand, during experiments and by analysing the LLM SPARQL responses, we have noticed inconsistent use and sometime incorrect use of property and entity URIs mainly due to incorrect use of prefixes. Moreover, the LLM SPARQL responses in terms of binding prefixes to URIs were inconsistent over multiple runs for the same NL question. As shown in P1, P2, and P3, including explicit namespaces or PREFIX directives such as “Always use namespaces prefix bindings” and “Always include prefixes and namespaces used in the RDF in your response” can contributed to the consistency of URIs used within the generated SPARQL query over multiple runs for the same NL question, eventually eliminate incorrect URIs prefix binding inconsistency and improve human readability. Another noticeable source of error is that LLM’s tend to unnecessarily complicate SPARQL queries with modifiers, this can be mitigated by instructing the LLM with the directive “Do not use any modifiers unless it is instructed” to force LLM not to use any SPARQL modifiers unless it is explicitly stated in the original input question. This directive has contributed to the simplicity and consistency of the generated SPARQL and reduced SPARQL structural errors.

As illustrated in Fig. 1, once the prompts are generated as shown in the previous section, it is passed to the LLM API to generate the SPARQL query. The generated SPARQL is then passed to the ERA SPARQL endpoint to retrieve the answers. Minor text processing can be applied to eliminate spaces, carriage returns, and escape characters in the generated responses produced by certain LLMs such as Gemini.

Two different LLM models were implemented, Gemini (gemini-pro) from Google9, and GPT-3.5 (GPT-3.5 Turbo) from OpenAI. Gemini is used for its considerable input token capacity and its superior performance metrics in code generation relative to GPT-3.5 (Hou and Ji, 2025; Islam and Ahmed, 2024). Gemini has an input token limit of 30,720 tokens, making it a perfect choice to pass our reduced RINF Graph as RDF file. GPT 3.5 is used due to its wide acceptance and popularity in the research community during the time this work was conducted. We utilized the API provided by Google and OpenAI to pass prompts and retrieve responses.

All prompts, including its supplementary information, are passed as text to LLM. Throughout the experiments, LLMs temperature was retained at 0.4 to maintain consistency across all results. It is important to recognize that reducing the temperature leads to a decrease in randomness and diversity and produces more deterministic responses that reflect our use case requirements of being accurate rather than creative (Tufek et al, 2025; Fakhoury et al, 2023). Our model is implemented using Python (version 3.12.1., Python Software Foundation, Wilmington, DE, USA).

In general, our experiments test the ability of LLM to translate NL questions into SPARQL queries based only on the content passed from the domain-specific KG information. As illustrated in Section 3.1, our experiment employed 20 NL competency questions formulated based on RINF KG. Moreover, all NL questions were passed as text without any kind of Natural Language Processing (NLP), allowing us to examine the impact of different KG data arrangements on LLM responses. The correctness and evaluation of the returned SPARQL queries were manually evaluated and checked by taking the LLM responses and executing it against the complete original KG through the ERA endpoint. The results of the experiment are summarized in Table 2. As described in Section 3.5, P1 refers to the prompt template that only incorporates discrete vocabulary data derived from the minimized RINF graph; recall Section 3.4. In contrast, P2 and P3 prompts employ a fully reduced RINF graph extracted as described in Section 3.3. These three prompts were passed to two distinct LLMs (GPT-3.5 and Gemini) to retrieve their corresponding SPARQL queries.

A quick analysis on the results of Table 2, GPT 3.5 turbo has consistently achieved 10 correct responses out of 20 achieving a precision of 50% for each prompt (P1, P2, and P3). In contrast, Gemini achieved 17 correct answers generated from P1 and P3, and 15 correct answers against P2, achieving an accuracy of 85% for P1 and P3, and 75% for P2. consequently, in terms of overall performance and based on the statistics above, it can be concluded that Gemini outperformed GPT 3.5 turbo in all scenarios.

Furthermore, a careful analysis and contrasting of the two LLM models responses corresponding to P1 (Col1 and Col6) with the responses of P2 and P3 (Col2, Col3, Col4 and Col5), it is evident that GPT-3.5 Turbo exhibits consistent performance across all types of prompt, indicating that augmenting different arrangements of KG information (vocabulary vs. RDF triples) does not have a noticeable effect. In contrast, Gemini exhibits high performance with both types of KG data arrangements but with a slight dip in accuracy for P2. This suggests that although Gemini is effective with both structured RDF triples and vocabulary, it is evident that LLM augmented with vocabulary information can provide the same level of accuracy compared to RDF triples. Hence, it can be concluded that the integration of vocabulary information achieves a competitive accuracy to that observed with P2 and P3 using GPT 3.5 and Gemini. This is a very interesting insight, because the word count is much smaller when comparing the vocabulary context size with the reduced graph size. This gives a clear indication that LLM can select correct entities and relationships and generate correct SPARQL quires based on discrete vocabulary information provided and not fully dependent on the semantics that can be deduced from a reduced KG triples at least on questions that do not require logical reasoning over multiple triple paths, which we believe that LLM need to do reason over triple paths within the KG to answer such complex questions.

A careful review of the responses generated using Gemini indicates that the responses were much more focused and detailed in terms of outcome representation than those obtained with GPT-3.5 Turbo. More specifically, for example, Q1 corresponding to the NL question “Please give me a list of all operational points.” has been answered correctly by both models using all three prompts. Although the responses of P1, P2, and P3 produced correct answers by returning the answers as URIs, Gemini generated SPARQL queries in which its responses (corresponding to P2 and P3) are more human-readable in terms of the presentation of the returned answer by returning only the labels of the “operational point” rather than the URIs which are more easily interpreted and understood by users. This indicates that Gemini has a greater ability to understand the semantics of the question and to produce more human-readable responses based on the question rather than returning only the URIs, which sometimes need to be investigated to get the required answer.

On the Question-Level Performance side, it is obvious that both models have accurately answered simple lookup questions (e.g., Q1 through Q5) across all proposed prompts. With more complex questions that involves filtering, aggregation, or multiple triples (e.g., Q6 to Q20), Gemini maintained higher accuracy, while GPT-3.5’s performance dropped significantly. Other questions (e.g., Q9, Q11, Q12, Q18, and, Q20) were challenging for both models, indicating difficulties with complex query structures or specific domain vocabulary.

The outstanding performance of Gemini compared to GPT-3.5 in translating NL to SPARQL can be owed to several reasons including architectural design as well as training advancements. Recent studies showed Gemini model features a large context window size up to 1 million tokens (Team et al, 2024), whereas GPT-3.5 is only limited to 16K tokens (Brown et al, 2020). The extended capacity allowed Gemini to be augmented with extensive vocabulary and ontology information within the prompt, providing more contextual accurate SPARQL translation. Moreover, according to (Li et al, 2024), Gemini demonstrated stronger ability to synthesize syntactically correct and semantically meaningful code which in return contributed to improve the performance on task related to code generation which is closely aligned with the NL to SPARQL translation task. The aforementioned factors collectively can justify to some extent the outstanding performance of Gemini over GPT-3.5 in the domain of NL to SPARQL translation.

In a more general prospective, our experiments show competent performance when augmenting KG vocabulary information rather than reduced KG of RDF triples, which makes it suitable for large domain-specific KGs, given that: (1) the vocabulary approach prevents instance and relation hallucinations during the LLM inference process; (2) our model accounts for any updates as it generates vocabulary information directly from the KG endpoint; (3) reducing the size of tokens passed to the LLM which gives more usable space to include more information in the future that may assess the LLM model to perform better.

A detailed look at the SPARQL responses in all scenarios shows that the main source of error is due to incorrect use or inference of properties, instances and missing prefixes. By further analysing the generated SPARQL responses, errors presented can be summarized into the following categories:

1. Mismanagement and misuse of prefixes: Several queries either missed required prefixes or defined incorrectly, for instance, the namespace for ns2 (http://data.europa.eu/949/) was inconsistently included or entirely missing, especially in GPT-3.5 prompt variants, this can be prevented by giving an explicit instruction to include prefixes in the prompt to as shown in Fig. 6.

Fig. 6.

Prompt namespace directives.

2. Misuse or incorrect inference of properties: Some queries used generic Resource Description Framework/Web Ontology Language (RDF/OWL) properties such as rdf:type, or rdfs:label instead of targeted domain-specific properties, for example ns2:operationalPoint and ns2:trackGauge. In more complex questions such as Q4 and Q5, wrong properties were inferred by the LLM resulting in a syntactically correct but semantically incorrect queries.

3. Query Structure Issues: in some responses that requires nested SPARQL query, the structure of the SPARQL query were mishandled. For example, the questions involving numeric comparisons (e.g., maxSpeed properties) led to misuse of FILTER clauses or incomplete OPTIONAL blocks.

4. Vocabulary Hallucinations: There were several instances where incorrect vocabularies were introduced, indicating confusion in entity/property mapping due to LLM hallucinations, this can be mitigated if more information is augmented thorough prompts.

Most of these issues can be bypassed either by giving a detailed prompt instructions and directives to include prefixes as in P2 and P3, clearly illustrated in Fig. 6 or by using NLP methods.