Conversational agents based on Large Language Models (LLMs) such as ChatGPT can provide numerous benefits to multiple stakeholders inside and outside an organisation such as RMIT University. One of the primary benefits is that they can help different stakeholders in reducing the cost of tackling various tasks, saving time and resources (e.g., translate data into different formats, prepare draft of documents, etc.). Note that the aim is not to replace humans doing the tasks, but to assist them to be more cost-effective. In this setting, one of the major concerns is the risk of giving away sensitive data, compromising the privacy and security of individuals. This can make it harder for members of the organization to comply with data management policies, as they may be uncertain about what data is being collected by third-parties such as OpenAI.

We wanted to combine the diverse expertise we have within the Interaction, Technology, and Information discipline at the RMIT School of Computing Technologies (SCT) to demonstrate ourselves (and to learn on doing so) how far we can get in designing and deploying our in-house version of a conversational LLM. To make this long-term goal more reachable, we identified a more tangible milestone: can we deploy our own LLM to assist us to build a chatbot for RMIT Open Day? We considered a manually curated Frequently Asked Questions document from the School of Computing Technologies (SCT) as “sensitive data”, and we challenged ourselves to see how we can use our internally deployed LLM to create training phrases (and alternative paraphrases of the correct answers in the FAQ), to create Walert, a conversational agent that answers questions about SCT programs. We also experimented with Retrieval-Augmented Generation (RAG) to better understand how to evaluate and compare the effectiveness of different designs of conversational agents, i.e., intent-based vs. RAG.

On March 2024, Walert made it all the way to Sheffield, UK.

Our team members Sachin and Futoon presented Walert at CHIIR'24, the 2024 ACM SIGIR Conference on Human Information Interaction and Retrieval.

Walert was designed and developed in the unceded lands of the Woi Wurrung and Boon Wurrung peoples of the eastern Kulin Nation. We pay our respects to their Ancestors and Elders, past, present, and emerging. This research is partially supported by the Australian Research Council (DE200100064, CE200100005) and is undertaken with the assistance of computing resources from RACE (RMIT AWS Cloud Supercomputing). We thank Amina Hossain and Santha Sumanasekara for their valuable contributions.and the RMIT RACE Hub. We thank Amina Hossain and Santha Sumanasekara for their valuable contributions.

Walert was designed and developed in the unceded lands of the Woi Wurrung and Boon Wurrung peoples of the eastern Kulin Nation. We pay our respects to their Ancestors and Elders, past, present, and emerging. This research is partially supported by the Australian Research Council (DE200100064, CE200100005) and is undertaken with the assistance of computing resources from RACE (RMIT AWS Cloud Supercomputing). We thank Amina Hossain and Santha Sumanasekara for their valuable contributions.and the RMIT RACE Hub. We thank Amina Hossain and Santha Sumanasekara for their valuable contributions.