Zalo AI announces free Vietnamese proficiency assessment standards

VMLU (Vietnamese Multitask Language Understanding) was developed by Zalo AI in collaboration with the Japan Advanced Institute of Science and Technology (JAIST) to help the AI research and development community in Vietnam have more tools to evaluate the output quality of Vietnamese Generative AI models.

Accordingly, this is a multi-faceted, multi-level Vietnamese assessment standard set that meets the most diverse needs on the market today with 10,880 multiple-choice questions revolving around 58 different topics.

zalo ai.jpg — The Vietnamese proficiency assessment standard set was researched and developed by Zalo AI engineers in collaboration with JAIST to assess the ability to understand and apply Vietnamese language of AI models, especially Generative AI.

Why does AI Vietnam need a complete set of Vietnamese language proficiency assessment standards?

The explosive growth of ChatGPT has created a new race: Generative AI. According to statistics, since the introduction of ChatGPT, there are currently about 16,000 models similar to ChatGPT in the world.

Vietnam is not out of that trend as there are many research groups with different scales and potentials who are also wanting to experiment with Generative AI using Vietnamese. This has led to the need for a Vietnamese proficiency assessment set for these AI models to measure the level of knowledge and thinking in Vietnamese.

In the current market, most LLM research groups in Vietnam have to build their own evaluation toolkits with their own standards for their models. These are internal evaluation tools that have not been made public. Zalo AI's evaluation toolkit is aimed at general needs, can be a common standard for LLM models and is provided to the AI community. This helps small research groups access comprehensive evaluation data sets and allows parties to compare results with each other. From there, it creates motivation to improve the model.

Motivating Vietnamese AI to join the world's Generative AI wave

In November 2023, Zalo AI officially announced the VMLU Vietnamese language proficiency assessment standards. This is a set of standards researched and developed by Zalo AI engineers in collaboration with the JAIST Institute to evaluate the ability to understand and apply the Vietnamese language of AI models, especially Generative AI.

The birth of VMLU has motivated individuals, startups or small research groups to develop new Vietnamese AI models. This creates conditions for new research, lays the foundation for measuring the accuracy and upgrading the results of basic models, helping to complete the development process of Vietnamese language AI applications, created by Vietnamese people to serve Vietnamese people.

This is also one of the important factors promoting the development of Generative AI in Vietnam to go faster, catching up with the wave of AI development in the world.

What are the Vietnamese language proficiency assessment standards?

Accordingly, this is a set of multi-faceted, multi-level Vietnamese language assessment standards that meets the most diverse needs in the Vietnamese Generative AI research and development market, focusing on two main parts: Data (test dataset) and a set of assessment standards, as a basis for testing AI models applying Vietnamese language.

Specifically, the dataset includes 10,880 multiple-choice questions revolving around 58 different topics. Each topic has about 200 questions and is distributed across 4 fields including: STEM, Social Sciences, Humanities and a broad category "Expanded". With this dataset, VMLU has a difficulty stratification with 4 levels: Primary, Secondary, High School and Vocational - for university and postgraduate. From there, the toolkit helps to effectively evaluate the Vietnamese language proficiency of AI models in both basic knowledge and solving complex problems.

In addition, to help research groups easily evaluate the capabilities of their Vietnamese AI models, the Zalo AI engineering team has designed instructions so that groups can use them quickly and simply.

Contribute to the Vietnamese AI community

The VMLU standard set was born as a research product aimed at contributing to and developing the Vietnamese AI research community in particular and the information technology community in general, without charging any users, research groups or businesses.

Previously, Zalo AI has implemented, deployed and organized a series of competitions and programs for the Vietnamese AI community such as: Zalo AI Challenge, Zalo AI Hackathon, Zalo AI Summit... These activities not only create a playground for the Vietnamese AI community but also encourage the application of AI in life, solve urgent social problems, and serve the needs of millions of Vietnamese people.

Dr. Chau Thanh Duc, Head of Zalo AI Research Department - Lecturer at the University of Natural Sciences, Ho Chi Minh City National University affirmed: “Zalo AI always aims to contribute to the Vietnamese AI community, creating motivation for Vietnamese AI to develop. From there, we expect more and more AI products by Vietnamese people, for Vietnamese people”.

Source