Evaluating Natural Language Understanding of LLMs with Turkish General Knowledge
This project aims to evaluate the language understanding capabilities of large language models (LLMs) for Turkish common sense. The project involves creating a common sense dataset for Turkish by selecting entities covering various topics such as history, art, and sports. It includes collecting relevant texts and generating a question-answering dataset, primarily based on Turkish Wikipedia. The project could be extended with structured knowledge and multi-hop reasoning.
Relevant links:
- Having Beer after Prayer? Measuring Cultural Bias in Large Language Models
- Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations
- History (H), Traditional Culture and Arts (CA), Daily Life and Customs (LC), Entertainment (E), Public Figures (F), Geography (G), and Chinese Language (L)
- KoCommonGEN v2: A Benchmark for Navigating Korean Commonsense Reasoning Challenges in Large Language Models