TokenROC: A ROC-Based Evaluation Method for XAI Techniques in Software Vulnerability Token Detection 


Vol. 14,  No. 9, pp. 677-686, Sep.  2025
https://doi.org/10.3745/TKIPS.2025.14.9.677


  Abstract

Recently, deep learning-based models have advanced automated software vulnerability detection. However, most provide only coarse information, such as vulnerability presence or affected code lines, and cannot precisely identify the actual vulnerable tokens. To address this, explainable artificial intelligence (XAI) techniques have been applied to generate token-level explanations. Yet, systematic methods for quantitatively evaluating their accuracy remain lacking. This paper proposes TokenROC, a framework that measures how accurately XAI methods detect security-critical tokens. It compares attribution maps with ground-truth labels and evaluates performance using ROC curves and AUC scores. We apply TokenROC to VulBERTa and VulDeBERT using four XAI methods: TIS, AttCAT, Rollout, and Grad-SAM. Results are also compared with the performance of the StagedVulBERT model. Experiments show that XAI methods can meaningfully identify vulnerable tokens. TokenROC offers a practical framework for evaluating and improving explainability-driven vulnerability analysis.

  Statistics
  Cite this article

[IEEE Style]

J. Kim, S. Cheon, S. Han, S. Lee, "TokenROC: A ROC-Based Evaluation Method for XAI Techniques in Software Vulnerability Token Detection," The Transactions of the Korea Information Processing Society, vol. 14, no. 9, pp. 677-686, 2025. DOI: https://doi.org/10.3745/TKIPS.2025.14.9.677.

[ACM Style]

Jaehong Kim, Sebeom Cheon, Sungmin Han, and Sangkyun Lee. 2025. TokenROC: A ROC-Based Evaluation Method for XAI Techniques in Software Vulnerability Token Detection. The Transactions of the Korea Information Processing Society, 14, 9, (2025), 677-686. DOI: https://doi.org/10.3745/TKIPS.2025.14.9.677.

Forms

Search
(IN TITLE, AUTHOR, ABSTRACT,KEYWORDS)

Advanced Search ▼

POPULAR KEYWORDS
(TOP 10 KEYWORDS)

Recent Publications
(LAST 3 YEARS)

Old Journals

Indexing

All publications of TKIPS is indexed in DOI, EBSCO, Google Scholar, Crossref, and CrossCheck.

TKIPS is also selected as the Journal for Accreditation by NRF (National Research Foundation of Korea).

Related Journals