Large Language Models: A New Approach for Privacy Policy Analysis at Scale

Abstract

This article investigates the use of large language models (LLMs), including ChatGPT and Llama 2, to extract and classify data practice disclosures from privacy policies. It evaluates prompt engineering strategies, parameter tuning, and model configurations to optimize LLM performance for legal text analysis.

The study shows that LLMs can outperform symbolic and statistical NLP approaches—achieving an F1 score exceeding 93% on benchmark datasets like MAPP and OPP-115—while reducing development costs and annotation requirements. The findings highlight the potential of LLMs to become the new standard for scalable, automated privacy policy analysis.

Key Contributions

• 🔍 Achieves over 93% F1 score in privacy practice classification using ChatGPT-4 Turbo.
• ⚖️ Outperforms symbolic tools (e.g., PolicyLint) and statistical models (e.g., SVM).
• 🧠 Proposes optimal prompt and parameter configurations for reproducible and deterministic results.
• 🧪 Evaluated on four privacy policy corpora: MAPP, OPP-115, APP-350, and IT-100.
• 💡 Demonstrates generalization capabilities to new privacy practices (e.g., international data transfers).
• 💰 Highlights cost-efficiency tradeoffs between LLMs (GPT-3.5, GPT-4, Llama 2) and traditional methods.

👉 Read the full paper

Recommended citation: D. Rodriguez, I. Yang, J.M. Del Alamo, N. Sadeh. "Large Language Models: A New Approach for Privacy Policy Analysis at Scale." Computing (2024). https://doi.org/10.1007/s00607-024-01331-9
Download Paper