AI in cybersecurity: ChatGPT shows promise but hallucinates under pressure

A newly published peer-reviewed study examines the cybersecurity capabilities and limitations of generative large language models (GenLLMs) like ChatGPT, testing their aptitude in three real-world use cases. Titled “Cybersecurity Applications of Near-Term Large Language Models” and published in Electronics (2025, Vol. 14, Article 2704), the paper explores whether GenLLMs can serve as effective cybersecurity digital assistants, threat analysts, and components of identity and access management systems under a zero-trust framework.

Specifically, the study evaluates ChatGPT v3.0 across a battery of experimental conditions, including industry certification tests, log analysis, and resilience against jailbreak prompt attacks to determine the readiness of near-term GenLLMs for deployment in active cybersecurity operations.

Can ChatGPT operate as a certified cybersecurity assistant?

The first use case tests whether ChatGPT can fulfill the role of a digital assistant by passing industry-standard cybersecurity certification exams. Drawing from extensive question banks modeled after official certification standards, including New York State's program, CompTIA Security+, and the NSA CAE-CD curriculum, the researchers compiled multiple-choice exams on core cybersecurity fundamentals, ethical hacking, and mobile device security.

ChatGPT averaged 83% on cybersecurity fundamentals, 87% on ethical hacking, and 80% on mobile security across a series of 17, 10, and 10 exams, respectively. These scores, while demonstrating baseline competency, fall short of high-performing human candidates. Human undergraduate cybersecurity students routinely scored above 90% on similar assessments, with some nearing perfect scores.

When tasked with writing a response for the IEEE Cybersecurity Ethics competition, ChatGPT produced a 10-page essay that was anonymously judged alongside human submissions. The AI-generated entry received 73 out of 100 points, enough to pass, but notably lower than the winning entries, which scored in the high 80s to low 90s. Key weaknesses included hallucinations (e.g., mislabeling elements of the IEEE Code of Ethics), fabricated citations, and failure to link ethical principles accurately to case studies.

Notably, the study found that ChatGPT sometimes confidently provided incorrect information. For instance, it falsely asserted contradictory statements about the Advanced Encryption Standard (AES), a critical misstep in any security context. In another example, the model failed to recall the 2013 Metcalf Incident despite the information pre-dating its January 2022 knowledge cutoff.

Can ChatGPT support incident response and threat hunting?

Secondly, the study assesses ChatGPT’s ability to assist in incident response and threat hunting, specifically in parsing system logs and analyzing network scans. Using the publicly available GFEK Real-CyberSecurity dataset with over 555,000 entries and Nmap scans, the researchers compared the AI's performance against that of a human analyst.

ChatGPT showed partial effectiveness. For short, structured inputs such as small Nmap scans, it accurately identified open ports, filtered connections, and interpreted the network’s topology. However, the model struggled with large data sets due to input token limits, often failing with prompts such as “this conversation is too long.” Attempts to break logs into smaller parts were only marginally successful and not scalable for enterprise-grade systems.

In honeypot traffic analysis, ChatGPT’s performance was particularly limited. While it correctly identified a few complex commands as potentially malicious, it only detected a small fraction of unauthorized login attempts compared to a human analyst. Specifically, the human detected 1,199 login attempts with 237 successful identifications, while ChatGPT only managed to flag 25 attempts, of which just 9 were deemed successful.

These shortcomings highlight the current GenLLM inability to handle high-volume log data or integrate effectively with real-time security operations. Although promising in its ability to parse individual commands or generate preliminary reports, ChatGPT falls significantly short of matching the depth, accuracy, and contextual understanding of trained human professionals or purpose-built intrusion detection systems.

Can ChatGPT be trusted in access control under zero trust?

The final use case assesses whether ChatGPT can be manipulated through jailbreak prompts to violate cybersecurity protocols, particularly in identity and access management (IAM) systems under a zero trust architecture. Given the risk of prompt injection attacks, a known vulnerability in prompt-based LLMs, the researchers tested ChatGPT’s resistance to four commonly known jailbreak prompts: Always Intelligent and Machiavellian (AIM), Freemode, Mongo Tom, and Do Anything Now (DAN).

Each prompt was applied 20 times to different ChatGPT instances. In every case, the model resisted the jailbreak attempts. It refused to generate malicious code or violate access control policies, even when prompted in role-play or obfuscated language scenarios.

This robust defense suggests OpenAI has actively patched the model against known jailbreak vectors. However, the study cautions that unknown or unpublished prompt engineering methods may still pose risks. Since ChatGPT is a black-box model with proprietary training data and architecture, its vulnerability window to emerging attack techniques remains unquantified.

While encouraging in terms of resilience, the model’s trustworthiness for IAM deployment is not yet absolute. Continuous monitoring and rapid patch cycles remain vital to guard against evolving threats.

Future cybersecurity use of GenLLMs like ChatGPT will likely hinge on fine-tuning models with high-quality, structured security data, improving interpretability, and integrating them into hybrid systems with human oversight. Until then, their role will remain supportive.