Cyber winds ༄༄

Cybersecurity and infrastructure software trends, supported by 33N ventures

Figure 1. “Generative AI wave against cybersecurity headwinds” (source: Runwayml)

In the wake of a transformative year in 2023, where the promise of GenAI captured the world's imagination, 2024 dawned as the year to bring these lofty ideals into tangible reality. However, three months into this pivotal year, critical voices are now raising concerns about the balance between the benefits and the risks associated with adopting GenAI use cases.

What are the real impediments in the path of widespread adoption? Beyond the challenges of prioritising use cases, hiring/retaining the right technical talent, designing & building the tech stack, among others, there are still substantial technological hurdles yet to be solved (and enough space to build!).

Nascent Adoption of Large Language Models (LLMs)

Enterprises, while enthusiastic about the potential of LLMs, remain in the experimental phase, hesitating to transition from exploration to production. GSI Accenture announced bookings of a whooping $600 million in AI revenue, underscoring the significant investments being made in this arena.

According to Gartner, the initial response from enterprises was one of caution, actively blocking usage of public LLM’s by its staff to avoid issues such as data leakage or IP protection, focusing on damage control measures. Notably, emerging security vendors like Prompt Security, among others, stepped up to address these concerns by offering ‘Shadow AI detection’ tools to monitor the usage of GenAI as well as LLM firewalls to protect for data leakage to outside the organisation.

Figure 2. Five Tactics to Quickly Adapt to Uncontrolled LLM App Consumption (source: Gartner)

Medium to long-term, however, enterprises have started to embrace LLMs, initially for internal use cases which provide higher risk control, while building confidence for future external ones. At present, as a recent report from a16z shows, the primary use cases for LLMs are predominantly internal-facing, reflecting a cautious approach to deployment.

Figure 3. Enterprise LLM use cases (source: a16z).

Persistent Challenges in LLM Security

Security remains a paramount concern in the adoption of LLMs, with issues such as bias, hallucinations, and vulnerabilities yet to be fully resolved. The Open Web Application Security Project (OWASP) has identified the top 10 threats posed by LLMs, with proactive measures being advocated to mitigate risks. Caleb Sima, Chair of the CSA AI Safety Initiative, provides detailed insights into the top-3 threats for LLMs today and the most mature solution types the industry has so far found:

• Prompt Injection: Manipulating the prompts provided to the LLM to induce unintended behaviour. Most promising solutions so far include prompt vulnerability scanners (pre-production) as well as LLM firewalls, dual LLM approaches (as suggested by Simon Willison) or ChatML model (in production).

• Data Poisoning: Attackers injecting malicious data into the training datasets, leading to skewed model outputs. Most considered solutions are still evolving but mainly rely on pre-GenAI tools for data verification, outlier detection and trusted domain enforcement.

• Data Leakage: Unauthorised access to model outputs or training data poses a significant risk. Access control mechanisms (built into the application architecture design) as well as LLM firewalls are essential for mitigating this threat.

Several players have emerged to cover these issues (mapped in the several market maps, eg. Menlo ventures’ map) but the landscape is yet made up of very nascent players, while ‘traditional ML’ security players (eg. Robust Intelligence, Protect AI, HiddenLayer or CalypsoAI with different scopes across the ML dev pipeline) quickly evolved their offering or acquired smaller vendors to cover LLM security.

Figure 4. Security for AI market map in Feb-2024 (source: Menlo ventures)

Navigating LLM Compliance Amid Regulatory Shifts

The regulatory landscape surrounding LLMs is evolving rapidly, with recent legislative developments demanding compliance from stakeholders. The European Union's AI Act, one of the most comprehensive regulatory frameworks, is set to have a profound impact on the adoption and deployment of AI technologies (ML and LLM broadly). Most importantly the act sets key references in terms of risks for AI systems and different timelines for application of the legislation for each risk level:

• Timeline Overview: The act outlines a phased timeline for implementation, but generally becomes fully applicable 24 months after its entry into force (with some exceptions, including obligations for high-risk systems which will apply 36 months after enter into force).

• Risk Overview: While this depends per use case, certain industries such as insurance and financial services, are poised to be among the first to feel the impact of these regulatory changes (eg. credit scoring used for loans were deemed as high risk) - industry leaders like Allianz recently highlighted the need for proactive measures to ensure compliance with the EU AI Act.

Figure 5. Regulatory Framework’s 4 levels of risk for AI systems (source: EU AI Act)

As stakeholders grapple with the complexities of compliance, collaboration and knowledge sharing will be essential in navigating the regulatory landscape effectively.

Unlocking the Potential

Despite the formidable challenges on the horizon, the journey towards widespread LLM adoption presents a unique opportunity to unlock unprecedented productivity gains.

At 33N, we remain optimistic in the cyber ecosystem's ability to address the security and compliance hurdles on the horizon. We also continue deeply grounded on the tech stack needed to overcome these hurdles and would enjoy to connect with any practitioners paving the way in the ecosystem, as a vendor, end-customer enterprise or integration partner. As the industry navigates through these challenges, the potential for transformative advancements in GenAI adoption remains very promising.

If you want to reach-out, connect at info@33n.vc or find us at any key industry events, in the next few weeks at:

• CyberTech in Tel-Aviv, Israel, 8th to 10th April 2024

• EIC Scaling Club Ignition Forum in Brussels, Belgium, 9th and 10th April 2024

• RSA in San Francisco, USA, 6th to 9th May 2024

Let the winds keep blowing!