Trustworthiness characteristics evaluated and documented
Description
Validity, reliability, safety, security, resilience, accountability, transparency, explainability, privacy, and fairness are all evaluated and reported.
⚠️ Risk Impact
Optimising only one trustworthiness dimension (e.g. accuracy) without measuring the others produces models that fail in the unmeasured dimensions. The Samsung ChatGPT leak (April 2023) is a privacy-dimension failure that wouldn't have shown in accuracy metrics.
🔍 How EchelonGraph Detects This
EchelonGraph's Tier 1 Cloud Scanner automatically checks for this condition across all connected cloud accounts. Violations are flagged as high-severity findings with remediation guidance.
🔧 Remediation
Maintain a trustworthiness dashboard with each AI-RMF dimension scored or RAG-rated per model. Update on release; brief leadership quarterly.
💀 Real-World Attack Scenario
Samsung Semiconductor engineers pasted internal source code into ChatGPT to debug a defect (April 2023). OpenAI's training pipeline absorbed the data; the snippets were potentially recoverable through prompt completion by competitors. Samsung banned ChatGPT internally within 7 days; estimated IP exposure included proprietary semiconductor designs worth an undisclosed multi-million-dollar amount.
💰 Cost of Non-Compliance
Samsung ChatGPT leak (Apr 2023): undisclosed but material IP exposure. Enterprise AI data-loss incidents in 2024: avg $3.8M per occurrence (IBM). EU AI Act Article 15 cybersecurity: €15M / 3% revenue.
📋 Audit Questions
- 1.Show me the trustworthiness scorecard for your top 3 deployed models.
- 2.Which dimensions scored 'red' in the last quarter? What action followed?
- 3.How is the scorecard surfaced to non-technical stakeholders?
- 4.What is your data-leak prevention process for staff using third-party LLMs?
🎯 MITRE ATT&CK Mapping
⚡ Common Pitfalls
- ⛔Scoring only the easy dimensions (accuracy, fairness) and skipping the harder ones (explainability, accountability)
- ⛔Letting the trustworthiness scorecard fall stale (>90 days old) — auditors view stale as worse than absent
- ⛔No alerting on dimension-level regression — the scorecard is purely a snapshot
📈 Business Value
Multi-dimensional trustworthiness assessment + alerting catches failures across the long tail of trustworthiness — privacy leaks, robustness regressions, fairness drift — that single-metric monitoring misses.
⏱️ Effort Estimate
2-4 weeks initial scorecard build per model
EchelonGraph ships scorecards per workload with auto-updated dimension scores from telemetry + eval
🔗 Cross-Framework References
Automate NIST AI-RMF MEASURE-2.6 compliance
EchelonGraph continuously monitors this control across all your cloud accounts.
Start Free →