Product Verification¶
The product verification activity is conducted by an accredited assessor organization and establishes the product implementation score. The goal is to establish that a product meets the claims the RTP made about it, that is: “does the product prevent unintended outcomes?”
For initial product submissions and extensive changes in a {term}`product revision submission
Methodology¶
For more information about what is expected for the product verification activity, see the provider submission activity and the RABET-V security requirements.
For initial product submissions, a full system test is performed. A full system test will review automated test results and perform a systemwide functional test and penetration test.
For product revision submissions, the test plan determination activity outlines required tests.
Inputs¶
Component definitions from architecture assessment
System-level architecture diagram
System details from organizational assessment
Product revision submission materials, if applicable
Outputs¶
Results of verification test methods
Product implementation score based on the product implementation rubric
Verification Methods¶
An accredited assessor organization will use one or more of the following techniques, as indicated in the test plan. The scope of the testing (i.e., which components to test) will also be indicated by the test plan.
Artifact Review¶
This method will review an artifact provided by the RTP. The review will look for gaps or concerns in relevant controls based on the information provided. Each type of artifact will have various indicators of acceptability. Types of RTP artifacts include:
Automated source code unit test results
Automated vulnerability test results
Automated configuration verification results
Security event audit logs
Third-party security analysis results (automated or manual)
The artifacts must be evaluated as “reliable” during the organizational assessment activity in order to be used for product verification.
Automated Testing¶
Automated testing is a broad type of testing that relies on software to perform test routines against the product or product component. Automated testing will execute the testing software against its target and produce results which will be evaluated by the accredited assessor organization. The type of automated test will depend on the target. Types of automated testing may include:
Configuration testing
Vulnerability analysis
Source code analysis
Accessibility testing
Browser compatibility testing
Penetration Testing¶
Penetration tests evaluate the product to find security vulnerabilities that an attacker could exploit. The scope of a penetration test may be the product’s network, computer systems, hardware components, or software application(s). Penetration testing is typically a combination of manual and automated testing. Automated tools help with web application pen tests but must be used by skilled and experienced testers.
RABET-V relies on OWASP’s Web Application Security Testing Guide for web application and web service penetration testing options.
In addition to a full penetration testing option, the following web application penetration testing subtypes are supported:
Configuration and deployment
Identity management
Authentication
Authorization
Session management
Input validation
Error handling
Cryptography
Limited penetration testing may be used if the changes do not warrant full penetration testing.
Product Implementation Rubric¶
The product implementation rubric provides a maturity score for each of the control families based on how well the product meets the requirements within those families. The scores range from zero to three, where three is the best.
The requirements are a binary pass or fail. Any assumptions made about the configuration or setup of the product must be documented with the result.
The scores are calculated by summing the percentage of applicable requirements that pass at each maturity level. For instance, meeting 100% of requirements at maturity level one, 25% at maturity level two, and 0% at maturity level three would result a score of 1.25.
Security Test Method Descriptions¶
Fuzzing - Test of the application’s ability to accept a wide variety of inputs without causing it to enter an unexpected or undefined state
Penetration Testing - Testing that verifies the extent to which a system, device or process resists active attempts to compromise its security [NIST SP 800-152]
Functional - Test that evaluates the functionality of a component against a design specification. Can be automated, but because the function will be implemented differently by each product, a custom test script may be required for each
Web Testing - A functional test that exercises one or more parts of the web stack and verifies the expected output
Failover and Restore Testing - Test that evaluates the resiliency of a system by making components of the system inoperable and evaluating the result
Code Analysis - A white box test involving the use of code artifacts, such as source code or unobfuscated binaries in order to verify certain properties
Bill of Materials (BOM) Analysis - Analysis of the bill of materials, such as software and their versions
Configuration Audit - Test to verify that the configuration of a component is configured as required
Data Audit - Test to verify the presence or absence of certain records, such as the inappropriate collecting of PII or the lack of authentication logs, can be combined with functional testing to provide a higher level of confidence
Artifact Review - Review of RTP-supplied artifacts from their development, testing, integration, and deployment process or artifacts provided by the RTP’S hosting environment
Documentation Audit - Review of the RTP-supplied documentation for presence of required content or presence of poor guidance (i.e. direction to use insecure password)
Vendor Attestation - A statement made by the vendor indicating the existence of one or more security controls
Accessibility Test Method Descriptions¶
Conformance - Test that validates the conformance of a component, page, or application to a specific accessibility standard. Conformance testing can be automated during development to test components and after development to test full applications. For example, tools like Accessibility Insights can check Android, web, and Windows applications, eslint-plugin-jsx-a11y can perform static analysis on React applications, axe DevTools can be used to test web applications, and SiteImprove is a paid-service that can automate accessibility, spell-checking, and readability checks on web applications
Functional - Test that evaluates the functionality of a component against a set of accessibility expectations. This must include the ability to interact with only keyboard navigation and should include testing with assistive technology (e.g., screen reader, braille display) and plain-language analysis (e.g., ideal Flesch-Kincaid score)
Artifact Review - Review of RTP-supplied artifacts from their automated, functional, or third-party testing
Vendor Attestation - A statement made by the vendor indicating the adherence to one or more accessibility controls
Usability Test Method Descriptions¶
Artifact Review - Review of RTP-supplied artifacts from their functional or third-party testing
Vendor Attestation - A statement made by the vendor indicating the adherence to one or more usability controls
Product Verification Baseline¶
RABET-V uses baseline scoring in organizational, architecture, and product verification to determine whether a product is Verified, Conditionally Verified, or Returned. The product verification baseline contains all the security requirements at level one and 50% of the requirements from levels two and three combined. When the RTP completes the product claims workbook, they will identify at least 50% of the Level two and Level three requirements from each of the control families. These claimed requirements will be part of the testing in the basic and streamlined testing scenarios.
The overall minimum baseline for verification is 106 of 153 requirements, combining all of Level one and 50% of the combined Level two and three requirements in each of the security control families.
The authentication baseline is passing all level one requirements (below) and 6 of 12 level two and three requirements.
Security Control Family |
Requirement |
---|---|
Authentication |
1.1.1 Default passwords are not used or are automatically changed as part of set up |
Authentication |
1.1.2 Authentication is applied consistently through the application |
Authentication |
1.1.3 Encrypt or hash all authentication credentials |
Authentication |
1.1.4 Customer admins have access to an inventory of their user accounts |
Authentication |
1.1.5 Implement protections against brute force attacks |
Authentication |
1.1.6 Require multi-factor authentication for all administrative access |
The authorization baseline is passing all level one requirements (below) and 4 of 8 level two and three requirements.
Security Control Family |
Requirement |
---|---|
Authorization |
2.1.1 Platform provides an authorization system |
Authorization |
2.1.2 Applications and middleware should run With minimal privileges |
Authorization |
2.1.3 Apply the principle of least privilege |
Authorization |
2.1.4 Use tokens to prevent forged requests |
The boundary protection baseline is passing all level one requirements (below) and 6 of 12 level two and three requirements.
Security Control Family |
Requirement |
---|---|
Boundary Protection |
3.1.1 Deny communications with known malicious IP addresses |
Boundary Protection |
3.1.2 Deny communication over unauthorized ports |
Boundary Protection |
3.1.3 Deploy network-based IDS sensors |
Boundary Protection |
3.1.4 Document traffic configuration rules |
Boundary Protection |
3.1.5 Use MFA for managing network infrastructure |
Boundary Protection |
3.1.6 Configure perimeter devices to prevent common types of attacks |
Boundary Protection |
3.1.7 Disable wireless access on devices if it is not required |
Boundary Protection |
3.1.8 Documentation clearly identifies wireless capabilities |
Boundary Protection |
3.1.9 Provide dedicated wireless networks |
Boundary Protection |
3.1.10 Disable wireless peripheral access to devices |
The data confidentiality and integrity baseline is passing all level one requirements (below) and 8 of 15 level two and three requirements.
Security Control Family |
Requirement |
---|---|
Data Confidentiality |
4.1.1 Use valid HTTPS certificates from a reputable certificate authority |
Data Confidentiality |
4.1.2 Encrypt transmittal of username and authentication credentials |
Data Confidentiality |
4.1.3 Use the strict-transport-security header |
Data Confidentiality |
4.1.4 Disable data caching using cache control headers and autocomplete |
Data Confidentiality |
4.1.5 Updated TLS configuration on servers |
Data Confidentiality |
4.1.6 Use TLS everywhere |
Data Confidentiality |
4.1.7 Disable HTTP access for all TLS enabled resources |
Data Confidentiality |
4.1.8 Do not disclose too much information in error messages |
Data Confidentiality |
4.1.9 Display generic error messages |
Data Confidentiality |
4.1.10 Store user passwords using a atrong, iterative, salted hash |
The system availability baseline is passing all level one requirements (below) and 3 of 6 level two and three requirements.
Security Control Family |
Requirement |
---|---|
System Availability |
5.1.1 Ensure regular automated backups |
System Availability |
5.1.2 Backup data should be restorable |
System Availability |
5.1.3 Local distributed storage capability |
System Availability |
5.1.4 Local distributed processing capability |
The injection prevention baseline is passing all level one requirements (below) and 4 of 7 level two and three requirements.
Security Control Family |
Requirement |
---|---|
Injection Prevention |
6.1.1 Use secure HTTP response headers |
Injection Prevention |
6.1.2 Validate uploaded files |
Injection Prevention |
6.1.3 Set the encoding for your application |
Injection Prevention |
6.1.4 Validate all input |
The logging and alerting baseline is passing all level one requirements (below) and 7 of 14 level two and three requirements.
Security Control Family |
Requirement |
---|---|
Logging and Alerting |
7.1.1 Activate audit logging |
Logging and Alerting |
7.1.2 Ensure adequate storage for logs |
Logging and Alerting |
7.1.3 Log all authentication activities |
Logging and Alerting |
7.1.4 Log all privilege changes |
Logging and Alerting |
7.1.5 Do not log inappropriate data |
Logging and Alerting |
7.1.6 Store logs securely |
Logging and Alerting |
7.1.7 Log and alert on changes to administrative group membership |
The secret management baseline is passing all level one requirements (below) and 3 of 5 level two and three requirements.
Security Control Family |
Requirement |
---|---|
Secret Management |
8.1.1 Don’t hardcode credentials |
Secret Management |
8.1.2 Store credentials securely |
Secret Management |
8.1.3 Credentials for non-production and production environments are different |
The system integrity baseline is passing all level one requirements (below) and 6 of 12 level two and three requirements.
Security Control Family |
Requirement |
---|---|
System Integrity |
9.1.1 Install the latest stable version of any security-related updates on all network devices |
System Integrity |
9.1.2 Ensure anti-malware software and signatures are updated |
System Integrity |
9.1.3 Configure devices to not auto-run content |
System Integrity |
9.1.4 Use port protectors on unused ports |
System Integrity |
9.1.5 Configure anti-malware scanning of removable devices |
The user session baseline is passing all level one requirements and 3 of 6 level two and three requirements.