Test Design and Execution Skill

This skill enables AI to design, structure, and execute comprehensive tests across various domains including software testing, educational assessment, experimental design, and quality assurance protocols.

1. Requirements Analysis

• Define testing objectives and success criteria
• Identify stakeholders and their expectations
• Determine scope, constraints, and resources
• Establish timeline and deliverables

2. Test Planning

• Select appropriate testing methodologies
• Create test strategy document
• Design test cases and scenarios
• Prepare test environment requirements
• Define entry and exit criteria

3. Test Design

• Develop detailed test procedures
• Create test data sets
• Design validation checkpoints
• Establish measurement criteria
• Plan for edge cases and failure scenarios

4. Execution Preparation

• Set up test environment
• Validate test tools and infrastructure
• Brief testing team on procedures
• Conduct dry runs if necessary
• Prepare documentation templates

5. Test Execution

• Execute tests according to plan
• Document results in real-time
• Track defects and anomalies
• Maintain detailed logs
• Communicate progress to stakeholders

6. Analysis and Reporting

• Analyze results against criteria
• Identify patterns and trends
• Generate comprehensive reports
• Provide recommendations
• Plan follow-up actions

Test Coverage Metrics

• Requirement Coverage: % of requirements covered by test cases (Target: >95%)
• Code Coverage: % of code executed during testing (Target: >80% for critical modules)
• Path Coverage: % of execution paths tested (Target: >70%)
• Boundary Value Coverage: % of boundary conditions tested (Target: 100%)

Test Effectiveness Metrics

• Defect Detection Rate: Defects found per test hour (Baseline varies by domain)
• Test Case Pass Rate: % of test cases passing (Target varies by project phase)
• Defect Escape Rate: Critical defects found in production vs. testing (Target: <5%)
• Test Execution Velocity: Test cases executed per day/hour

Quality Metrics

• Defect Density: Defects per unit of product (e.g., per KLOC, per feature)
• Mean Time to Failure: Average time between failures in testing
• Test Case Effectiveness: % of test cases that find defects
• Customer Satisfaction Score: For educational/training assessments

Software Testing

Functional Testing Metrics:

• API response time (<200ms for critical endpoints)
• User interface responsiveness (<3 seconds page load)
• Database transaction accuracy (100%)
• Security vulnerability detection rate

Example Test Case:

Test: E-commerce checkout process
Metrics: Transaction completion rate >99%, payment processing time <5 seconds
Edge Cases: Network timeouts, insufficient funds, expired cards
Automation: Yes - regression suite candidate

Educational Assessment

Assessment Quality Metrics:

• Content Validity: % of learning objectives covered (Target: 100%)
• Reliability Coefficient: Cronbach's alpha >0.7 for high-stakes tests
• Item Discrimination Index: 0.3-0.7 for effective questions
• Test-Retest Reliability: Correlation >0.8

Assessment Rubric Template:

Learning Objective: [Specific skill/knowledge]
Performance Levels:
- Exemplary (4): [Detailed criteria]
- Proficient (3): [Clear expectations]
- Developing (2): [Partial mastery indicators]
- Beginning (1): [Minimal competency markers]
Scoring Guide: [Specific examples for each level]

Scientific Experiments

Experimental Design Metrics:

• Statistical Power: >0.8 for detecting meaningful effects
• Sample Size Adequacy: Based on effect size calculations
• Control Variable Consistency: <5% variation in controlled conditions
• Reproducibility Rate: >80% successful replication

Control Protocols:

• Environmental controls (temperature ±2°C, humidity ±5%)
• Randomization procedures
• Blinding protocols (single/double-blind)
• Measurement instrument calibration

Manufacturing QA

Quality Control Metrics:

• First Pass Yield: % of products passing initial inspection (Target: >95%)
• Defect Parts Per Million: Quality defects per million units (Target: <100 DPPM)
• Process Capability Index: Cpk >1.33 for critical characteristics
• Customer Return Rate: <0.5% for quality issues

QA Protocol Example:

Product: Automotive brake components
Inspection Points: Dimensional accuracy (±0.01mm), material hardness (45-50 HRC)
Sample Size: 10% of lot or minimum 50 pieces
Test Equipment: CMM calibrated within 30 days, hardness tester verified daily
Documentation: Certificate of compliance for each lot

When to Automate

High Priority for Automation:

• Regression tests executed frequently (>5 times per release)
• Data-driven tests with multiple input combinations
• Performance and load testing scenarios
• Repetitive functional tests with clear pass/fail criteria
• Tests requiring precise timing or large datasets

Keep Manual:

• Exploratory testing and usability evaluation
• Complex business logic validation requiring human judgment
• Tests with frequently changing requirements
• One-time testing scenarios
• Tests requiring physical manipulation or sensory evaluation

Tool Selection Criteria

Technical Criteria:

• Platform compatibility (Windows, Mac, Linux, mobile)
• Programming language support and team expertise
• Integration capabilities with CI/CD pipeline
• Reporting and analytics features
• Maintenance effort and learning curve

Business Criteria:

• Total cost of ownership (licensing, training, maintenance)
• Vendor support and community ecosystem
• Scalability for team size and project growth
• Compliance with organizational security standards

Automation Framework Template

1. Test Data Management
   - External data files (CSV, JSON, database)
   - Test data generation and cleanup
   
2. Page Object Model (for UI testing)
   - Reusable element locators
   - Business logic separation
   
3. Reporting Framework
   - Test execution dashboards
   - Failure analysis and screenshots
   
4. CI/CD Integration
   - Trigger conditions and scheduling
   - Result notification mechanisms

Planning Phase

• Start with clear, measurable objectives
• Involve all relevant stakeholders early
• Use risk-based prioritization
• Plan for both positive and negative scenarios
• Build in time buffers for unexpected issues

Design Phase

• Create modular, reusable test components
• Use clear, unambiguous language
• Include both functional and non-functional aspects
• Design for traceability and reproducibility
• Consider automation opportunities

Execution Phase

• Follow procedures consistently
• Document everything, including deviations
• Communicate issues immediately
• Maintain objectivity and avoid bias
• Validate results before proceeding

Reporting Phase

• Present findings clearly and concisely
• Focus on actionable insights
• Include supporting evidence
• Tailor communication to audience
• Provide concrete recommendations

Basic Test Case Template

Test Case ID: [Unique identifier]
Title: [Brief descriptive name]
Objective: [What this test validates]
Preconditions: [Setup requirements]
Test Steps: [Numbered procedure]
Expected Results: [What should happen]
Actual Results: [What did happen]
Status: [Pass/Fail/Blocked]
Notes: [Additional observations]
Automation Candidate: [Yes/No with rationale]

Test Plan Template

1. Test Objectives
2. Scope and Approach
3. Test Items
4. Features to Test
5. Features Not to Test
6. Test Environment
7. Entry/Exit Criteria
8. Risk Assessment
9. Schedule
10. Resources Required
11. Success Metrics and KPIs
12. Automation Strategy

Test Report Template

Executive Summary
Test Objectives Achievement
Test Coverage Summary (with metrics)
Defect Summary (with severity breakdown)
Key Performance Indicators
Risk Assessment
Automation Results
Recommendations
Appendices (detailed results)

Traceability Matrix Template

Requirement ID | Test Case ID | Test Type | Automation Status | Coverage Level
REQ-001       | TC-001       | Functional| Automated         | Complete
REQ-001       | TC-002       | Negative  | Manual            | Partial

Good Test Design

Software Login Function Test

• Clear objective: "Validate user authentication with valid credentials"
• Specific steps: "1. Navigate to login page 2. Enter username 'testuser' 3. Enter password 'Test123!' 4. Click Login button"
• Measurable outcome: "User redirected to dashboard within 3 seconds"
• Edge cases included: Invalid credentials, empty fields, SQL injection attempts
• Metrics: Response time <3 seconds, 100% functional coverage

Poor Test Design

Vague Login Test

• Unclear objective: "Check if login works"
• Vague steps: "Try to log in"
• No specific criteria: "Should work properly"
• Missing edge cases: Only tests happy path
• No metrics defined

Educational Assessment Example

Good Assessment Design:

Course: Introduction to Statistics
Learning Objective: Students can calculate and interpret standard deviation
Assessment Type: Problem-solving with rubric
Success Criteria: 80% of students score "Proficient" or higher
Sample Size: All enrolled students (n=150)
Reliability Target: Cronbach's α > 0.75

Manufacturing QA Example

Good QA Protocol:

Product: Pharmaceutical tablets
Test: Content uniformity
Acceptance Criteria: 95-105% of labeled amount, RSD ≤6%
Sample Size: 30 tablets per batch
Equipment: HPLC with validated method
Frequency: Every batch
Documentation: Certificate of analysis with individual results

Common Mistakes

1. Insufficient Planning

   • Starting testing without clear objectives
   • Skipping risk assessment
   • Not defining success criteria upfront
   • Failing to establish baseline metrics

2. Poor Test Design

   • Writing overly complex test cases
   • Missing critical edge cases
   • Not considering user workflows
   • Ignoring non-functional requirements
   • Creating tests that can't be automated when automation would be beneficial

3. Execution Errors

   • Deviating from planned procedures without documentation
   • Not recording intermediate results
   • Rushing through test steps
   • Ignoring environmental factors
   • Mixing manual and automated results incorrectly

4. Documentation Failures

   • Incomplete or unclear test records
   • Not tracking defects properly
   • Missing traceability to requirements
   • Poor communication of results
   • Failing to maintain automation scripts

5. Analysis Pitfalls

   • Drawing conclusions from insufficient data
   • Ignoring statistical significance
   • Not considering confounding variables
   • Failing to validate unexpected results
   • Misinterpreting automation results

6. Automation Mistakes

   • Automating unstable or frequently changing tests
   • Creating unmaintainable automation scripts
   • Not validating automated test reliability
   • Over-relying on automation for exploratory testing

Red Flags to Avoid

• Testing without clear requirements or metrics
• Accepting untested assumptions
• Ignoring negative test scenarios
• Skipping regression testing
• Not involving end users in validation
• Rushing through critical test phases
• Inadequate defect tracking and resolution
• Automating tests that should remain manual
• Creating automation without proper maintenance plans
• Mixing different testing methodologies without clear rationale
• Ignoring domain-specific quality standards
• Not calibrating measurement instruments in technical testing
• Failing to establish statistical significance in experimental designs