Skip to content

README-TEST.md

Latest commit

 

History

History
332 lines (233 loc) · 11.7 KB

README-TEST.md

File metadata and controls

332 lines (233 loc) · 11.7 KB
Raw

README-TEST: wp-admin-flow benchmark

This document explains what the wp-admin-flow benchmark does, which URL patterns it requests, what data it collects, and how to write your own tests in this package.

1) What this test does

The test script is:

  • tests/examples/wp-admin-flow.test.js

It simulates a real WordPress/WooCommerce admin session with one or more full iterations:

  1. Login page + login action
  2. Navigate to admin dashboard and WC analytics
  3. Browse and interact with orders/coupons/customers/products
  4. Create disposable benchmark artifacts (order + product draft + coupon + customer user)
  5. Add order note
  6. Cleanup created artifacts
  7. Logout

Primary use case:

  • Compare PHP runtime behavior across different CPU hardware using:
    • response header X-PHP-Runtime-Ms (or configured equivalent)
    • CSV output from k6 (php_runtime_ms samples and step tags)

1.1) PHP runtime header setup (required for php_runtime_ms)

To populate php_runtime_ms, the target site must emit a response header with PHP execution time in milliseconds.

Purpose

  • Gives a server-side timing signal that is less sensitive to network jitter than client-side http_req_duration.
  • Enables per-request stack-variant comparisons across runs (sampler, stack, testid).

Example prepend script

Use your PHP auto-prepend script:

<?php
$__start = hrtime(true);
register_shutdown_function(function () use ($__start) {
    $ms = (hrtime(true) - $__start) / 1e6;
    if (!headers_sent()) {
        header('X-PHP-Runtime-Ms: ' . number_format($ms, 3, '.', ''));
    }
});

Configure auto-prepend

Set this in PHP config for the benchmark environment:

auto_prepend_file=/absolute/path/to/your/prepend-runtime.php

Then restart PHP-FPM/Apache as needed.

Verify before benchmark

curl -I https://your-site.example/wp-admin/

Expected header:

X-PHP-Runtime-Ms: <number>

If this header is missing, k6 still runs, but php_runtime_ms will not be populated and php_runtime_header_missing will increase.

1.2) Step catalog (default run)

Sampler names appear in CSV extra_tags and are the primary grouping key for php_runtime_ms.

Step Sampler When it runs
A1–A3 Login page, login POST, dashboard Always (auth bundle)
R1 WC Admin Home Always (analytics bundle)
O1–O2 View orders, filter by status Always (orders bundle)
O3 Search orders WP_ORDER_SEARCH_QUERY set
O4 Create order form / save / edit WP_ADMIN_ORDER_CREATE=true (default)
O5 Edit/update fixture order WP_ADMIN_ORDER_ID set and WP_ADMIN_ORDER_SAVE=true
O7 Add order note (AJAX) After order create; may lack php_runtime_ms if header absent on AJAX
C1 View coupons Always (coupons bundle)
C2 Search coupons WP_COUPON_SEARCH_QUERY set
C3–C4 Create / edit / update coupon WP_ADMIN_COUPON_CREATE=true (default)
U1 View customers Always (customers bundle)
U2 Search customers WP_CUSTOMER_SEARCH_QUERY set
U3–U4 Create / edit / update customer WP_ADMIN_CUSTOMER_CREATE=true (default)
U5 Customer orders tab After customer create
P1 View products Always (catalog bundle)
P3 Search products WP_PRODUCT_SEARCH_QUERY set
P4–P5 Edit / update fixture product WP_ADMIN_PRODUCT_ID set; save requires WP_ADMIN_PRODUCT_SAVE=true
P6 Add product form / save draft WP_ADMIN_PRODUCT_CREATE=true (default)
X1–X4 Order / product / coupon / customer cleanup WP_ADMIN_CLEANUP=true (default)
A4 Logout Always (auth bundle)

There is no P2, P7, or P8 in the current script.

1.3) Pacing and CSV timestamps

Dev mode (K6_DEV_MODE=true, default)

  • One VU runs iterations sequentially (shared-iterations executor).
  • humanSleep() inserts a fixed 2 second pause after most admin steps. This is hard-coded in dev mode; WP_ADMIN_SLEEP_MIN / WP_ADMIN_SLEEP_MAX are not used.
  • WP_ADMIN_ITERATION_SLEEP_SEC (default 2) adds pause between full flow repetitions only.
  • Some sub-flows have no step sleep between requests (e.g. O4 form → save → edit inside createBenchmarkOrder()).

Load mode (K6_DEV_MODE=false)

  • Uses WP_ADMIN_SLEEP_MIN / WP_ADMIN_SLEEP_MAX (random range per step; compose defaults 0/0, code fallback 5–10 if env vars are absent).

Reading CSV rows

  • Do not infer request latency from gaps in the timestamp column (1-second buckets + pacing).
  • Use php_runtime_ms for server-side PHP time and http_req_duration for end-to-end HTTP time.

2) URL patterns requested

The exact requests depend on enabled bundles and fixture variables. Core request patterns include:

Auth

  • GET /<WP_LOGIN_PATH>
  • POST /<WP_LOGIN_PATH> (login payload: log, pwd, wp-submit, redirect_to, testcookie)
  • Optional hide-login fields when configured:
    • WP_LOGIN_EXTRA_PARAM (default aio_special_field) + WP_LOGIN_EXTRA_FIELD (token/value)
  • GET /wp-admin/
  • GET <logout href from admin page>

Analytics

  • GET /wp-admin/admin.php?page=wc-admin

Orders

  • GET /wp-admin/edit.php?post_type=shop_order
  • GET /wp-admin/edit.php?post_type=shop_order&post_status=<status>
  • GET /wp-admin/edit.php?...&post_type=shop_order... (search, optional)
  • GET /wp-admin/post-new.php?post_type=shop_order
  • POST /wp-admin/post.php (create order)
  • GET /wp-admin/post.php?post=<id>&action=edit
  • POST /wp-admin/admin-ajax.php (woocommerce_add_order_note)
  • Optional fixture save path:
    • GET /wp-admin/post.php?post=<fixture_id>&action=edit
    • POST /wp-admin/post.php

Coupons

  • GET /wp-admin/edit.php?post_type=shop_coupon
  • GET /wp-admin/edit.php?...&post_type=shop_coupon... (search, optional)
  • GET /wp-admin/post-new.php?post_type=shop_coupon
  • POST /wp-admin/post.php (create disposable coupon)
  • GET /wp-admin/post.php?post=<id>&action=edit
  • POST /wp-admin/post.php (update disposable coupon)

Customers

  • GET /wp-admin/admin.php?page=wc-admin&path=%2Fcustomers
  • GET /wp-admin/users.php?s=<query> (optional)
  • GET /wp-admin/user-new.php
  • POST /wp-admin/user-new.php (create disposable customer user)
  • GET /wp-admin/user-edit.php?user_id=<id>
  • POST /wp-admin/user-edit.php (update disposable customer)
  • GET /wp-admin/admin.php?page=wc-admin&path=/customers/<id>

Catalog

  • GET /wp-admin/edit.php?post_type=product[...]
  • GET /wp-admin/edit.php?...&post_type=product... (search, optional)
  • GET /wp-admin/post-new.php?post_type=product
  • POST /wp-admin/post.php (save product draft)
  • optional fixture product edit via /wp-admin/post.php
  • optional fixture product save only when WP_ADMIN_PRODUCT_SAVE=true

Cleanup

  • Customer cleanup:
    • /wp-admin/users.php (list + delete action for created benchmark user)
  • Coupon cleanup:
    • /wp-admin/post.php?...action=trash...
    • /wp-admin/edit.php?post_type=shop_coupon&post_status=trash
  • Order cleanup paths (trash/delete via discovered admin links):
    • /wp-admin/post.php?...action=trash...
    • /wp-admin/edit.php?post_type=shop_order&post_status=trash
    • /wp-admin/admin.php?page=wc-orders&status=trash (fallback)
  • Product cleanup:
    • /wp-admin/post.php?...action=trash...

Cleanup matrix (state-safe default):

Entity Created in run Mutated in run Cleanup action
Order Yes Yes Trash + permanent delete
Product draft Yes Yes Trash
Coupon Yes Yes Trash + permanent delete
Customer user Yes Yes Delete user

3) What information we gather

The run writes CSV samples including built-in k6 metrics and custom metrics.

Built-in k6 metrics

  • http_req_duration, http_req_failed, http_reqs, etc.
  • Purpose: transport-level request health and latency overview.

Custom PHP runtime metrics

  • php_runtime_ms (Trend)
    • Parsed from configured response header (WP_ADMIN_PHP_RUNTIME_HEADER_NAME)
  • php_runtime_header_missing (Counter)
    • Header not found for a sampled request
  • php_runtime_header_invalid (Counter)
    • Header present but not parseable numeric milliseconds

Purpose:

  • Estimate server-side PHP time independent of network jitter
  • Compare per-step runtime across stack variants (STACK_LABEL) and run IDs (TEST_ID)

Tags used for analysis

  • testid: run identifier
  • stack: optional stack-variant label (STACK_LABEL, legacy fallback from CPU_LABEL)
  • sampler: step/action name (e.g. O1 View Orders, P6 Save Product Draft)
  • scenario: k6 scenario name (dev_admin_flow in dev mode, admin_flow in load mode; some built-in metric rows may still reference admin_flow)
  • step: main (and sometimes assets)

Purpose:

  • Build request-level comparison matrices in spreadsheets:
    • rows: sampler
    • columns: stack and/or testid
    • values: mean/p95/max of php_runtime_ms

4) Why this benchmark is structured this way

  • Uses realistic admin actions instead of synthetic ping endpoints
  • Keeps cleanup enabled by default to avoid leaving test data behind
  • Default state-safe mode mutates only disposable benchmark entities, never existing records
  • Supports multi-iteration runs with fixed pause between full iterations
  • Dev mode (K6_DEV_MODE=true) uses one VU, sequential steps, and a fixed 2s pause between most admin actions for reproducible CPU comparison

5) Writing your own tests (guidelines)

If you add new tests under tests/examples, follow these rules so results stay comparable and reproducible.

A) Keep tests self-contained

  • Reuse shared helpers from:
    • tests/lib/options.js
    • tests/lib/wp-admin.js
    • tests/lib/wp-admin-cleanup.js (if mutating data)
  • Avoid hidden dependencies on parent project files.

B) Tag every request consistently

  • Use visitAdminPage(...) where possible.
  • If direct http.* calls are needed, include tags:
    • sampler
    • scenario
    • step

This is required for clean CSV grouping.

C) Prefer deterministic benchmark settings

For CPU comparison runs:

  • K6_DEV_MODE=true
  • WP_ADMIN_RUN_ITERATIONS=<N> (e.g. 50 for published comparisons)
  • WP_ADMIN_ITERATION_SLEEP_SEC=2 (pause between full iterations)
  • WP_ADMIN_MAX_DURATION=<cap> if auto-estimate is too low (e.g. 180m for 50 iterations)
  • Ensure X-PHP-Runtime-Ms (or configured header) is present on admin responses

Note: in dev mode, per-step pacing is a fixed 2s (humanSleep()); WP_ADMIN_SLEEP_MIN / WP_ADMIN_SLEEP_MAX apply only when K6_DEV_MODE=false.

Avoid random behavior that adds noise (keep load mode off for CPU studies).

D) Treat cleanup as mandatory for mutating flows

  • If your test creates or edits entities, implement cleanup steps.
  • Record created IDs and remove them at the end of each iteration.

E) Keep bundle/action naming readable

  • Use short stable prefixes for steps (A/O/C/U/P/X/R style is fine).
  • Example: O8 View Refunds, P9 Publish Product.
  • Stable names make historical CSV analysis much easier.

F) Validate header instrumentation

  • Keep WP_ADMIN_PHP_RUNTIME_HEADER_ENABLED=true.
  • Watch php_runtime_header_missing and php_runtime_header_invalid.
  • If they rise, fix instrumentation before trusting comparisons.

G) Add a minimal smoke test when adding parser/helper logic

  • Put smoke tests in tests/smoke when adding new parsing or tagging helpers.
  • Validate locally before sharing benchmark changes.

6) Suggested CSV analysis workflow

  1. Run one test per stack variant with unique TEST_ID and STACK_LABEL.
  2. Merge/filter CSV rows where metric_name == php_runtime_ms.
  3. Pivot by sampler vs stack.
  4. Compare mean, p95, and max.
  5. Check header counters to verify data quality.