Test a command-line binary

A cli:// scope makes a scan target a local executable instead of a network host. The templates run the binary as a child process, feed it hostile arguments, files and environment, and decide from what they observe. The CLI Security Baseline is the published pack written for this target kind. This guide runs it against a deliberately flawed tool, then its fixed twin, so you can see what a confirmation and a refutation look like before you point it at your own.

The target runs on your machine

A cli:// scan executes the binary you name, many times, with hostile input, as you. cxg does not sandbox template execution. Point it only at a tool you own or are authorized to test, ideally on a machine or account you can throw away.

flowchart TB
    BIN["cli:///absolute/path<br/>to the binary"] --> PRE["--require-instrumentation:<br/>inspect the build first"]
    PACK["CLI Security Baseline<br/>templates"] --> RUN["each template runs the binary<br/>with hostile argv, files, env"]
    PRE --> RUN
    RUN --> V["confirmed, refuted,<br/>skipped, errored, timed-out"]

    class V emphasis

Before you start

• cxg installed. See Installation.
• Detection templates fetched. The baseline pack lives in cert-x-gen-templates, under templates/cli-baseline/, and cxg template update puts it at ~/.cert-x-gen/templates/official/templates/cli-baseline. The command prints a summary of what it installed.
• Python 3 and a C compiler, for the demo tool below.

Build a tool to test

The cxg repository carries a small synthetic tool for exactly this: one source per program, built twice, as a flawed twin that the checks must confirm and a fixed twin they must refute. It opens no network socket, and every planted defect has its one-line fix beside it.

$ git clone -q --depth 1 --branch v1.6.1 https://github.com/Bugb-Technologies/cert-x-gen.git
$ ./cert-x-gen/tests/fixtures/cli-baseline/build.sh ~/fixtures
fixtures built in ~/fixtures:
  memtoy_defective
  memtoy_defective.dSYM
  memtoy_fixed
  memtoy_fixed.dSYM
  notekeeper_defective.py
  notekeeper_fixed.py
  noteprint.py
  notesync.py

notekeeper is a notes CLI written in Python. memtoy is a C program built with AddressSanitizer, for the one check that needs an instrumented build.

Run the baseline

1. Scan the flawed twin with the whole pack. The scope is cli:// followed by an absolute path, so there are three slashes:

   cxg scan --scope cli://$HOME/fixtures/notekeeper_defective.py \
     --template-dir ~/.cert-x-gen/templates/official/templates/cli-baseline \
     --require-instrumentation

   Execution Status:
     CONFIRMED: 12
     REFUTED:   1
     SKIPPED:   1
     ERRORED:   0
     TIMED-OUT: 0
   
     confirmed  cli-baseline-b01-argument-injection → ~/fixtures/notekeeper_defective.py  [oracle=property(argv-canary-created) exit=0 sub=export class=B01]  (template-declared)
     confirmed  cli-baseline-b02-command-injection → ~/fixtures/notekeeper_defective.py  [oracle=property(shell-canary-created) exit=0 sub=render class=B02]  (template-declared)
     confirmed  cli-baseline-b03-path-traversal → ~/fixtures/notekeeper_defective.py  [oracle=property(canary-nonce-echoed) exit=0 probe=../outside.txt sub=show class=B03]  (template-declared)
     confirmed  cli-baseline-b04-archive-traversal → ~/fixtures/notekeeper_defective.py  [oracle=property(archive-member-escaped) exit=0 archive=b04-relative.tar sub=extract class=B04]  (template-declared)
     confirmed  cli-baseline-b05-argv-secrets → ~/fixtures/notekeeper_defective.py  [oracle=property(credential-in-child-argv procs=2) sub=login class=B05]  (template-declared)
     confirmed  cli-baseline-b06-insecure-temp-files → ~/fixtures/notekeeper_defective.py  [oracle=property(temp-files=1) exit=0 sub=convert class=B06]  (template-declared)
     confirmed  cli-baseline-b07-path-hijack → ~/fixtures/notekeeper_defective.py  [oracle=property(path-shim-executed=git) exit=0 sub=sync class=B07]  (template-declared)
     confirmed  cli-baseline-b08-terminal-escape → ~/fixtures/notekeeper_defective.py  [oracle=property(raw-esc-bytes=6 seq=CSI,OSC-0-title,OSC-8-hyperlink) exit=4 sub=show class=B08]  (template-declared)
     confirmed  cli-baseline-b09-environment-trust → ~/fixtures/notekeeper_defective.py  [oracle=property(env-pointed-file-loaded var=NOTEKEEPER_CONFIG) exit=0 sub=config class=B09]  (template-declared)
     confirmed  cli-baseline-b10-config-credential-handling → ~/fixtures/notekeeper_defective.py  [oracle=property(weak-state-file-modes=1) exit=0 sub=init class=B10]  (template-declared)
     skipped    cli-baseline-b11-memory-safety → ~/fixtures/notekeeper_defective.py  [no-instrumentation-detected]
     confirmed  cli-baseline-b12-crash-hang → ~/fixtures/notekeeper_defective.py  [oracle=unhandled-exception(unhandled exception escaped to the top level) exit=1 case=malformed probe=extract class=B12]  (template-declared)
     confirmed  cli-baseline-b13-toctou-symlink-race → ~/fixtures/notekeeper_defective.py  [oracle=property(symlink-followed attempt=1/6) exit=0 sub=touchfile class=B13]  (template-declared)
     refuted    cli-baseline-b14-format-string → ~/fixtures/notekeeper_defective.py  [specifiers came back verbatim through show render export extract convert banner -- weak evidence only: a tool may not echo the argument that reaches its format call class=B14 probes=12]  (template-declared)

   That is the end of the scan summary; the severity breakdown above it is omitted here. Each line names the oracle that decided it, the subcommand of the tool it reached (sub=), and the baseline class.

2. Scan the fixed twin the same way. The confirmations become refutations, each stating what the tool did instead:

     refuted    cli-baseline-b03-path-traversal → ~/fixtures/notekeeper_fixed.py  [target kept every traversal probe inside its own root across show render export convert class=B03 pr…

   Truncated after the first refutation, and that line after 160 characters. B11 is skipped with no-instrumentation-detected on both twins: an interpreted script never carries sanitizer instrumentation.

3. Run the memory-safety check against the AddressSanitizer build:

   cxg scan --scope cli://$HOME/fixtures/memtoy_defective \
     --templates ~/.cert-x-gen/templates/official/templates/cli-baseline/cli-baseline-b11-memory-safety.sh \
     --require-instrumentation

     confirmed  cli-baseline-b11-memory-safety → ~/fixtures/memtoy_defective  [oracle=asan(AddressSanitizer: stack-buffer-overflow on address 0x00016f53d850 at pc 0x0001011cea20 bp 0x0001) exit=134 probe=--label instrumentation=asan,debug-info class=B11]  (template-declared)

   Against memtoy_fixed the same check is refuted: instrumented build (asan,debug-info) handled every over-length input cleanly.

Read the verdicts

A CLI template reports one of five verdicts for each execution:

Verdict	What it means
confirmed	The template’s oracle observed the class’s signal: a canary file created, a sentinel read back, a sanitizer report, a signal.
refuted	The template drove the tool and the signal did not appear. It covers the probes tried, nothing wider.
skipped	The check could not be run meaningfully, and says why. A skip provides no refutation.
errored	The template itself failed.
timed-out	The execution ran past its time budget. Not a refutation.

Two lines in the run above are worth reading closely. B14 on the flawed twin is refuted, but its own reason calls that “weak evidence only”: format-string detection is the pack’s hint-grade class. And a B13 refutation says only that the race was not won in the attempts made. The pack’s README tabulates which classes are clean, partial and hint-grade; read it before you rely on a refutation.

The exit status does not reflect findings

All four scans above exited 0, including the one with twelve confirmed findings. Gate a pipeline on the results file, not on the exit status. See Scan a target.

Instrumentation

--require-instrumentation inspects each cli:// binary for sanitizer, coverage and debug-info markers before running anything against it. A template whose oracles need instrumentation the build lacks is recorded as skipped with no-instrumentation-detected, instead of running and reporting a clean result that looks like a refutation and is not evidence. Templates declaring only oracles that work on any build (exit, signal, timeout, exception, property) still run.

A scan does not compile the target. Build it yourself with its instrumented profile (clang -fsanitize=address -g for C, as the demo does), or see cxg build --instrument, which builds one for supported project types.

Feed a probe template your own input

The baseline finds the tool’s subcommands for itself. A probe template you write can instead take its input from the scan, through flags that set environment variables for the template:

Flag	What the template receives
--arg <ARG> (repeatable)	CERT_X_GEN_ARGV: a JSON string array, keeping argument boundaries and hyphen-leading values
--stdin-file <PATH>	CERT_X_GEN_STDIN_FILE: the path of a file whose bytes the template can feed to stdin
--input <DIR>	CERT_X_GEN_INPUT_DIR: a seed corpus directory; cxg does not mutate or minimise it
--target-env <KEY=VALUE> (repeatable)	CERT_X_GEN_TARGET_ENV: a JSON object for the target process; a repeated key takes the last value

These are absent when their flags are omitted, and the template has to consume them: cxg does not adapt a template to a tool’s command syntax. For a cli:// target, CERT_X_GEN_TARGET_KIND is cli and CERT_X_GEN_TARGET_HOST is the binary’s path.

Related

• The baseline’s checks in the template catalog, starting with B01, argument injection.
• Scan a target covers scopes, template selection and output formats for every target kind.
• cxg scan documents every flag.
• How cxg executes templates, on why a template runs with your privileges.