Text and description quoted verbatim from Microsoft DirectX 7 help file, included here for convenient reference.

The D3DSTENCILOP enumerated type describes the stencil operations for the D3DRENDERSTATE_STENCILFAIL, D3DRENDERSTATE_STENCILZFAIL, and D3DRENDERSTATE_STENCILPASS render states.

typedef enum _D3DSTENCILOP {
D3DSTENCILOP_KEEP = 1,
D3DSTENCILOP_ZERO = 2,
D3DSTENCILOP_REPLACE = 3,
D3DSTENCILOP_INCRSAT = 4,
D3DSTENCILOP_DECRSAT = 5,
D3DSTENCILOP_INVERT = 6,
D3DSTENCILOP_INCR = 7,
D3DSTENCILOP_DECR = 8,
D3DSTENCILOP_FORCE_DWORD = 0x7fffffff
} D3DSTENCILOP;

D3DSTENCILOP_KEEP
Do not update the entry in the stencil buffer. This is the default value.
D3DSTENCILOP_ZERO
Set the stencil-buffer entry to 0.
D3DSTENCILOP_REPLACE
Replace the stencil-buffer entry with reference value.
D3DSTENCILOP_INCRSAT
Increment the stencil-buffer entry, clamping to the maximum value. See remarks for information on the maximum stencil-buffer values.
D3DSTENCILOP_DECRSAT
Decrement the stencil-buffer entry, clamping to 0.
D3DSTENCILOP_INVERT
Invert the bits in the stencil-buffer entry.
D3DSTENCILOP_INCR
Increment the stencil-buffer entry, wrapping to 0 if the new value exceeds the maximum value. See remarks for information on the maximum stencil-buffer values.
D3DSTENCILOP_DECR
Decrement the stencil-buffer entry, wrapping to the maximum value if the new value is less than 0.
D3DSTENCILOP_FORCE_DWORD
Forces this enumeration to compile to 32 bits in size. This value is not used.
Remarks
Stencil-buffer entries are integer values ranging from 0 through 2n #1, where n is the bit depth of the stencil buffer.