Protein farnesyltransferase (FTase) catalyzes the attachment of a farnesyl lipid group to the cysteine residue located in the C-terminal tetrapeptide of many essential signal transduction proteins, including members of the Ras superfamily. Farnesylation is essential both for normal functioning of these proteins, and for the transforming activity of oncogenic mutants. Consequently FTase is an important target for anti-cancer therapeutics. Several FTase inhibitors are currently undergoing clinical trials for cancer treatment. Here, we present the crystal structure of human FTase, as well as ternary complexes with the TKCVFM hexapeptide substrate, CVFM non-substrate tetrapeptide, and L-739,750 peptidomimetic with either farnesyl diphosphate (FPP), or a nonreactive analogue. These structures reveal the structural mechanism of FTase inhibition. Some CaaX tetrapeptide inhibitors are not farnesylated, and are more effective inhibitors than farnesylated CaaX tetrapeptides. CVFM and L-739,750 are not farnesylated, because these inhibitors bind in a conformation that is distinct from the TKCVFM hexapeptide substrate. This non-substrate binding mode is stabilized by an ion pair between the peptide N terminus and the alpha-phosphate of the FPP substrate. Conformational mapping calculations reveal the basis for the sequence specificity in the third position of the CaaX motif that determines whether a tetrapeptide is a substrate or non-substrate. The presence of beta-branched amino acids in this position prevents formation of the non-substrate conformation; all other aliphatic amino acids in this position are predicted to form the non-substrate conformation, provided their N terminus is available to bind to the FPP alpha-phosphate. These results may facilitate further development of FTase inhibitors.