Computer-administered vs paper-and-pencil tests: Is there a difference?

It has become increasingly common for students in New Zealand schools to sit tests using a computer. Sometimes a computer-administered test and a paper-and-pencil version of the same test are used interchangeably, depending on which administrative mode is most convenient. Does this matter? Can we just replace a paper-and-pencil test with a computerised equivalent and assume we can compare results?