I passed him the pen I used to took down his phone number.我明确了一下逻辑。你这个汉语逻辑就没说清，应该是记下号码后再把笔给他。
"I handed him the pen. I used it to write down his telephone number"
