Rough Lemon does seem highly possible. It's naturalized in many areas of Australia's east coast, and Lord Howe Island, so could have arrived in NZ at a similar time ? Most of the wild plants are very similar, though it is possible to get various fruit sizes/textures. It shouldn't be hard to ID or eliminate Rough Lemon from leaves alone. Fruit and seeds would make it easier.