You have the right to refuse. According to the Hong Kong law, parking is not allowed in restricted areas (unless with special permits). If you encounter the above-mentioned situation, you can directly discuss with the customer and arrange another pick-up and drop-off point nearby.