robots.txt is a voluntary convention, written down as RFC 9309 in 2022. A crawler that reads it is expected to skip what it disallows.
Read the full version