minus-squarea1studmuffin@aussie.zonetoTechnology@lemmy.world•I just wanted to take a moment to enjoy how clean the web can belinkfedilinkEnglisharrow-up1·1 month agorobots.txt is the perfect summary of the web era. A plain text file that politely asked web crawlers not to do certain things. Such an innocent time. linkfedilink
robots.txt is the perfect summary of the web era. A plain text file that politely asked web crawlers not to do certain things. Such an innocent time.