Hello, when doing web scraping with Python, what rules should we pay attention to? For example, is scraping prohibited according to the terms of service of the website from which data is being extracted? Is it always a better approach to pull data from an API? What kind of risks are there within the framework of ownership/data protection laws? Any thoughts?
How ethical is web scraping to extract data in Python?
👁️ 10 views💬 5 replies❤️ 0 likes
5 Replies
Before scraping, always check the site's Terms of Service (ToS) and their robots.txt file: that's where the rules are often written. If scraping is prohibited, respect that or use the free API (Reddit, Twitter, and others have good ones) to avoid any trouble. I tested it on a small cultural blog: after 3 requests, their server banned me (403), while the API worked perfectly.
Guys, honestly, I've also had my first crush on web scraping for a while now, but then we got scared off by legal issues! Some sites like Scrapy, for example, have clear Terms of Service that say "automated data collection is prohibited." But what I've seen suggests that using an API can be a better alternative since the source stays consistently updated—especially if you're after data like news or weather. But most importantly, before you do anything, read the site's privacy policy!
Hmm, so like, the terms of service part always confuses me—some sites say "scraping is prohibited," but search engines do the same thing, right? Is there actually a difference or something?
Haha, I got banned yesterday while coding "\*king scraping\* #letitrain" 😅 First read the terms of use, then pray to the API, otherwise when you hit the law, you’ll be like "but Python... 🥲" trying to defend yourself.
When considering both legal and ethical aspects, checking the terms of service of websites should be the first step, and that's how I've progressed in my projects. If an API is available, directly fetching data from it is naturally preferable as it's legal and carries less risk of errors. However, since some sites don't provide an API, I'm forced to resort to scraping.