Author: Denas Grybauskas, Legal Counsel at Oxylabs
Navigating the legalities of web scraping is challenging – even for legal professionals. The landscape is always in a state of flux because of a lack of legislation, particularly as new case law comes into play affecting current and future legal cases.
Understanding the legal implications of web scraping is critically important, particularly for enterprises with business models that depend on the free-flow of data. Ever-changing in depth and complexity, navigating the legal landscape brings forward many questions and concerns for both consumers and businesses.
At Oxylabs, we believe that web scraping can be a force for good and promote ethical web scraping practices to ensure the fair transmission of public data. To that end, this article will address at least some of the concerns companies should have while offering resources to help them navigate the legal terrain of data acquisition.
Web scraping legality must be taken seriously
Web scraping has legal implications depending on where the scraping activity is taking place, the website terms of service (TOS) agreements and how the data will be used. Legal action has been taken from companies against businesses scraping data from their sites that ranges from cease and desist letters to lawsuits.
Further complicating the matter are other issues such as the lack of a standard framework concerning TOS and the legality of scraping data from the same company operating in different jurisdictions. As the world increasingly comes together, many applications and coordinated systems require data collection that spans borders.
Regulations are always changing
Laws concerning data collection simply aren’t moving at the same pace as industry innovation. Even fundamental issues such as the difference between public and non-public data aren’t exact in some cases, making it difficult to know where the line is drawn.
It is difficult for various and historical laws to adapt to the changing nature of data-based businesses. An example of current U.S. legislation includes rules found in the Computer Fraud and Abuse Act (also known as the “anti-hacking act”), enacted in 1986 and often cited by companies claiming that public data on their website was accessed without authorization or exceeding authorization. The European Union has its own crown jewel called the General Data Protection Regulation (GDPR), and the French Data Protection Authority (CNIL) – responsible for enforcement of GDPR in France – has already stated that publicly available data is still personal data and cannot be repurposed by web scrapers without the knowledge of the person who owns that data. These are just two examples from two different locations, and the mix of rules can make web scraping complex for businesses that collect data from across the world without legal guidance.
Ethical web scraping can benefit society at large
Our world is becoming increasingly data-driven and no matter what industry is considered, data is required to make progress on an increasingly digital landscape. To that end, Oxylabs provides guidelines for scraping non-personal data that could form the foundation for a future web scraping code of ethics with suggestions that include:
· Scraping only publicly-available web pages
· Ensuring that the data is requested at a fair rate and that it doesn’t compromise the web server
· Respecting the data obtained and any privacy issues relevant to the source website
· Studying the website’s legal documents, deciding if they will be accepted, and determining if the terms will be breached if accepted
· Making use of proxies that are procured ethically
Open access to data is critical to progress for the industry on all levels, and we believe that it is possible to engage in large-scale data collection while respecting both user privacy and the server infrastructure of public websites.
Key questions to ask before web scraping
There is no doubt that web scraping is here to stay. Until a court or government decides how and if web scraping should be regulated, it’s essential to stay informed on new laws and case law while engaging the services of a legal professional before undertaking any data extraction activities.
Some critical questions to ask include:
· Do web scraping laws differ between the U.S., Europe, Asia and other major markets?
· What must I do if I receive a cease and desist letter? Do I need to take action?
· When is data public and when is it non-public/private?
· Are there incoming regulations and court decisions I should be aware of?
· Does creating an account make a difference when it comes to legality?
· How is the TOS to be taken into consideration if the data is public?
This is only a sample of the questions concerning the legality of web scraping. In order to assist our wide range of clients and other members of the community, Oxylabs has gathered a panel of legal professionals for a discussion back on March 4. You can watch a recording of the discussion here.