A good web scraper will certainly be able to reliably extract the data you need without you encountering too many anti-scraping measures. Here are some key features to look out for. Of course, these are just two of the many use cases for web scraping. In this article, we`ll dive into the world of web scrapers, learn how they work, and how some websites try to block them. Read on to learn more and start scratching! Data that is exploited from external websites may be part of such a database. With regard to the eligibility of screenscraping / web crawling, the first step is therefore to check on the basis of the specific website whether there really is a database in this sense, i.e. whether, for example, significant investments have actually been made. Of course, all these measures are not necessary if you use web scraping responsibly. If you choose web scraping, remember to use it sparingly and respect your hosts! The term « procurement » refers to the search for existing independent elements and their collection in a database, as opposed to the creation of the elements that make up the contents of a database. Investments are protected for the acquisition and collection of already existing material, for its research, research, research, registration, processing and type of supply, which justify the ancillary copyright according to § 87b Abs. 1 S.

1 UrhG (see only OLG Hamburg, judgment of 8.6.2017 – file number 5 U 54/12 Rn. 336 with further references, case law). The works, data or other elements of a database must be independent of each other, i.e. they must be capable of being separated from each other without affecting the value of their informational, literary or other content (see Dreier, in: Dreier/Schulze, UrhG, 6th edition, § 87a Rn. 6 with further references). When creating descriptive texts from existing raw data, no new independent dates or other elements are created for which sufficient information value would remain after it has been extracted. Thus, the applicant does not generate new data when formulating or reformulating texts on the basis of the manufacturer`s existing specifications. On the contrary, the reformulation simply serves to make the existing data more understandable.

It is really difficult to determine the legality of web scraping in the age of digitization. a) In the interests of effective database protection, no excessively high requirements should be imposed as to the materiality of the investment (Kotthoff, in: Dreyer/Kotthoff/Meckel/Hentsch, UrhR, 4th edition 2018, § 87a definitions no. 30 with other references). In order to answer the question whether the acquisition, verification or presentation of the database requires a substantial investment in terms of its nature and scope, an objective criterion must be applied (Kotthoff, loc. cit., para. 31 with other references). In all cases, there is a significant investment in databases designed to be maintained continuously, especially updated. Indeed, the effort made in this context for a certain period of time will sooner or later be considered material. It would be pointless not to grant database protection to the maker of such a database from the outset.

The volume and complexity of data entered into a database may indicate a significant investment (Kotthoff, loc. cit., para. 32 with other references). Defendant for the payment of € 180,000.00 plus interest in the amount of nine percentage points above the base interest rate since 17.6.2016 and to pay an additional € 481.30 for the reimbursement of court fees and lawyers of the plaintiff for the implementation of the authorization procedure file number 203 O 64/15 LG Cologne plus interest. It dismissed the remainder of the action and ordered the plaintiff to pay 44% and the defendant jointly and severally 56% of the costs, with the exception of the costs of the independent evidence procedure (Case No. 14 OH 2/15), which were ordered to pay the defendant pursuant to its partial recognition judgment of 29.6.2017. It ordered the applicant to pay 44% of the extrajudicial costs incurred by the intervener at first instance. Otherwise, no refund should be made. The most important applications are manual web scraping and software scraping. While manual screen scraping enters the source code and copies it manually, software scraping tools help to read the visible information of a website, save it, and finally process it for your own purposes.

Many website scraping features are readily available in the form of web scraping tools. While there are many tools out there, they vary greatly in quality, price, and (unfortunately) ethics. In this case, § 60c(1) and (3) of the UrhG are particularly relevant. This regulation allows the reproduction of small works for the purpose of non-commercial scientific research to the extent necessary. Posts on social networks such as Facebook are usually small works. They are usually less extensive than poems, poetry or printed works of up to 25 pages, which are mentioned by the legal justification as typical works of small scope. Technical precautions can also ensure that harvesting procedures only cover contributions of a certain size. The copyrights of users are therefore not infringed by web scraping. Popular web harvesting tools include Octoparse, Import.io, and Parsehub.

The operators of the website have no quasi-exclusive rights to the data stored on their website. The compilation of data, on the other hand, may well be protected. Because there is in the EU what is called the right to produce databases. Websites such as rating portals, online exchanges or social networks are usually databases in this sense. It can therefore be assumed that most websites relevant to empirical research represent a database. Web scraping is now increasingly used in empirical research. This is a process by which data from the Internet is « retrieved ». « Scrapers » are small programs that call the desired websites and read the information, such as hotel prices, from there and save it to a file that researchers can then use for their investigation. For example, for one research project, price data from 30,000 hotels was collected by scraping online travel agencies such as Booking.com to analyze best price clauses. However, since neither Report I nor Report T.

do not provide clarity as to the amount of data available in the applicant`s database and the amount of data that was included, but only a selection of product pages was compared, the specific number of data records included cannot be reliably determined. However, all the experts and the District Court as well as the parties unquestionably assumed that at least 44,218 article descriptions had been taken up and used. The question of whether this number should be classified as material within the meaning of the standard with 189,655 records of active product data that were available can be left unanswered. Web scraping is also – and most importantly – used by hackers to illegally misuse website data. The objectives it pursues are as follows: Therefore, web scraping is only necessary if the desired web data is not in the form you need. Whether it means the formats you want aren`t available or the website just doesn`t provide all the data, web scraping allows you to get what you want. For example, if a web host notices that many requests are made by the same user with an outdated version of Mozilla Firefox, it can simply block that version and thus also the bot. These blocking features are included in most managed hosting plans. (b) These statements about the reasons for a deviation from the court`s opinion are ultimately unconvincing.

The expert Dr. H. had already based its assessment on a simple licence and nevertheless considered that, having regard to the other circumstances of the case, in particular the fact that it is a database built up over many years, the quality of which differs from that of other databases, for example by the possibility of comparing product search queries internally, and would be used by a direct competitor of the applicant, the total production costs are nevertheless reasonable as a minimum licence. On the other hand, the reasons given by the Regional Court for significantly reducing licence fees have an equally increasing effect, since in this respect it must also be taken into account that the applicant must grant a licence to a direct competitor who is in direct competition with him with the data in his database, which has been constituted and maintained for many years. The circumstances of the present case, which suggest that production costs for the assessment of royalties should exceptionally be taken into account, are set out in Private Report F. (pages 36-39) and – although essentially on the basis of the private report – in Mr H.`s expert report. A halving cannot be significantly justified by the nature of a mere license and the associated restrictions, as reasonable parties would also have agreed to change the data to be transferred.