Python의 스크랩(개발자를 위한 사용법)

Install Scrapy

Install Scrapy using pip by running the following command:

pip install scrapy

pip install scrapy

Define a Spider

To define your spider, create a new Python file (such as example.py) under the spiders/ directory. An illustration of a basic spider that extracts from a URL is provided here:

import scrapy

class QuotesSpider(scrapy.Spider):
    # Name of the spider
    name = 'quotes'
    # Starting URL
    start_urls = ['http://quotes.toscrape.com']

    def parse(self, response):
        # Iterate through each quote block in the response
        for quote in response.css('div.quote'):
            # Extract and yield quote details
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('span small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }
        # Identify and follow the next page link
        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            yield response.follow(next_page, self.parse)

import scrapy

class QuotesSpider(scrapy.Spider):
    # Name of the spider
    name = 'quotes'
    # Starting URL
    start_urls = ['http://quotes.toscrape.com']

    def parse(self, response):
        # Iterate through each quote block in the response
        for quote in response.css('div.quote'):
            # Extract and yield quote details
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('span small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }
        # Identify and follow the next page link
        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            yield response.follow(next_page, self.parse)

Configure Settings

To set up the Scrapy project parameters like user-agent, download delays, and pipelines, edit the settings.py file. This is an illustration of how to change the user-agent and make the pipelines functional:

# Obey robots.txt rules
ROBOTSTXT_OBEY = True
# Set user-agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'
# Configure pipelines
ITEM_PIPELINES = {
    'myproject.pipelines.MyPipeline': 300,
}

# Obey robots.txt rules
ROBOTSTXT_OBEY = True
# Set user-agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'
# Configure pipelines
ITEM_PIPELINES = {
    'myproject.pipelines.MyPipeline': 300,
}

What is IronPDF?

IronPDF is a powerful .NET library for creating, editing, and altering PDF documents programmatically in C#, VB.NET, and other .NET languages. Since it gives developers a wide feature set for dynamically creating high-quality PDFs, it is a popular choice for many programs.

Features of IronPDF

PDF Generation: Using IronPDF, programmers can create new PDF documents or convert existing HTML elements such as tags, text, images, and other file formats into PDFs. This feature is very useful for creating reports, invoices, receipts, and other documents dynamically.

HTML to PDF Conversion: IronPDF makes it simple for developers to transform HTML documents, including styles from JavaScript and CSS, into PDF files. This enables the creation of PDFs from web pages, dynamically generated content and HTML templates.

Modification and Editing of PDF Documents: IronPDF provides a comprehensive set of functionality for modifying and altering pre-existing PDF documents. Developers can merge several PDF files, separate them into separate documents, remove pages, and add bookmarks, annotations, and watermarks, among other features, to customize PDFs to their requirements.

How to install IronPDF

After making sure Python is installed on your computer, use pip to install IronPDF.

 pip install ironpdf

Scrapy project with IronPDF

To define your spider, create a new Python file (such as example.py) in the spider's directory of your Scrapy project (myproject/myproject/spiders). A code example of a basic spider that extracts quotes from a URL:

import scrapy
from IronPdf import *

class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    # Web page link
    start_urls = ['http://quotes.toscrape.com']

    def parse(self, response):
        quotes = []
        for quote in response.css('div.quote'):
            title = quote.css('span.text::text').get()
            content = quote.css('span small.author::text').get()
            quotes.append((title, content))  # Append quote to list

        # Generate PDF document using IronPDF
        renderer = ChromePdfRenderer()
        pdf = renderer.RenderHtmlAsPdf(self.get_pdf_content(quotes))
        pdf.SaveAs("quotes.pdf")

    def get_pdf_content(self, quotes):
        # Generate HTML content for PDF using extracted quotes
        html_content = "<html><head><title>Quotes</title></head><body>"
        for title, content in quotes:
            html_content += f"<h2>{title}</h2><p>Author: {content}</p>"
        html_content += "</body></html>"
        return html_content

import scrapy
from IronPdf import *

class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    # Web page link
    start_urls = ['http://quotes.toscrape.com']

    def parse(self, response):
        quotes = []
        for quote in response.css('div.quote'):
            title = quote.css('span.text::text').get()
            content = quote.css('span small.author::text').get()
            quotes.append((title, content))  # Append quote to list

        # Generate PDF document using IronPDF
        renderer = ChromePdfRenderer()
        pdf = renderer.RenderHtmlAsPdf(self.get_pdf_content(quotes))
        pdf.SaveAs("quotes.pdf")

    def get_pdf_content(self, quotes):
        # Generate HTML content for PDF using extracted quotes
        html_content = "<html><head><title>Quotes</title></head><body>"
        for title, content in quotes:
            html_content += f"<h2>{title}</h2><p>Author: {content}</p>"
        html_content += "</body></html>"
        return html_content

In the above code example of a Scrapy project with IronPDF, IronPDF is being used to create a PDF document using the data that has been extracted using Scrapy.

Here, the spider's parse method gathers quotes from the webpage and uses the get_pdf_content function to create the HTML content for the PDF file. This HTML material is subsequently rendered as a PDF document using IronPDF and saved as quotes.pdf.