> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oculusproxies.com/llms.txt
> Use this file to discover all available pages before exploring further.

# How to Set Up Oculus Proxies With Scrapy

> Streamline your web scraping tasks! Integrating Oculus Proxies with Scrapy ensures secure, anonymous, and efficient data collection. This guide walks you through setting up Scrapy with Oculus Proxies step by step, enabling you to scrape websites seamlessly while avoiding IP bans.

<img src="https://mintcdn.com/oculus/4CVh05wC6-hsrqWV/integration-guides/img/oculus_integrations/Scrapy.png?fit=max&auto=format&n=4CVh05wC6-hsrqWV&q=85&s=60978d4adcfb5b95e222620eef51deac" alt="title" width="720" height="200" data-path="integration-guides/img/oculus_integrations/Scrapy.png" />

## What is Scrapy?

Scrapy is a powerful, open-source Python framework for web scraping and crawling. It automates the process of extracting data from websites and saves it in structured formats like JSON or CSV. With its asynchronous request handling, built-in support for proxies and cookies, and customizable spiders, Scrapy is a go-to tool for tasks such as:

* **Price Tracking**

* **Market Research**

* **Data Collection**

Scrapy’s spider components allow you to define how to crawl and scrape specific data from web pages, making it both flexible and scalable for various scraping needs.

<Tip>
  If you're using **Oculus** to access search engines like **Google, Bing, or Yandex** and facing connection issues, the proxy type could be the reason. **ISP Premium Proxies** ensure stable and unrestricted access, preventing blocks that standard proxies might encounter. Switching to **ISP Premium Proxies** can help maintain smooth and reliable performance.
</Tip>

## How to Set Up Oculus Proxies With Scrapy

<Steps>
  <Step title="Install Scrapy">
    Open your terminal and install Scrapy using `pip`:

    ```bash theme={null}
    pip install scrapy
    ```
  </Step>

  <Step title="Create a New Scrapy Project">
    1\. Start a new Scrapy project:

    ```bash theme={null}
    scrapy startproject <project_name>
    ```

    Replace `<project_name>` with your desired project name.

    2\. Navigate into the project directory:

    ```bash theme={null}
    cd <project_name>
    ```
  </Step>

  <Step title="Generate a New Spider">
    1\. Create a spider to scrape a specific website:

    ```bash theme={null}
    scrapy genspider <spider_name> <target_url>
    ```

    For example, to scrape [`http://httpbin.org/ip`](http://httpbin.org/ip), lets create a spider named `OculusExample`:

    ```bash theme={null}
    scrapy genspider OculusExample http://httpbin.org/ip
    ```

    2\. This will create a new spider file inside the `spiders/` directory.
  </Step>

  <Step title="Configure Oculus Proxy in Your Spider">
    Edit your newly created spider (`OculusExample.py`) and configure the proxy:

    ```python theme={null}
    import scrapy

    class OculusExampleSpider(scrapy.Spider):
        name = "OculusExample"
        start_urls = ['http://httpbin.org/ip']

        def start_requests(self):
            # Define the proxy
            proxy = "http://[USERNAME]:[PASSWORD]@[HOST]:[PORT]"  # Replace with your Oculus Proxy credentials
            
            # Use the proxy for all requests
            for url in self.start_urls:
                yield scrapy.Request(url, meta={'proxy': proxy})

        def parse(self, response):
            # Parse and return the IP address
            yield {
                'proxy_ip': response.text
            }
    ```

    <Tip>
      **For country-specific proxies, you can enter a format like `your-username-country-US` to receive a US exit node.**
    </Tip>
  </Step>

  <Step title="Run the Spider">
    1\. Navigate to your project directory and execute:

    ```bash theme={null}
    scrapy crawl OculusExample
    ```

    2\. To save the data to a file, use:

    ```bash theme={null}
    scrapy crawl OculusExample -o output.json
    ```
  </Step>

  <Step title="Verify the Output">
    When the spider runs successfully, it should return the IP address used by the proxy:

    ```json theme={null}
    [
        {
            "proxy_ip": "{\n  \"origin\": \"123.45.67.89\"\n}"
        }
    ]
    ```
  </Step>
</Steps>

**Congratulations!** You’ve successfully integrated **Oculus Proxies** with **Scrapy**. Now you can securely and efficiently scrape data while avoiding detection and IP bans.