# Web Scraping with Python Using Regular Expressions

## Introduction

Web scraping is a technique used to extract data from websites automatically. Python provides several libraries for web scraping, and one of the most powerful tools is regular expressions (regex). In this tutorial, we will explore how to scrape web pages using Python and regular expressions.

## Prerequisites

To follow along with this tutorial, you should have a basic understanding of Python programming and some knowledge of HTML structure.

## Step 1: Installing Dependencies

Before we start, we need to install the necessary libraries. Open your terminal or command prompt and execute the following command:

```plaintext
pip install requests beautifulsoup4
```

## Step 2: Importing Required Libraries

Let's begin by importing the libraries we will be using: `requests`, `re`, and `BeautifulSoup`. The `requests` library helps us send HTTP requests to websites, `re` is the regular expressions library, and `BeautifulSoup` allows us to parse HTML documents.

```python
import requests
import re
from bs4 import BeautifulSoup
```

## Step 3: Sending a Request

To scrape a web page, we first need to send an HTTP request to the website. We can do this using the `requests.get()` method. Let's retrieve the HTML content of a web page:

```python
url = 'https://example.com'
response = requests.get(url)
html_content = response.text
```

## Step 4: Parsing HTML with BeautifulSoup

Now that we have obtained the HTML content of the webpage, we need to parse it using BeautifulSoup. This will allow us to extract specific elements from the HTML structure.

```python
soup = BeautifulSoup(html_content, 'html.parser')
```

## Step 5: Using Regular Expressions for Scraping

Regular expressions provide a powerful way to search, match, and extract data from text. We can utilize regex patterns to extract specific information from the HTML content. Let's see some examples.

### Example 1: Extracting Email Addresses

Suppose we want to extract all the email addresses from the web page. We can use a regex pattern to achieve this:

```python
email_pattern = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
emails = re.findall(email_pattern, html_content)
print(emails)
```

### Example 2: Extracting URLs

To extract all the URLs from the web page, we can use the following regex pattern:

```python
url_pattern = r'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+'
urls = re.findall(url_pattern, html_content)
print(urls)
```

## Step 6: Cleaning and Processing the Extracted Data

After extracting the desired data using regular expressions, you might need to clean or process it further. You can iterate over the extracted data and apply additional regex patterns or string manipulation techniques to refine the results.

## Conclusion

In this tutorial, we learned how to perform web scraping using Python and regular expressions. We covered the basics of sending HTTP requests, parsing HTML content with BeautifulSoup, and using regex patterns to extract specific information.

Remember to respect website terms of service and be mindful of the legality and ethics of web scraping in your use case.