Web Scraping Using Python

Performing a Data collection using Web Scrapping:

What is Web Scraping ?

Web scraping is the process of using Bots to extract contents and data from a website. Unlike screen scraping, which only copies pixels displayed onscreen, web scraping extracts underlying HTML code and, with it, data stored in a database. The scraper can then replicate entire website content elsewhere.

Python libraries use for Web scraping are as follow:

There are many different libraries available in python for web scrapping, but here we have used Requests, BeautifulSoup and Pandas.

1. Requests: It allows you to send HTTP/1.1 requests with ease and it does not require to manually add query strings to your URLs, or to form-encode your POST data.
2. Beautiful Soup : is used for web scraping purposes to pull the data out of HTML and XML files. It creates a parse tree from page source code that can be used to extract data in a hierarchical and more readable manner.
3. Pandas: Pandas is mainly used for data analysis. Pandas allows importing data from various file formats such as comma-separated values, JSON, SQL, Microsoft Excel. Pandas allows various data manipulation operations such as merging, reshaping, selecting, as well as data cleaning, and data wrangling features.

The following Steps are required to extract data by Web Scraping.

1. Find the URL that you want to scrape.

2. Inspecting the Page.

3. Find the data you want to extract.

4. Write the code.

5. Run the code and extract the data.

6. Store the data in the required format.

Import the required python libraries:

Create Empty Variable to store Scraped Data

Now enter the url from where you want the data. Request the library that is used to make html request to server.

Separate specific data from content using respective class-names and respective tags and then store that data into list.

Now using Pandas Libary, create a dataframe in which the data is stored in structured way so that you can export it into the desired file format. Here I have exported the data in .csv format.

18IT134_P1.csv file.

This is a basic program to perform web scraping. By performing this, you get to learn how to scrape data from the internet and format it for further analysis.