Selected work

Project 4 of 4

Data engineering · Site-specific extraction · 2024–2025

Web Extraction Pipelines

14 site-specific spiders, with inspectable extraction and export methods.

Role
Site-specific extraction and processing
Deliverable
Spiders, source-text files, and JSON/CSV processing examples
Focus
Content selectors · source URLs · schema checks
RepositoryExternal context

Overview

The repository contains 14 non-initialiser spider files across college and university folders. Each site has its own extraction logic. The Canadore and Durham examples read URL lists, select page content with BeautifulSoup, and save source URLs alongside extracted text.

  • Python
  • Scrapy
  • BeautifulSoup
  • csv / json
  • Gemini API
Saved repository output.txt

Source, kept with the text.

An excerpt from a college-program extraction.

SOURCE URL

https://www.canadorecollege.ca/programs/part-time/accounting-foundations-certificate-program

Accounting Foundations Certificate Program

Code:   020110
Type:   Part-Time
Campus: Online
14 site-specific spider filesScrapy / BeautifulSoup
Fields from a saved extraction, with its original source URL.Inspect the output

Extraction source

Canadore course-content spider

969Furqan / public repository · GitHub source · Inspected October 2026

The source reads links.csv, skips 404 responses, selects the main content, sanitises filenames, and writes the source URL with the extracted text.

canadore_spider.py

Visit the product site

Scope

Work completed and source material.

Scope and evidence

Scope and contribution.

The evidence supports site-specific extraction methods and structured-output examples. ETL_Pipeline-1/readme.md credits asadali27232/ETL_Pipeline, so the included ETL code is not presented as an entirely original framework. The source does not establish 10,000 client records, anti-bot bypass, or universal pagination support.

  1. The repository tree contains 14 spider files excluding __init__.py.

  2. Canadore's spider handles missing pages, selects main content, and sanitises output filenames.

  3. Durham's spider combines banner and main-program content and saves the source URL with the text.

  4. ETL_Pipeline-1/json_to_csv.py rejects invalid JSON objects and mismatched field keys before writing ordered CSV rows.

  5. The ETL README links its external pipeline source; txt_to_json.py calls a Gemini summarisation model.

  6. Saved Canadore College text outputs retain the original URL alongside extracted program content. The portfolio excerpt reproduces the Accounting Foundations program fields from the inspected repository snapshot.

Problem

Main technical constraint.

Problem framing

Each website exposes useful content differently.

Course pages use different content containers and heading structures. Extraction needs site-specific selectors, traceable source URLs, safe filenames, and a defined set of fields for later CSV conversion.

Implementation

4 implementation steps.

Implementation

Keep the extraction method visible.

  1. Step1

    Read prepared URL lists and use a separate spider for each site's content structure.

  2. Step2

    Extract relevant text, retain the source URL, and sanitise filenames before writing files.

  3. Step3

    Keep included text-to-JSON examples attributed to the pipeline named in their README.

  4. Step4

    Check JSON object shape and expected keys before writing CSV fields in a defined order.

Result

Methods that can be inspected.

The public code demonstrates site-specific extraction and structured-output conversion. The 14-spider count is a repository fact; it is separate from any unverified client record volume or delivery-performance claim.

What I learned

Source URLs, exact selectors, and field checks provide concrete evidence of how data moves from a page into a structured file.