Building an AI-Powered Web Scraping & Report Generation System

A comprehensive look at the engineering challenges and solutions behind combining async web crawling, document parsing, and LLM integration in a Streamlit application.

Tech Stack at a Glance

UI Framework

Streamlit 1.51

Web Scraping

Crawl4AI 0.7.6

Video Transcripts

youtube-transcript-api 1.2.3

LLM

Google Gemini 0.8.5

Document Parsing

PyPDF2, python-docx

Browser Automation

Playwright 1.55.0

Challenge #1: Async Operations Inside Streamlit's Sync Context

The Problem

Crawl4AI is built on async/await patterns for efficient I/O operations. However, Streamlit runs in a synchronous context that may already have an event loop running. Calling asyncio.run() inside an existing loop raises:

RuntimeError: This event loop is already running

The Solution: Thread Delegation with Queue-Based Communication

Python
def run(url: str) -> str:
    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        loop = None

    if loop and loop.is_running():
        # Already in an async context - delegate to a separate thread
        q: queue.Queue[tuple[bool, object]] = queue.Queue()

        def worker():
            new_loop = asyncio.new_event_loop()
            asyncio.set_event_loop(new_loop)
            try:
                res = new_loop.run_until_complete(scrape(url))
                q.put((True, res))
            except Exception as e:
                q.put((False, e))
            finally:
                new_loop.close()

        t = threading.Thread(target=worker, daemon=True)
        t.start()
        ok, val = q.get()  # Block until result ready
        if ok:
            return val
        raise val
    else:
        # No existing loop - safe to use asyncio.run()
        return asyncio.run(scrape(url))
Thread isolation

Each async operation gets its own event loop in a dedicated thread

Queue-based result passing

Thread-safe communication without shared mutable state

Daemon threads

Auto-cleanup on application exit

Tuple unpacking for errors

Preserves exception type and traceback

Windows-Specific Event Loop Policy

Windows requires special handling for async subprocess operations:

asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())

The ProactorEventLoop is required on Windows for proper subprocess and pipe handling that Crawl4AI uses internally.

Challenge #2: Unified Content Extraction from Heterogeneous Sources

The Problem

Different source types require different extraction strategies:

Web URLs

Need headless browser rendering for JavaScript-heavy sites

YouTube URLs

Standard scraping fails—need transcript API

PDFs

Binary format requiring specialized parsing

DOCX

XML-based format with paragraph structure

The Solution: Smart Routing with Fallback Chains

Python - Smart URL Routing
async def scrape(url: str) -> str:
    # Route based on URL pattern
    if _is_youtube_url(url):
        vid = _extract_youtube_video_id(url)
        if vid:
            try:
                return _fetch_youtube_transcript_text(vid)
            except Exception:
                pass  # Fallback to web crawling
    
    # Default: headless browser crawl
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url)
        return result.markdown or ""

YouTube URL Detection Flow

Input URL
↓
Is YouTube URL?
Yes
↓
Extract Video ID
↓
youtu.be/ID
?v=ID
/shorts/ID
/embed/ID
↓
Fetch Transcript
No
↓
Web Crawler

Challenge #3: LLM API Reliability and Model Fallback

The Problem

LLM APIs can fail for various reasons. A single API call failure shouldn't break the entire workflow.

Rate limitingModel deprecationTemporary outagesQuota exhaustion

The Solution: Cascading Model Fallback

Python - Cascading Fallback
def summarize_with_gemini(text: str, prompt: str | None = None) -> str:
    genai.configure(api_key=api_key)
    
    # Ordered by preference: latest → stable fallbacks
    choices = [
        "gemini-2.5-flash",
        "gemini-2.0-flash",
        "gemini-2.5-flash-lite",
        "gemini-1.5-flash",
        "gemini-1.5-pro",
    ]
    
    last_err: Exception | None = None
    for name in choices:
        try:
            model = genai.GenerativeModel(name)
            resp = model.generate_content(full_prompt)
            
            # Handle different response structures
            if hasattr(resp, "text") and resp.text:
                return resp.text
            ...
        except Exception as e:
            last_err = e
            continue  # Try next model
    
    if last_err:
        raise last_err
    return ""

Engineering Considerations

  • Response structure handling: Gemini SDK response objects vary—defensive attribute access required
  • Preserving the last error: If all models fail, raise the most recent exception for debugging
  • Order matters: Prefer faster/cheaper models first, fall back to more capable ones

Challenge #4: Safe Filename Generation from URLs

The Problem

URLs contain characters that are invalid in filenames across different operating systems. Additionally, different URLs might generate the same filename, causing data overwrites.

/\?&=:

The Solution: Slugification with Hash-Based Uniqueness

1
Parse URLhost + path
→
2
SlugifyInvalid → _
→
3
Add HashSHA1[:8]
→
4
TruncateMax 64 chars

Why SHA1?

Fast computation
8-character prefix provides ~4 billion unique values
Full URL is hashed, preserving query string uniqueness

Challenge #5: Markdown to DOCX Conversion with Formatting Preservation

The Problem

Enterprise users need Word documents, but LLM output is Markdown. The conversion must preserve all formatting elements.

H1-H6

Heading hierarchy

BI

Bold & Italic

`code`

Inline code

• —

Lists

```

Code blocks

The Solution: Regex-Based Markdown Parser

Python - Inline Markdown Processing
def _add_inline_markdown(paragraph, text: str):
    """Process inline markdown: **bold**, *italic*, `code`"""
    pattern = re.compile(r"(\*\*[^*]+\*\*|\*[^*]+\*|`[^`]+`)")
    pos = 0
    
    for m in pattern.finditer(text):
        # Add plain text before match
        if m.start() > pos:
            paragraph.add_run(text[pos:m.start()])
        
        token = m.group(0)
        if token.startswith("**"):
            r = paragraph.add_run(token[2:-2])
            r.bold = True
        elif token.startswith("*"):
            r = paragraph.add_run(token[1:-1])
            r.italic = True
        elif token.startswith("`"):
            r = paragraph.add_run(token[1:-1])
            r.font.name = "Courier New"
        
        pos = m.end()
    
    # Add remaining text
    if pos < len(text):
        paragraph.add_run(text[pos:])

Challenge #6: State Management and Data Isolation

The Problem

Multiple users share the same application instance. Each user's data must be isolated, persisted across sessions, and efficiently queryable.

The Solution: TSV-Based Index Files with User ID Column

Index File Structure

URL\tFilename\tTimestamp\tUserID\tUploadFlag\tTitle

System Architecture

STREAMLIT UI (app.py)
Home (Extract)
Extracted Data
Build Report (Synthesis)
↓
CORE ENGINE (main.py)
ASYNC SCRAPING LAYER
Crawl4AI (Browser)
YouTube Transcript API
Thread Delegation
DOCUMENT PARSING LAYER
PyPDF2 (PDF)
python-docx
Plain Text/Markdown
LLM INTEGRATION LAYER
Gemini API → Model Fallback Chain → Response Normalization
PERSISTENCE LAYER
extracted_data/
processed_data/
generated_reports/

Key Takeaways

ChallengeSolutionKey Technique
Async in sync contextThread delegationthreading.Thread + queue.Queue
Heterogeneous sourcesSmart routing with fallbackURL pattern matching + exception handling
LLM reliabilityModel cascadeOrdered fallback with preserved errors
Safe filenamesSlugification + hashingRegex sanitization + SHA1 prefix
Markdown → DOCXLine-by-line parsingRegex pattern matching + python-docx
User isolationIndex file filteringTSV with UserID column

Conclusion

Building a production-ready web scraping and LLM integration system involves solving numerous edge cases and platform-specific issues. The key patterns demonstrated here—thread-based async isolation, cascading fallbacks, and defensive data handling—are applicable to any Python application integrating external APIs with complex I/O patterns.

The combination of Streamlit for rapid prototyping, Crawl4AI for robust web scraping, and Gemini for intelligent summarization creates a powerful foundation that can be extended to various domains beyond financial analysis.

Built by Flairminds — Turning complex data pipelines into simple workflows.

#WebScraping#AI#LLM#Python#Streamlit#Automation