How to Use BeautifulSoup in n8n in 1 Minute (No Docker Required)

If you use n8n, you already know it’s one of the most powerful visual automation tools out there. But you probably also know the frustration of hitting its Python limitations.

The moment you need to run a web scraper using BeautifulSoup, process data with pandas, or fetch stocks with yfinance, you hit a wall. You either have to:

1. Build and maintain a custom Docker image to run pip install.

2. Fight with timeout limits and memory issues in the native Code node.

3. Spin up a separate VPS just to host a 20-line Python API.

That’s a lot of DevOps friction just to scrape a few headlines.

Today, I’m going to show you a completely serverless workaround that lets you write Python with external packages and drop it into your n8n canvas in under 60 seconds.

---

⚡️ The Workaround: Generating a Pasteable HTTP Node

Instead of forcing n8n to execute heavy Python environments locally, we are going to use a serverless engine called V-RUN to handle the heavy lifting, and pass the data to n8n via a simple clipboard trick.

Here is the 30-second proof of concept where I scrape the top posts from Hacker News and drop the data straight into n8n:

https://www.youtube.com/watch?v=bJCn4hJuKPg

---

🛠️ Step-by-Step Guide

Here is how you can replicate this workflow in under a minute:

Step 1: Write your Python script (or use AI)

Head over to V-RUN. You don't need to worry about requirements.txt or environments. Just write your code.

For example, a quick Hacker News scraper:

import requests
from bs4 import BeautifulSoup


def main(args):
    """Scrape top 5 posts from Hacker News."""
    url = "https://news.ycombinator.com/"
    
    try:
        # Fetch the Hacker News homepage
        headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
        }
        response = requests.get(url, headers=headers, timeout=10)
        response.raise_for_status()
        
        # Parse HTML
        soup = BeautifulSoup(response.content, "html.parser")
        
        # Find story rows (Hacker News uses .athing class for story rows)
        story_rows = soup.find_all("tr", class_="athing")
        
        posts = []
        for i, story in enumerate(story_rows[:5]):  # Get top 5
            # Extract title and link
            title_cell = story.find("span", class_="titleline")
            if title_cell:
                title_link = title_cell.find("a")
                if title_link:
                    title = title_link.get_text()
                    link = title_link.get("href", "")
                    
                    # Handle relative URLs
                    if link.startswith("item?"):
                        link = f"https://news.ycombinator.com/{link}"
                    
                    posts.append({
                        "rank": i + 1,
                        "title": title,
                        "url": link
                    })
        
        print(f"Successfully scraped {len(posts)} posts from Hacker News")
        return {
            "success": True,
            "posts": posts,
            "total": len(posts)
        }
    
    except requests.RequestException as e:
        print(f"Error fetching Hacker News: {e}")
        return {
            "success": False,
            "error": str(e),
            "posts": []
        }

(Pro-tip: If you are feeling lazy, V-RUN has a built-in AI called V-Dev that can write this exact code for you from a simple prompt).

Step 2: Click "Copy to n8n"

Once your code is ready, hit the "Copy to n8n" button on the platform.

This is where the magic happens. It doesn't copy the raw code; it generates the exact JSON structure that n8n uses for its nodes under the hood.

Step 3: Paste into n8n

Open your n8n workflow canvas and simply hit Ctrl + V (or Cmd + V).

Instantly, a fully configured HTTP Request Node will appear.

• The Webhook URL is automatically mapped to your V-RUN script.

• Headers and Authorization are securely pre-configured.

Step 4: Execute!

Click Test Node in n8n. V-RUN spins up a serverless environment in the background, installs BeautifulSoup, runs your code, and returns the perfectly formatted JSON data straight into your n8n workflow.

---

🚀 Why this approach wins

By isolating the Python execution from the n8n environment, you keep your n8n instance lightweight and fast. You get the full power of the Python ecosystem (pip) without touching a single terminal command or Dockerfile.

If you want to try this copy-paste Python workflow for your own automations, you can try the free tier at V-Run.

Happy automating! 🤖