+
+
+
![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jEkuKbcRrPcK"
+ },
+ "source": [
+ "## 🕷️ Chat with your webpage with `scrapegraph`, `burr` and `lancedb`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "dsJlQnfA60D0"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IzsyDXEWwPVt"
+ },
+ "source": [
+ "### 🔧 Install `dependencies`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "id": "os_vm0MkIxr9"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!pip install scrapegraph-py burr[start,opentelemetry] lancedb openai opentelemetry-instrumentation-openai opentelemetry-instrumentation-lancedb"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "apBsL-L2KzM7"
+ },
+ "source": [
+ "### 🔑 Import `ScrapeGraph` and `OpenAI` API keys"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ol9gQbAFkh9b"
+ },
+ "source": [
+ "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "sffqFG2EJ8bI",
+ "outputId": "bd2fbb27-980c-420f-ef50-fc1f23bdae0a"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Scrapegraph API key:\n",
+ "··········\n",
+ "OpenAI API key:\n",
+ "··········\n"
+ ]
+ }
+ ],
+ "source": [
+ "import getpass\n",
+ "import os\n",
+ "\n",
+ "if not os.environ.get(\"SGAI_API_KEY\"):\n",
+ " os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")\n",
+ "\n",
+ "if not os.environ.get(\"OPENAI_API_KEY\"):\n",
+ " os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API key:\\n\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "cDGH0b2DkY63"
+ },
+ "source": [
+ "### 🚀 Define the extraction and query flow using `burr` and run it"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "`burr` is an open-source `orchestrator framework` that makes it easy to develop applications that make decisions (chatbots, agents, simulations, etc...). It also features a cool self-hosted UI to trace what's happening in the application.\n",
+ "\n",
+ "> Check the [Github Repo](https://github.com/DAGWorks-Inc/burr)"
+ ],
+ "metadata": {
+ "id": "M3gmZCI0eqS1"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Our goal is to define a flow (DAG) that:\n",
+ "\n",
+ "1. Fetches markdown from webpages ([scrapegraph](https://scrapegraphai.com/))\n",
+ "2. Chunks the content and stores it in a vector store ([lancedb](https://lancedb.com/))\n",
+ "3. Allows to query the db and generate an answer using a LLM\n",
+ "\n",
+ "We can see all of this as *Nodes* connected together in a *Graph*, where the *Nodes* are the `actions` we want to perform.\n",
+ "\n",
+ "In burr we define actions by simply adding the `action decorator` to each function and specifying what that function needs to `read` and `write` from the graph's state.\n"
+ ],
+ "metadata": {
+ "id": "359QH2__gAZo"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "All imports"
+ ],
+ "metadata": {
+ "id": "2aWKZrJgijJR"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from burr.core import action, State, ApplicationBuilder\n",
+ "\n",
+ "from scrapegraph_py import Client\n",
+ "from scrapegraph_py.logger import sgai_logger\n",
+ "\n",
+ "import re\n",
+ "import lancedb\n",
+ "from lancedb.pydantic import LanceModel, Vector\n",
+ "from lancedb.embeddings import get_registry\n",
+ "\n",
+ "import openai\n",
+ "import tiktoken\n",
+ "from typing import List, Optional\n",
+ "\n",
+ "from opentelemetry.instrumentation.lancedb import LanceInstrumentor\n",
+ "from opentelemetry.instrumentation.openai import OpenAIInstrumentor"
+ ],
+ "metadata": {
+ "id": "HeYtTm0DigBM"
+ },
+ "execution_count": 66,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "#### 🔍 Define `fetch_webpage` action to fetch and convert a webpage into `markdown`"
+ ],
+ "metadata": {
+ "id": "i1Ei6gDTdzYW"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "M1KSXffZopUD"
+ },
+ "source": [
+ "Here we use `markdownify` to fetch a webpage and convert it into markdown format, which is suitable to LLM ingestion.\n",
+ "\n",
+ "You can find more info in the [official scrapegraph documentation](https://docs.scrapegraphai.com/services/markdownify)\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "sgai_logger.set_logging(level=\"INFO\") #Optional\n",
+ "\n",
+ "# Define scrapegraph sync client (API KEY is a env variable)\n",
+ "sgai_client = Client()\n"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "JS3DZIAhbjD1",
+ "outputId": "de22fff8-96b1-489f-bbd7-e2eef24c99c5"
+ },
+ "execution_count": 67,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "💬 2024-12-29 13:56:08,576 🔑 Initializing Client\n",
+ "INFO:scrapegraph:🔑 Initializing Client\n",
+ "💬 2024-12-29 13:56:08,582 ✅ Client initialized successfully\n",
+ "INFO:scrapegraph:✅ Client initialized successfully\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 68,
+ "metadata": {
+ "id": "ySoE0Rowjgp1"
+ },
+ "outputs": [],
+ "source": [
+ "@action(reads=[], writes=[\"markdown_content\"])\n",
+ "def fetch_webpage(state: State, webpage_url: str) -> State:\n",
+ " \"\"\"Fetch a webpage and convert it to Markdown.\"\"\"\n",
+ " try:\n",
+ " response = sgai_client.markdownify(website_url=webpage_url)\n",
+ " markdown_content = response[\"result\"]\n",
+ " print(f\"Request ID: {response['request_id']}\")\n",
+ " print(f\"Markdown Content: {markdown_content[:200]}... (truncated)\")\n",
+ " return state.update(markdown_content=markdown_content)\n",
+ " except Exception as e:\n",
+ " sgai_logger.error(f\"Failed to fetch webpage: {e}\")\n",
+ " raise"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "#### 📁 Define `embed_and_store` action to chunk the markdown and store it in a local vector store"
+ ],
+ "metadata": {
+ "id": "BNXvZkkLsOed"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Define the data structure to hold the chunks in the `lancedb` vector store"
+ ],
+ "metadata": {
+ "id": "zj-fQ7rbt4uc"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "embedding_model = get_registry().get(\"openai\").create()\n",
+ "\n",
+ "class TextDocument(LanceModel):\n",
+ " \"\"\"Simple data structure to hold a piece of text associated with a url.\"\"\"\n",
+ " url: str\n",
+ " position: int\n",
+ " text: str = embedding_model.SourceField()\n",
+ " vector: Vector(dim=embedding_model.ndims()) = embedding_model.VectorField()"
+ ],
+ "metadata": {
+ "id": "dqVyq0ONl6DY"
+ },
+ "execution_count": 69,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Utils to create chunks based on the `number of tokens`"
+ ],
+ "metadata": {
+ "id": "R97nRo-EtsGP"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Constants\n",
+ "CHUNK_SIZE = 2000 # Target size of each chunk in tokens\n",
+ "MIN_CHUNK_LENGTH_TO_EMBED = 5 # Discard chunks shorter than this\n",
+ "\n",
+ "# Initialize the tokenizer\n",
+ "tokenizer = tiktoken.get_encoding(\"cl100k_base\")\n",
+ "\n",
+ "def get_text_chunks(text: str, chunk_token_size: Optional[int] = CHUNK_SIZE) -> List[str]:\n",
+ " \"\"\"\n",
+ " Split text into chunks of approximately `chunk_token_size` tokens.\n",
+ "\n",
+ " Args:\n",
+ " text: The input text to chunk.\n",
+ " chunk_token_size: Target size of each chunk in tokens.\n",
+ "\n",
+ " Returns:\n",
+ " A list of text chunks.\n",
+ " \"\"\"\n",
+ " if not text.strip():\n",
+ " return []\n",
+ "\n",
+ " tokens = tokenizer.encode(text)\n",
+ " chunks = []\n",
+ " chunk_size = chunk_token_size or CHUNK_SIZE\n",
+ "\n",
+ " while tokens:\n",
+ " # Extract chunk tokens and decode to text\n",
+ " chunk_tokens = tokens[:chunk_size]\n",
+ " chunk_text = tokenizer.decode(chunk_tokens).strip()\n",
+ "\n",
+ " # Skip chunks that are too short\n",
+ " if len(chunk_text) > MIN_CHUNK_LENGTH_TO_EMBED:\n",
+ " chunks.append(chunk_text)\n",
+ "\n",
+ " # Remove processed tokens\n",
+ " tokens = tokens[len(chunk_tokens):]\n",
+ "\n",
+ " return chunks"
+ ],
+ "metadata": {
+ "id": "OfFv8Zb9qaTD"
+ },
+ "execution_count": 52,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's define the action. It creates a webpages `local vector store` if not present and add the chunks to the `chunks table`"
+ ],
+ "metadata": {
+ "id": "Ep4THhetuTey"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "@action(reads=[\"markdown_content\"], writes=[])\n",
+ "def embed_and_store(state: State, webpage_url: str) -> State:\n",
+ " \"\"\"Embed and store the Markdown content.\"\"\"\n",
+ " markdown_content = state[\"markdown_content\"]\n",
+ "\n",
+ " chunks = get_text_chunks(markdown_content)\n",
+ "\n",
+ " # Embed and store the chunks using LanceDB\n",
+ " con = lancedb.connect(\"./webpages\")\n",
+ " table = con.create_table(\"chunks\", exist_ok=True, schema=TextDocument)\n",
+ " table.add([\n",
+ " {\"text\": c, \"url\": webpage_url, \"position\": i}\n",
+ " for i, c in enumerate(chunks)\n",
+ " ])\n",
+ "\n",
+ " return state"
+ ],
+ "metadata": {
+ "id": "RMk8gjUtk0gL"
+ },
+ "execution_count": 53,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "#### 💬 Define `ask_question` action to retrieve the most relevant chunks from the vector store and query them with a llm"
+ ],
+ "metadata": {
+ "id": "zbP0km8BtN73"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Fetches the first 3 relevant chunks based on the user query and generate and answer"
+ ],
+ "metadata": {
+ "id": "HTShnZowukj8"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "@action(reads=[], writes=[\"llm_answer\"])\n",
+ "def ask_question(state: State, user_query: str) -> State:\n",
+ " \"\"\"Reply to the user's query using the webpage's content.\"\"\"\n",
+ " # Retrieve the most relevant chunks\n",
+ " chunks_table = lancedb.connect(\"./webpages\").open_table(\"chunks\")\n",
+ " search_results = (\n",
+ " chunks_table\n",
+ " .search(user_query)\n",
+ " .select([\"text\", \"url\", \"position\"])\n",
+ " .limit(3)\n",
+ " .to_list()\n",
+ " )\n",
+ " relevant_content = \"\\n\".join([r[\"text\"] for r in search_results])\n",
+ "\n",
+ " # Prompt the LLM with the relevant content\n",
+ " system_prompt = (\n",
+ " \"Answer the user's questions based on the provided webpage content. \"\n",
+ " f\"WEBPAGE CONTENT:\\n{relevant_content}\"\n",
+ " )\n",
+ "\n",
+ " client = openai.OpenAI()\n",
+ " response = client.chat.completions.create(\n",
+ " model=\"gpt-4o-mini\",\n",
+ " messages=[\n",
+ " {\"role\": \"system\", \"content\": system_prompt},\n",
+ " {\"role\": \"user\", \"content\": user_query}\n",
+ " ],\n",
+ " )\n",
+ " llm_answer = response.choices[0].message.content\n",
+ " return state.update(llm_answer=llm_answer)"
+ ],
+ "metadata": {
+ "id": "VXHbMtHxldKQ"
+ },
+ "execution_count": 54,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "#### 🤖 Define the burr `application graph` and run it"
+ ],
+ "metadata": {
+ "id": "OOnKVmJTvORy"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Instrumenting OpenTelemetry for LanceDB and OpenAI\n",
+ "OpenAIInstrumentor().instrument()\n",
+ "LanceInstrumentor().instrument()"
+ ],
+ "metadata": {
+ "id": "PL9oRtxpnYHy"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Define the application graph\n",
+ "app = (\n",
+ " ApplicationBuilder()\n",
+ " .with_actions(fetch_webpage=fetch_webpage, embed_and_store=embed_and_store, ask_question=ask_question)\n",
+ " .with_transitions((\"fetch_webpage\", \"embed_and_store\"), (\"embed_and_store\", \"ask_question\"))\n",
+ " .with_entrypoint(\"fetch_webpage\")\n",
+ " .with_tracker(project=\"chat-webpage-simple-rag\", use_otel_tracing=True)\n",
+ " .build()\n",
+ ")"
+ ],
+ "metadata": {
+ "id": "W3f0m3CCm50A"
+ },
+ "execution_count": 63,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Run the application graph\n",
+ "action_name, results, state = app.run(\n",
+ " halt_after=[\"ask_question\"],\n",
+ " inputs={\n",
+ " \"webpage_url\": \"https://scrapegraphai.com/\",\n",
+ " \"user_query\": \"Who are the founders?\"\n",
+ " }\n",
+ ")"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "TRPUyVtbnGBZ",
+ "outputId": "9ddfbfbf-60fe-4592-aee7-caba292ce2eb"
+ },
+ "execution_count": 64,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "💬 2024-12-29 13:51:54,568 🔍 Starting markdownify request for https://scrapegraphai.com/\n",
+ "INFO:scrapegraph:🔍 Starting markdownify request for https://scrapegraphai.com/\n",
+ "💬 2024-12-29 13:51:54,577 🚀 Making POST request to https://api.scrapegraphai.com/v1/markdownify\n",
+ "INFO:scrapegraph:🚀 Making POST request to https://api.scrapegraphai.com/v1/markdownify\n",
+ "💬 2024-12-29 13:51:57,646 ✅ Request completed successfully: POST https://api.scrapegraphai.com/v1/markdownify\n",
+ "INFO:scrapegraph:✅ Request completed successfully: POST https://api.scrapegraphai.com/v1/markdownify\n",
+ "💬 2024-12-29 13:51:57,649 ✨ Markdownify request completed successfully\n",
+ "INFO:scrapegraph:✨ Markdownify request completed successfully\n"
+ ]
+ },
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Request ID: d646737f-2dbd-4c6d-aecb-fc5f2c3e132d\n",
+ "Markdown Content: [Star us on GitHub0](https://github.com/ScrapeGraphAI/Scrapegraph-ai)\n",
+ "\n",
+ "## Transform Websites into \n",
+ "Structured Data\n",
+ "\n",
+ "### Just One Prompt Away\n",
+ "\n",
+ "Transform any website into clean, organized data for AI a... (truncated)\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "print(state[\"llm_answer\"])"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "j1Ht3OCdozIY",
+ "outputId": "fc4ff9c3-1675-447e-e8f8-b228731de9fc"
+ },
+ "execution_count": 65,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "The founders of ScrapeGraphAI are:\n",
+ "\n",
+ "1. **Marco Perini** - Founder & Technical Lead\n",
+ " - [LinkedIn profile of Marco Perini](https://www.linkedin.com/in/perinim/)\n",
+ "\n",
+ "2. **Marco Vinciguerra** - Founder & Software Engineer\n",
+ " - [LinkedIn profile of Marco Vinciguerra](https://www.linkedin.com/in/marco-vinciguerra-7ba365242/)\n",
+ "\n",
+ "3. **Lorenzo Padoan** - Founder & Product Engineer\n",
+ " - [LinkedIn profile of Lorenzo Padoan](https://www.linkedin.com/in/lorenzo-padoan-4521a2154)\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "2as65QLypwdb"
+ },
+ "source": [
+ "### 🖼️ Visualize the traces with `Burr UI`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# execute to load the Burr extension\n",
+ "%load_ext burr.integrations.notebook"
+ ],
+ "metadata": {
+ "id": "C_amp8mOwcx3"
+ },
+ "execution_count": 70,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "%burr_ui\n",
+ "\n",
+ "# If you want to open the ui in another window\n",
+ "# from google.colab import output\n",
+ "# output.serve_kernel_port_as_window(7241)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 421
+ },
+ "id": "6mr_3Mp5wgrL",
+ "outputId": "567a1499-346d-416f-c4ca-9045bb7bb26b"
+ },
+ "execution_count": 74,
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "\n",
+ " ![\"ScrapeGraph](\"https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png\"\n",)
\n",
+ "
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️ Extract Company Info with langchain-scrapegraph"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"8JeaikoxiiLZ"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install langchain-scrapegraph"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734622068116,"user_tz":-60,"elapsed":2038,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"14655445-fe17-4d6d-e09e-f198f3d0aad3"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | company_name | \n","description | \n","logo | \n","contact_emails | \n","privacy_policy | \n","terms_of_service | \n","api_status | \n","github | \n","||
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n","ScrapeGraphAI | \n","ScrapeGraphAI is a powerful AI scraping API de... | \n","https://scrapegraphai.com/images/scrapegraphai... | \n","contact@scrapegraphai.com | \n","https://scrapegraphai.com/privacy | \n","https://scrapegraphai.com/terms | \n","https://scrapegraphapi.openstatus.dev | \n","https://www.linkedin.com/company/101881123 | \n","https://x.com/scrapegraphai | \n","https://github.com/ScrapeGraphAI/Scrapegraph-ai | \n","
| \n"," | name | \n","role | \n","|
|---|---|---|---|
| 0 | \n","Marco Perini | \n","Founder & Technical Lead | \n","https://www.linkedin.com/in/perinim/ | \n","
| 1 | \n","Marco Vinciguerra | \n","Founder & Software Engineer | \n","https://www.linkedin.com/in/marco-vinciguerra-... | \n","
| 2 | \n","Lorenzo Padoan | \n","Founder & Product Engineer | \n","https://www.linkedin.com/in/lorenzo-padoan-452... | \n","
| \n"," | tier | \n","price | \n","credits | \n","
|---|---|---|---|
| 0 | \n","Free | \n","$0 | \n","100 | \n","
| 1 | \n","Starter | \n","$20/month | \n","5000 | \n","
| 2 | \n","Growth | \n","$100/month | \n","40000 | \n","
| 3 | \n","Pro | \n","$500/month | \n","250000 | \n","
| \n"," | partner | \n","
|---|---|
| 0 | \n","PostHog | \n","
| 1 | \n","AWS | \n","
| 2 | \n","NVIDIA | \n","
| 3 | \n","JinaAI | \n","
| 4 | \n","DagWorks | \n","
| 5 | \n","Browserbase | \n","
| 6 | \n","ScrapeDo | \n","
| 7 | \n","HackerNews | \n","
| 8 | \n","Medium | \n","
| 9 | \n","HackADay | \n","
\n"," ![\"ScrapeGraph](\"https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png\"\n",")
\n","
\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jEkuKbcRrPcK"
+ },
+ "source": [
+ "## 🕷️ Extract Company Info with `llama-index-tools-scrapegraphai`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "RaHnNF9bYX8N"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IzsyDXEWwPVt"
+ },
+ "source": [
+ "### 🔧 Install `dependencies`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "os_vm0MkIxr9"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!pip install llama-index-tools-scrapegraphai"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "apBsL-L2KzM7"
+ },
+ "source": [
+ "### 🔑 Import `ScrapeGraph` API key"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ol9gQbAFkh9b"
+ },
+ "source": [
+ "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "sffqFG2EJ8bI"
+ },
+ "outputs": [],
+ "source": [
+ "import getpass\n",
+ "import os\n",
+ "\n",
+ "if not os.environ.get(\"SGAI_API_KEY\"):\n",
+ " os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jnqMB2-xVYQ7"
+ },
+ "source": [
+ "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VZvxbjfXvbgd"
+ },
+ "source": [
+ "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n",
+ "\n",
+ "| \n", + " | company_name | \n", + "description | \n", + "logo | \n", + "contact_emails | \n", + "privacy_policy | \n", + "terms_of_service | \n", + "api_status | \n", + "github | \n", + "||
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "ScrapeGraphAI | \n", + "ScrapeGraphAI is a powerful AI scraping API de... | \n", + "https://scrapegraphai.com/images/scrapegraphai... | \n", + "contact@scrapegraphai.com | \n", + "https://scrapegraphai.com/privacy | \n", + "https://scrapegraphai.com/terms | \n", + "https://scrapegraphapi.openstatus.dev | \n", + "https://www.linkedin.com/company/101881123 | \n", + "https://x.com/scrapegraphai | \n", + "https://github.com/ScrapeGraphAI/Scrapegraph-ai | \n", + "
| \n", + " | name | \n", + "role | \n", + "|
|---|---|---|---|
| 0 | \n", + "Marco Perini | \n", + "Founder & Technical Lead | \n", + "https://www.linkedin.com/in/perinim/ | \n", + "
| 1 | \n", + "Marco Vinciguerra | \n", + "Founder & Software Engineer | \n", + "https://www.linkedin.com/in/marco-vinciguerra-... | \n", + "
| 2 | \n", + "Lorenzo Padoan | \n", + "Founder & Product Engineer | \n", + "https://www.linkedin.com/in/lorenzo-padoan-452... | \n", + "
| \n", + " | tier | \n", + "price | \n", + "credits | \n", + "
|---|---|---|---|
| 0 | \n", + "Free | \n", + "$0 | \n", + "100 | \n", + "
| 1 | \n", + "Starter | \n", + "$20/month | \n", + "5000 | \n", + "
| 2 | \n", + "Growth | \n", + "$100/month | \n", + "40000 | \n", + "
| 3 | \n", + "Pro | \n", + "$500/month | \n", + "250000 | \n", + "
| \n", + " | partner | \n", + "
|---|---|
| 0 | \n", + "PostHog | \n", + "
| 1 | \n", + "AWS | \n", + "
| 2 | \n", + "NVIDIA | \n", + "
| 3 | \n", + "JinaAI | \n", + "
| 4 | \n", + "DagWorks | \n", + "
| 5 | \n", + "Browserbase | \n", + "
| 6 | \n", + "ScrapeDo | \n", + "
| 7 | \n", + "HackerNews | \n", + "
| 8 | \n", + "Medium | \n", + "
| 9 | \n", + "HackADay | \n", + "
\n",
+ " \n",
+ "
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️ Extract Company Info with Official Scrapegraph SDK"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"3Q5VM3SsRlxO"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install scrapegraph-py"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734532300517,"user_tz":-60,"elapsed":6877,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"f6b837cd-0f00-49cc-cb6f-f2bca57544f5"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | company_name | \n","description | \n","logo | \n","contact_emails | \n","privacy_policy | \n","terms_of_service | \n","api_status | \n","github | \n","||
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n","ScrapeGraphAI | \n","ScrapeGraphAI is a powerful AI scraping API de... | \n","https://scrapegraphai.com/images/scrapegraphai... | \n","contact@scrapegraphai.com | \n","https://scrapegraphai.com/privacy | \n","https://scrapegraphai.com/terms | \n","https://scrapegraphapi.openstatus.dev | \n","https://www.linkedin.com/company/101881123 | \n","https://x.com/scrapegraphai | \n","https://github.com/ScrapeGraphAI/Scrapegraph-ai | \n","
| \n"," | name | \n","role | \n","|
|---|---|---|---|
| 0 | \n","Marco Perini | \n","Founder & Technical Lead | \n","https://www.linkedin.com/in/perinim/ | \n","
| 1 | \n","Marco Vinciguerra | \n","Founder & Software Engineer | \n","https://www.linkedin.com/in/marco-vinciguerra-... | \n","
| 2 | \n","Lorenzo Padoan | \n","Founder & Product Engineer | \n","https://www.linkedin.com/in/lorenzo-padoan-452... | \n","
| \n"," | tier | \n","price | \n","credits | \n","
|---|---|---|---|
| 0 | \n","Free | \n","$0 | \n","100 | \n","
| 1 | \n","Starter | \n","$20/month | \n","5000 | \n","
| 2 | \n","Growth | \n","$100/month | \n","40000 | \n","
| 3 | \n","Pro | \n","$500/month | \n","250000 | \n","
| \n"," | partner | \n","
|---|---|
| 0 | \n","PostHog | \n","
| 1 | \n","AWS | \n","
| 2 | \n","NVIDIA | \n","
| 3 | \n","JinaAI | \n","
| 4 | \n","DagWorks | \n","
| 5 | \n","Browserbase | \n","
| 6 | \n","ScrapeDo | \n","
| 7 | \n","HackerNews | \n","
| 8 | \n","Medium | \n","
| 9 | \n","HackADay | \n","
\n"," \n","
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️🦜 Extract Github Trending Repositories with langchain-scrapegraph\n"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"IhozYNwsgJzt"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install langchain-scrapegraph"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734619154386,"user_tz":-60,"elapsed":4395,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"07af4bbe-c226-4fb3-8f68-7ccd429f59cc"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | name | \n","description | \n","stars | \n","forks | \n","today_stars | \n","language | \n","
|---|---|---|---|---|---|---|
| 0 | \n","XiaoMi/ha_xiaomi_home | \n","Xiaomi Home Integration for Home Assistant | \n","11097 | \n","472 | \n","3023 | \n","Python | \n","
| 1 | \n","comet-ml/opik | \n","Open-source end-to-end LLM Development Platform | \n","2741 | \n","169 | \n","91 | \n","Java | \n","
| 2 | \n","EbookFoundation/free-programming-books | \n","📚 Freely available programming books | \n","341919 | \n","62038 | \n","225 | \n","HTML | \n","
| 3 | \n","konfig-dev/konfig | \n","Sunset as of December 2024 | \n","689 | \n","192 | \n","224 | \n","TypeScript | \n","
| 4 | \n","anoma/anoma | \n","Reference implementation of Anoma | \n","9451 | \n","452 | \n","4129 | \n","Elixir | \n","
| 5 | \n","stripe/stripe-ios | \n","Stripe iOS SDK | \n","2292 | \n","1004 | \n","49 | \n","Swift | \n","
| 6 | \n","Guovin/iptv-api | \n","IPTV live TV source update tool | \n","9385 | \n","2010 | \n","91 | \n","Python | \n","
| 7 | \n","facebookresearch/AnimatedDrawings | \n","Code to accompany \"A Method for Animating Chil... | \n","11473 | \n","988 | \n","398 | \n","Python | \n","
| 8 | \n","apache/airflow | \n","Apache Airflow - A platform to programmaticall... | \n","37690 | \n","14411 | \n","25 | \n","Python | \n","
| 9 | \n","seleniumbase/SeleniumBase | \n","Python APIs for web automation, testing, and b... | \n","6646 | \n","1028 | \n","624 | \n","Python | \n","
\n"," \n","
\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jEkuKbcRrPcK"
+ },
+ "source": [
+ "## 🕷️ Extract Github Trending Repositories with llama index and scrapegraphai's APIs\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IhozYNwsgJzt"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IzsyDXEWwPVt"
+ },
+ "source": [
+ "### 🔧 Install `dependencies`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "os_vm0MkIxr9"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!pip install llama-index\n",
+ "!pip install llama-index-tools-scrapegraphai\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "apBsL-L2KzM7"
+ },
+ "source": [
+ "### 🔑 Import `ScrapeGraph` API key"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ol9gQbAFkh9b"
+ },
+ "source": [
+ "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "sffqFG2EJ8bI",
+ "outputId": "07af4bbe-c226-4fb3-8f68-7ccd429f59cc"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "SGAI_API_KEY not found in environment.\n",
+ "SGAI_API_KEY has been set in the environment.\n"
+ ]
+ }
+ ],
+ "source": [
+ "import os\n",
+ "from getpass import getpass\n",
+ "\n",
+ "# Check if the API key is already set in the environment\n",
+ "sgai_api_key = os.getenv(\"SGAI_API_KEY\")\n",
+ "\n",
+ "if sgai_api_key:\n",
+ " print(\"SGAI_API_KEY found in environment.\")\n",
+ "else:\n",
+ " print(\"SGAI_API_KEY not found in environment.\")\n",
+ " # Prompt the user to input the API key securely (hidden input)\n",
+ " sgai_api_key = getpass(\"Please enter your SGAI_API_KEY: \").strip()\n",
+ " if sgai_api_key:\n",
+ " # Set the API key in the environment\n",
+ " os.environ[\"SGAI_API_KEY\"] = sgai_api_key\n",
+ " print(\"SGAI_API_KEY has been set in the environment.\")\n",
+ " else:\n",
+ " print(\"No API key entered. Please set the API key to continue.\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jnqMB2-xVYQ7"
+ },
+ "source": [
+ "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VZvxbjfXvbgd"
+ },
+ "source": [
+ "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n",
+ "\n",
+ "| \n", + " | name | \n", + "description | \n", + "stars | \n", + "forks | \n", + "today_stars | \n", + "language | \n", + "
|---|---|---|---|---|---|---|
| 0 | \n", + "XiaoMi/ha_xiaomi_home | \n", + "Xiaomi Home Integration for Home Assistant | \n", + "11097 | \n", + "472 | \n", + "3023 | \n", + "Python | \n", + "
| 1 | \n", + "comet-ml/opik | \n", + "Open-source end-to-end LLM Development Platform | \n", + "2741 | \n", + "169 | \n", + "91 | \n", + "Java | \n", + "
| 2 | \n", + "EbookFoundation/free-programming-books | \n", + "📚 Freely available programming books | \n", + "341919 | \n", + "62038 | \n", + "225 | \n", + "HTML | \n", + "
| 3 | \n", + "konfig-dev/konfig | \n", + "Sunset as of December 2024 | \n", + "689 | \n", + "192 | \n", + "224 | \n", + "TypeScript | \n", + "
| 4 | \n", + "anoma/anoma | \n", + "Reference implementation of Anoma | \n", + "9451 | \n", + "452 | \n", + "4129 | \n", + "Elixir | \n", + "
| 5 | \n", + "stripe/stripe-ios | \n", + "Stripe iOS SDK | \n", + "2292 | \n", + "1004 | \n", + "49 | \n", + "Swift | \n", + "
| 6 | \n", + "Guovin/iptv-api | \n", + "IPTV live TV source update tool | \n", + "9385 | \n", + "2010 | \n", + "91 | \n", + "Python | \n", + "
| 7 | \n", + "facebookresearch/AnimatedDrawings | \n", + "Code to accompany \"A Method for Animating Chil... | \n", + "11473 | \n", + "988 | \n", + "398 | \n", + "Python | \n", + "
| 8 | \n", + "apache/airflow | \n", + "Apache Airflow - A platform to programmaticall... | \n", + "37690 | \n", + "14411 | \n", + "25 | \n", + "Python | \n", + "
| 9 | \n", + "seleniumbase/SeleniumBase | \n", + "Python APIs for web automation, testing, and b... | \n", + "6646 | \n", + "1028 | \n", + "624 | \n", + "Python | \n", + "
\n",
+ " \n",
+ "
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️ Extract Github Trending Repositories with Official Scrapegraph SDK"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"d7Zro0xiuo-l"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install scrapegraph-py"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734439787062,"user_tz":-60,"elapsed":5826,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"ab74193e-e746-4de6-d65d-33a2a26b5d86"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | name | \n","description | \n","stars | \n","forks | \n","today_stars | \n","language | \n","
|---|---|---|---|---|---|---|
| 0 | \n","Byaidu/PDFMathTranslate | \n","PDF scientific paper translation with preserve... | \n","8902 | \n","633 | \n","816 | \n","Python | \n","
| 1 | \n","bigskysoftware/htmx | \n","htmx - high power tools for HTML | \n","39143 | \n","1324 | \n","186 | \n","JavaScript | \n","
| 2 | \n","commaai/openpilot | \n","openpilot is an operating system for robotics.... | \n","50945 | \n","9206 | \n","132 | \n","Python | \n","
| 3 | \n","google-gemini/cookbook | \n","Examples and guides for using the Gemini API | \n","8108 | \n","1011 | \n","1221 | \n","Jupyter Notebook | \n","
| 4 | \n","stripe/stripe-ios | \n","Stripe iOS SDK | \n","2179 | \n","994 | \n","19 | \n","Swift | \n","
| 5 | \n","RIOT-OS/RIOT | \n","RIOT - The friendly OS for IoT | \n","5234 | \n","2017 | \n","168 | \n","C | \n","
| 6 | \n","zju3dv/EasyVolcap | \n","EasyVolcap: Accelerating Neural Volumetric Vid... | \n","802 | \n","52 | \n","30 | \n","Python | \n","
| 7 | \n","TEN-framework/TEN-Agent | \n","TEN Agent is a conversational AI powered by TE... | \n","3245 | \n","313 | \n","296 | \n","Python | \n","
| 8 | \n","DS4SD/docling | \n","Get your documents ready for gen AI | \n","15201 | \n","774 | \n","281 | \n","Python | \n","
| 9 | \n","Guovin/iptv-api | \n","📺IPTV电视直播源更新工具：✨央视频、📱卫视、☕各省份地方台、🌏港·澳·台、🎥电影、🎮游戏... | \n","9046 | \n","1938 | \n","101 | \n","Python | \n","
| 10 | \n","fatedier/frp | \n","A fast reverse proxy to help you expose a loca... | \n","87828 | \n","13502 | \n","64 | \n","Go | \n","
| 11 | \n","facebookresearch/AnimatedDrawings | \n","Code to accompany \"A Method for Animating Chil... | \n","10766 | \n","955 | \n","38 | \n","Python | \n","
| 12 | \n","gorilla/websocket | \n","Package gorilla/websocket is a fast, well-test... | \n","22633 | \n","3495 | \n","13 | \n","Go | \n","
| 13 | \n","DefiLlama/chainlist | \n","NA | \n","2368 | \n","2476 | \n","5 | \n","JavaScript | \n","
| 14 | \n","open-telemetry/opentelemetry-collector | \n","OpenTelemetry Collector | \n","4570 | \n","1497 | \n","4 | \n","Go | \n","
| 15 | \n","RocketChat/Rocket.Chat | \n","The communications platform that puts data pro... | \n","41169 | \n","10877 | \n","73 | \n","TypeScript | \n","
| 16 | \n","langgenius/dify | \n","Dify is an open-source LLM app development pla... | \n","54976 | \n","8083 | \n","127 | \n","TypeScript | \n","
\n"," \n","
\n",""]},{"cell_type":"markdown","metadata":{"id":"jEkuKbcRrPcK"},"source":["## 🕷️ Extract Houses Listing with langchain-scrapegraph"]},{"cell_type":"markdown","metadata":{"id":"5cVkde_LpVkF"},"source":[""]},{"cell_type":"markdown","metadata":{"id":"IzsyDXEWwPVt"},"source":["### 🔧 Install `dependencies`"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install langchain-scrapegraph"]},{"cell_type":"markdown","metadata":{"id":"apBsL-L2KzM7"},"source":["### 🔑 Import `ScrapeGraph` API key"]},{"cell_type":"markdown","metadata":{"id":"ol9gQbAFkh9b"},"source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","outputId":"c588274d-64ce-4d13-f12d-0458d9c4839d"},"outputs":[{"name":"stdout","output_type":"stream","text":["SGAI_API_KEY found in environment.\n"]}],"source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"]},{"cell_type":"markdown","metadata":{"id":"jnqMB2-xVYQ7"},"source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"]},{"cell_type":"markdown","metadata":{"id":"VZvxbjfXvbgd"},"source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | price | \n","bedrooms | \n","bathrooms | \n","square_feet | \n","address | \n","city | \n","state | \n","zip_code | \n","tags | \n","agent_name | \n","agency | \n","
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n","549000 | \n","1 | \n","1 | \n","0 | \n","380 14th St Unit 405 | \n","San Francisco | \n","CA | \n","94103 | \n","[New construction] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 1 | \n","1799000 | \n","4 | \n","2 | \n","2735 | \n","123 Grattan St | \n","San Francisco | \n","CA | \n","94117 | \n","[Edwardian-style, investment, owner-occupants] | \n","Sean Engmann | \n","eXp Realty of Northern CA Inc. | \n","
| 2 | \n","1995000 | \n","7 | \n","3 | \n","3330 | \n","1590 Washington St | \n","San Francisco | \n","CA | \n","94109 | \n","[Nob Hill, 3-unit building, investment] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 3 | \n","549000 | \n","0 | \n","1 | \n","477 | \n","240 Lombard St Unit 835 | \n","San Francisco | \n","CA | \n","94111 | \n","[Bay view, studio, modern appliances] | \n","Tim Gullicksen | \n","Corcoran Icon Properties | \n","
| 4 | \n","5495000 | \n","10 | \n","7 | \n","6505 | \n","1057 Steiner St | \n","San Francisco | \n","CA | \n","94115 | \n","[Victorian, Bed & Breakfast, Gilded Age] | \n","Bonnie Spindler | \n","Corcoran Icon Properties | \n","
| 5 | \n","925000 | \n","2 | \n","1 | \n","779 | \n","2 Fallon Place Unit 57 | \n","San Francisco | \n","CA | \n","94133 | \n","[Russian Hill, views, exclusive-use deck] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 6 | \n","898000 | \n","2 | \n","2 | \n","1175 | \n","5160 Diamond Heights Blvd Unit 208C | \n","San Francisco | \n","CA | \n","94131 | \n","[] | \n","Joe Polyak | \n","Rise Homes | \n","
| 7 | \n","1700000 | \n","4 | \n","2 | \n","1950 | \n","1351 26th Ave | \n","San Francisco | \n","CA | \n","94122 | \n","[] | \n","Glenda Queensbury | \n","Referral Realty-BV | \n","
| 8 | \n","1899000 | \n","3 | \n","2 | \n","1560 | \n","340 Yerba Buena Ave | \n","San Francisco | \n","CA | \n","94127 | \n","[] | \n","Jeannie Anderson | \n","Coldwell Banker Realty | \n","
| 9 | \n","850000 | \n","2 | \n","2 | \n","1055 | \n","588 Minna Unit 604 | \n","San Francisco | \n","CA | \n","94103 | \n","[] | \n","Mohamed Lakdawala | \n","Remax Prestigious Properties | \n","
| 10 | \n","1990000 | \n","3 | \n","1 | \n","1280 | \n","1450 Diamond St | \n","San Francisco | \n","CA | \n","94131 | \n","[] | \n","Mary Anne Villamil | \n","Kinetic Real Estate | \n","
| 11 | \n","849000 | \n","1 | \n","1 | \n","855 | \n","81 Lansing St Unit 401 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Kristen Haenggi | \n","Compass | \n","
| 12 | \n","1080000 | \n","2 | \n","2 | \n","936 | \n","451 Kansas St Unit 466 | \n","San Francisco | \n","CA | \n","94107 | \n","[] | \n","Maureen DeBoer | \n","LKJ Realty | \n","
| 13 | \n","1499000 | \n","4 | \n","2 | \n","2145 | \n","486 Yale St | \n","San Francisco | \n","CA | \n","94134 | \n","[] | \n","Alicia Atienza | \n","Statewide Realty | \n","
| 14 | \n","1140000 | \n","2 | \n","2 | \n","998 | \n","588 Minna Unit 801 | \n","San Francisco | \n","CA | \n","94103 | \n","[] | \n","Milan Jezdimirovic | \n","Compass | \n","
| 15 | \n","1988000 | \n","2 | \n","1 | \n","3800 | \n","183 19th Ave | \n","San Francisco | \n","CA | \n","94121 | \n","[Amazing Property, Marina Style, Needs TLC] | \n","Leo Cheung | \n","eXp Realty of California, Inc | \n","
| 16 | \n","1218000 | \n","2 | \n","2 | \n","1275 | \n","1998 Pacific Ave Unit 202 | \n","San Francisco | \n","CA | \n","94109 | \n","[Light-filled, Freshly painted, Walker's parad... | \n","Grace Sun | \n","Compass | \n","
| 17 | \n","895000 | \n","1 | \n","1 | \n","837 | \n","425 1st St Unit 2501 | \n","San Francisco | \n","CA | \n","94105 | \n","[Unobstructed bay bridge views, Open layout] | \n","Matt Fuller | \n","Jackson Fuller Real Estate | \n","
| 18 | \n","1499000 | \n","3 | \n","1 | \n","1500 | \n","Unlisted Address | \n","San Francisco | \n","CA | \n","NA | \n","[Contractor's Special, Fixer-upper] | \n","Jaymee Faith Sagisi | \n","IMPACT | \n","
| 19 | \n","900000 | \n","1 | \n","1 | \n","930 | \n","1101 Green St Unit 302 | \n","San Francisco | \n","CA | \n","94109 | \n","[Historic Art Deco, Abundant natural light] | \n","NA | \n","NA | \n","
| 20 | \n","858000 | \n","1 | \n","1 | \n","1104 | \n","260 King St Unit 557 | \n","San Francisco | \n","CA | \n","94107 | \n","[] | \n","Miyuki Takami | \n","eXp Realty of California, Inc | \n","
| 21 | \n","945000 | \n","2 | \n","1 | \n","767 | \n","307 Page St Unit 1 | \n","San Francisco | \n","CA | \n","94102 | \n","[] | \n","NA | \n","NA | \n","
| 22 | \n","1099000 | \n","2 | \n","2 | \n","1330 | \n","1080 Sutter St Unit 202 | \n","San Francisco | \n","CA | \n","94109 | \n","[] | \n","Annette Liberty | \n","Coldwell Banker Realty | \n","
| 23 | \n","950000 | \n","4 | \n","3 | \n","2090 | \n","3328 26th St Unit 3330 | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Isaac Munene | \n","Coldwell Banker Realty | \n","
| 24 | \n","1088000 | \n","2 | \n","2 | \n","1065 | \n","1776 Sacramento St Unit 503 | \n","San Francisco | \n","CA | \n","94109 | \n","[] | \n","Marilyn Becklehimer | \n","Dio Real Estate | \n","
| 25 | \n","1788888 | \n","4 | \n","3 | \n","1856 | \n","2317 15th St | \n","San Francisco | \n","CA | \n","94114 | \n","[] | \n","Joel Gile | \n","Sequoia Real Estate | \n","
| 26 | \n","1650000 | \n","3 | \n","2 | \n","1547 | \n","2475 47th Ave | \n","San Francisco | \n","CA | \n","94116 | \n","[] | \n","Lucy Goldenshteyn | \n","Redfin | \n","
| 27 | \n","998000 | \n","2 | \n","2 | \n","1202 | \n","50 Lansing St Unit 201 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Tracey Broadman | \n","Vanguard Properties | \n","
| 28 | \n","1595000 | \n","3 | \n","5 | \n","1995 | \n","15 Joy St | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Mike Stack | \n","Vanguard Properties | \n","
| 29 | \n","1028000 | \n","2 | \n","2 | \n","1065 | \n","50 Lansing St Unit 403 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Robyn Kaufman | \n","Vivre Real Estate | \n","
| 30 | \n","999000 | \n","1 | \n","1 | \n","1021 | \n","338 Spear St Unit 6J | \n","San Francisco | \n","CA | \n","94105 | \n","[Spacious, Balcony, Bright courtyard views] | \n","Paul Hwang | \n","Skybox Realty | \n","
| 31 | \n","799800 | \n","2 | \n","2 | \n","1109 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction, 1-car garage] | \n","Lennar | \n","Lennar | \n","
| 32 | \n","529880 | \n","1 | \n","1 | \n","740 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction, 1-car garage] | \n","Lennar | \n","Lennar | \n","
| 33 | \n","489000 | \n","1 | \n","1 | \n","741 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction, 1-car garage] | \n","Lennar | \n","Lennar | \n","
| 34 | \n","1359000 | \n","4 | \n","2 | \n","1845 | \n","170 Thrift St | \n","San Francisco | \n","CA | \n","94112 | \n","[Updated, Natural light] | \n","Cristal Wright | \n","Vanguard Properties | \n","
| 35 | \n","1295000 | \n","3 | \n","1 | \n","1214 | \n","1922 43rd Ave | \n","San Francisco | \n","CA | \n","94116 | \n","[] | \n","Mila Romprey | \n","Premier Realty Associates | \n","
| 36 | \n","1098000 | \n","3 | \n","1 | \n","1006 | \n","150 Putnam St | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Genie Mantzoros | \n","Epic Real Estate & Asso. Inc. | \n","
| 37 | \n","1189870 | \n","3 | \n","2 | \n","1436 | \n","327 Ordway St | \n","San Francisco | \n","CA | \n","94134 | \n","[] | \n","Shawn Zahraie | \n","Affinity Enterprises, Inc | \n","
| 38 | \n","899000 | \n","2 | \n","1 | \n","1118 | \n","272 Farallones St | \n","San Francisco | \n","CA | \n","94112 | \n","[] | \n","Janice Lee | \n","Coldwell Banker Realty | \n","
| 39 | \n","30000 | \n","0 | \n","0 | \n","0 | \n","0 Evans Ave | \n","San Francisco | \n","CA | \n","94124 | \n","[Land, 0.12 Acre, $251,467 per Acre] | \n","Heidy Carrera | \n","Berkshire Hathaway HomeService | \n","
\n"," \n","
\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jEkuKbcRrPcK"
+ },
+ "source": [
+ "## 🕷️ Extract Houses Listing on Zillow with llama-index and ScrapegraphAI APIs"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5cVkde_LpVkF"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IzsyDXEWwPVt"
+ },
+ "source": [
+ "### 🔧 Install `dependencies`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "os_vm0MkIxr9"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!pip install llama-index\n",
+ "!pip install llama-index-tools-scrapegraphai"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "apBsL-L2KzM7"
+ },
+ "source": [
+ "### 🔑 Import `ScrapeGraph` API key"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ol9gQbAFkh9b"
+ },
+ "source": [
+ "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "sffqFG2EJ8bI",
+ "outputId": "c588274d-64ce-4d13-f12d-0458d9c4839d"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "SGAI_API_KEY not found in environment.\n",
+ "SGAI_API_KEY has been set in the environment.\n"
+ ]
+ }
+ ],
+ "source": [
+ "import os\n",
+ "from getpass import getpass\n",
+ "\n",
+ "# Check if the API key is already set in the environment\n",
+ "sgai_api_key = os.getenv(\"SGAI_API_KEY\")\n",
+ "\n",
+ "if sgai_api_key:\n",
+ " print(\"SGAI_API_KEY found in environment.\")\n",
+ "else:\n",
+ " print(\"SGAI_API_KEY not found in environment.\")\n",
+ " # Prompt the user to input the API key securely (hidden input)\n",
+ " sgai_api_key = getpass(\"Please enter your SGAI_API_KEY: \").strip()\n",
+ " if sgai_api_key:\n",
+ " # Set the API key in the environment\n",
+ " os.environ[\"SGAI_API_KEY\"] = sgai_api_key\n",
+ " print(\"SGAI_API_KEY has been set in the environment.\")\n",
+ " else:\n",
+ " print(\"No API key entered. Please set the API key to continue.\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jnqMB2-xVYQ7"
+ },
+ "source": [
+ "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VZvxbjfXvbgd"
+ },
+ "source": [
+ "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n",
+ "\n",
+ "| \n", + " | price | \n", + "bedrooms | \n", + "bathrooms | \n", + "square_feet | \n", + "address | \n", + "city | \n", + "state | \n", + "zip_code | \n", + "tags | \n", + "agent_name | \n", + "agency | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "449000 | \n", + "3 | \n", + "3 | \n", + "2166 | \n", + "3229 Jennings St | \n", + "San Francisco | \n", + "CA | \n", + "94124 | \n", + "[] | \n", + "Michelle K. Pender | \n", + "ENGEL & VOELKERS SAN FRANCISCO | \n", + "
| 1 | \n", + "950000 | \n", + "2 | \n", + "2 | \n", + "1686 | \n", + "401 Huron Ave | \n", + "San Francisco | \n", + "CA | \n", + "94112 | \n", + "[Cozy fireplace] | \n", + "Allison Chapleau | \n", + "COMPASS | \n", + "
| 2 | \n", + "207555 | \n", + "1 | \n", + "1 | \n", + "1593 | \n", + "2040 Fell St APT 10 | \n", + "San Francisco | \n", + "CA | \n", + "94117 | \n", + "[] | \n", + "Trista Elizabeth Bernasconi | \n", + "COMPASS | \n", + "
| 3 | \n", + "795000 | \n", + "4 | \n", + "2 | \n", + "2000 | \n", + "515 Athens St | \n", + "San Francisco | \n", + "CA | \n", + "94112 | \n", + "[Level fenced rear yard] | \n", + "Darin J. Holwitz | \n", + "COMPASS | \n", + "
| 4 | \n", + "599000 | \n", + "1 | \n", + "1 | \n", + "0 | \n", + "380 Dolores St #6 | \n", + "San Francisco | \n", + "CA | \n", + "94114 | \n", + "[] | \n", + "Melody A. Hultgren | \n", + "NOVA REAL ESTATE | \n", + "
| 5 | \n", + "875000 | \n", + "2 | \n", + "2 | \n", + "907 | \n", + "426 Fillmore St #A | \n", + "San Francisco | \n", + "CA | \n", + "94117 | \n", + "[Sleek finishes] | \n", + "NA | \n", + "NA | \n", + "
| 6 | \n", + "335512 | \n", + "2 | \n", + "2 | \n", + "886 | \n", + "1688 Pine St #101 | \n", + "San Francisco | \n", + "CA | \n", + "94109 | \n", + "[] | \n", + "Trista Elizabeth Bernasconi | \n", + "COMPASS | \n", + "
| 7 | \n", + "899000 | \n", + "4 | \n", + "2 | \n", + "1680 | \n", + "351 Chenery St | \n", + "San Francisco | \n", + "CA | \n", + "94131 | \n", + "[South-facing panoramic views] | \n", + "Easton S. Thodos | \n", + "THESEUS REAL ESTATE | \n", + "
| 8 | \n", + "155659 | \n", + "0 | \n", + "1 | \n", + "514 | \n", + "52 Kirkwood Ave #203 | \n", + "San Francisco | \n", + "CA | \n", + "94124 | \n", + "[Modern cabinetry] | \n", + "Lynn Anne Bell | \n", + "CHRISTIE'S INT'L R.E. SF | \n", + "
\n",
+ " \n",
+ "
\n",""]},{"cell_type":"markdown","metadata":{"id":"jEkuKbcRrPcK"},"source":["## 🕷️ Extract Houses Listing with Official Scrapegraph SDK"]},{"cell_type":"markdown","metadata":{"id":"8vZBkAWLq9C1"},"source":[""]},{"cell_type":"markdown","metadata":{"id":"IzsyDXEWwPVt"},"source":["### 🔧 Install `dependencies`"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install scrapegraph-py"]},{"cell_type":"markdown","metadata":{"id":"apBsL-L2KzM7"},"source":["### 🔑 Import `ScrapeGraph` API key"]},{"cell_type":"markdown","metadata":{"id":"ol9gQbAFkh9b"},"source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","outputId":"18dfce64-db37-4825-d316-fabd064100d0"},"outputs":[{"name":"stdout","output_type":"stream","text":["SGAI_API_KEY found in environment.\n"]}],"source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"]},{"cell_type":"markdown","metadata":{"id":"jnqMB2-xVYQ7"},"source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"]},{"cell_type":"markdown","metadata":{"id":"VZvxbjfXvbgd"},"source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | price | \n","bedrooms | \n","bathrooms | \n","square_feet | \n","address | \n","city | \n","state | \n","zip_code | \n","tags | \n","agent_name | \n","agency | \n","
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n","549000 | \n","1 | \n","1 | \n","477 | \n","380 14th St Unit 405 | \n","San Francisco | \n","CA | \n","94103 | \n","[New construction] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 1 | \n","1799000 | \n","4 | \n","2 | \n","2735 | \n","123 Grattan St | \n","San Francisco | \n","CA | \n","94117 | \n","[] | \n","Sean Engmann | \n","eXp Realty of Northern CA Inc. | \n","
| 2 | \n","1995000 | \n","7 | \n","3 | \n","3330 | \n","1590 Washington St | \n","San Francisco | \n","CA | \n","94109 | \n","[] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 3 | \n","549000 | \n","0 | \n","1 | \n","477 | \n","240 Lombard St Unit 835 | \n","San Francisco | \n","CA | \n","94111 | \n","[] | \n","Tim Gullicksen | \n","Corcoran Icon Properties | \n","
| 4 | \n","5495000 | \n","10 | \n","7 | \n","6505 | \n","1057 Steiner St | \n","San Francisco | \n","CA | \n","94115 | \n","[] | \n","Bonnie Spindler | \n","Corcoran Icon Properties | \n","
| 5 | \n","925000 | \n","2 | \n","1 | \n","779 | \n","2 Fallon Place Unit 57 | \n","San Francisco | \n","CA | \n","94133 | \n","[] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 6 | \n","898000 | \n","2 | \n","2 | \n","1175 | \n","5160 Diamond Heights Blvd Unit 208C | \n","San Francisco | \n","CA | \n","94131 | \n","[] | \n","Joe Polyak | \n","Rise Homes | \n","
| 7 | \n","1700000 | \n","4 | \n","2 | \n","1950 | \n","1351 26th Ave | \n","San Francisco | \n","CA | \n","94122 | \n","[] | \n","Glenda Queensbury | \n","Referral Realty-BV | \n","
| 8 | \n","1899000 | \n","3 | \n","2 | \n","1560 | \n","340 Yerba Buena Ave | \n","San Francisco | \n","CA | \n","94127 | \n","[] | \n","Jeannie Anderson | \n","Coldwell Banker Realty | \n","
| 9 | \n","850000 | \n","2 | \n","2 | \n","1055 | \n","588 Minna Unit 604 | \n","San Francisco | \n","CA | \n","94103 | \n","[] | \n","Mohamed Lakdawala | \n","Remax Prestigious Properties | \n","
| 10 | \n","1990000 | \n","3 | \n","1 | \n","1280 | \n","1450 Diamond St | \n","San Francisco | \n","CA | \n","94131 | \n","[] | \n","Mary Anne Villamil | \n","Kinetic Real Estate | \n","
| 11 | \n","849000 | \n","1 | \n","1 | \n","855 | \n","81 Lansing St Unit 401 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Kristen Haenggi | \n","Compass | \n","
| 12 | \n","1080000 | \n","2 | \n","2 | \n","936 | \n","451 Kansas St Unit 466 | \n","San Francisco | \n","CA | \n","94107 | \n","[] | \n","Maureen DeBoer | \n","LKJ Realty | \n","
| 13 | \n","1499000 | \n","4 | \n","2 | \n","2145 | \n","486 Yale St | \n","San Francisco | \n","CA | \n","94134 | \n","[] | \n","Alicia Atienza | \n","Statewide Realty | \n","
| 14 | \n","1140000 | \n","2 | \n","2 | \n","998 | \n","588 Minna Unit 801 | \n","San Francisco | \n","CA | \n","94103 | \n","[] | \n","Milan Jezdimirovic | \n","Compass | \n","
| 15 | \n","1988000 | \n","2 | \n","1 | \n","3800 | \n","183 19th Ave | \n","San Francisco | \n","CA | \n","94121 | \n","[Amazing Property, Marina Style, Needs TLC] | \n","Leo Cheung | \n","eXp Realty of California, Inc | \n","
| 16 | \n","1218000 | \n","2 | \n","2 | \n","1275 | \n","1998 Pacific Ave Unit 202 | \n","San Francisco | \n","CA | \n","94109 | \n","[Light-filled, Freshly painted, Walker's parad... | \n","Grace Sun | \n","Compass | \n","
| 17 | \n","895000 | \n","1 | \n","1 | \n","837 | \n","425 1st St Unit 2501 | \n","San Francisco | \n","CA | \n","94105 | \n","[Unobstructed bay bridge views, Open layout] | \n","Matt Fuller | \n","Jackson Fuller Real Estate | \n","
| 18 | \n","1499000 | \n","3 | \n","1 | \n","1500 | \n","Unlisted Address | \n","San Francisco | \n","CA | \n","NA | \n","[Contractor's Special, Fixer-upper] | \n","Jaymee Faith Sagisi | \n","IMPACT | \n","
| 19 | \n","900000 | \n","1 | \n","1 | \n","930 | \n","1101 Green St Unit 302 | \n","San Francisco | \n","CA | \n","94109 | \n","[Historic Art Deco, Iconic views] | \n","NA | \n","NA | \n","
| 20 | \n","858000 | \n","1 | \n","1 | \n","1104 | \n","260 King St Unit 557 | \n","San Francisco | \n","CA | \n","94107 | \n","[] | \n","Miyuki Takami | \n","eXp Realty of California, Inc | \n","
| 21 | \n","945000 | \n","2 | \n","1 | \n","767 | \n","307 Page St Unit 1 | \n","San Francisco | \n","CA | \n","94102 | \n","[] | \n","NA | \n","NA | \n","
| 22 | \n","1099000 | \n","2 | \n","2 | \n","1330 | \n","1080 Sutter St Unit 202 | \n","San Francisco | \n","CA | \n","94109 | \n","[] | \n","Annette Liberty | \n","Coldwell Banker Realty | \n","
| 23 | \n","950000 | \n","4 | \n","3 | \n","2090 | \n","3328 26th St Unit 3330 | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Isaac Munene | \n","Coldwell Banker Realty | \n","
| 24 | \n","1088000 | \n","2 | \n","2 | \n","1065 | \n","1776 Sacramento St Unit 503 | \n","San Francisco | \n","CA | \n","94109 | \n","[] | \n","Marilyn Becklehimer | \n","Dio Real Estate | \n","
| 25 | \n","1788888 | \n","4 | \n","3 | \n","1856 | \n","2317 15th St | \n","San Francisco | \n","CA | \n","94114 | \n","[] | \n","Joel Gile | \n","Sequoia Real Estate | \n","
| 26 | \n","1650000 | \n","3 | \n","2 | \n","1547 | \n","2475 47th Ave | \n","San Francisco | \n","CA | \n","94116 | \n","[] | \n","Lucy Goldenshteyn | \n","Redfin | \n","
| 27 | \n","998000 | \n","2 | \n","2 | \n","1202 | \n","50 Lansing St Unit 201 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Tracey Broadman | \n","Vanguard Properties | \n","
| 28 | \n","1595000 | \n","3 | \n","5 | \n","1995 | \n","15 Joy St | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Mike Stack | \n","Vanguard Properties | \n","
| 29 | \n","1028000 | \n","2 | \n","2 | \n","1065 | \n","50 Lansing St Unit 403 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Robyn Kaufman | \n","Vivre Real Estate | \n","
| 30 | \n","999000 | \n","1 | \n","1 | \n","1021 | \n","338 Spear St Unit 6J | \n","San Francisco | \n","CA | \n","94105 | \n","[Spacious, Balcony, Bright courtyard views] | \n","Paul Hwang | \n","Skybox Realty | \n","
| 31 | \n","799800 | \n","2 | \n","2 | \n","1109 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction] | \n","Lennar | \n","Lennar | \n","
| 32 | \n","529880 | \n","1 | \n","1 | \n","740 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction] | \n","Lennar | \n","Lennar | \n","
| 33 | \n","489000 | \n","1 | \n","1 | \n","741 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction] | \n","Lennar | \n","Lennar | \n","
| 34 | \n","1359000 | \n","4 | \n","2 | \n","1845 | \n","170 Thrift St | \n","San Francisco | \n","CA | \n","94112 | \n","[Updated, Single-family home] | \n","Cristal Wright | \n","Vanguard Properties | \n","
| 35 | \n","1295000 | \n","3 | \n","1 | \n","1214 | \n","1922 43rd Ave | \n","San Francisco | \n","CA | \n","94116 | \n","[] | \n","Mila Romprey | \n","Premier Realty Associates | \n","
| 36 | \n","1098000 | \n","3 | \n","1 | \n","1006 | \n","150 Putnam St | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Genie Mantzoros | \n","Epic Real Estate & Asso. Inc. | \n","
| 37 | \n","1189870 | \n","3 | \n","2 | \n","1436 | \n","327 Ordway St | \n","San Francisco | \n","CA | \n","94134 | \n","[] | \n","Shawn Zahraie | \n","Affinity Enterprises, Inc | \n","
| 38 | \n","899000 | \n","2 | \n","1 | \n","1118 | \n","272 Farallones St | \n","San Francisco | \n","CA | \n","94112 | \n","[] | \n","Janice Lee | \n","Coldwell Banker Realty | \n","
| 39 | \n","30000 | \n","0 | \n","0 | \n","0 | \n","0 Evans Ave | \n","San Francisco | \n","CA | \n","94124 | \n","[Land, 0.12 Acre, $251,467 per Acre] | \n","Heidy Carrera | \n","Berkshire Hathaway HomeService | \n","
\n"," \n","
\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jEkuKbcRrPcK"
+ },
+ "source": [
+ "## 🕷️ Research Agent with `scrapegraph`, `langgraph`, and `tavily`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "cJrlyZbLwQek"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IzsyDXEWwPVt"
+ },
+ "source": [
+ "### 🔧 Install `dependencies`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "id": "os_vm0MkIxr9"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!pip install langgraph langchain-scrapegraph langchain-openai \"langchain-community>=0.2.11\" tavily-python"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "apBsL-L2KzM7"
+ },
+ "source": [
+ "### 🔑 Import `ScrapeGraph`, `Tavily` and `OpenAI` API keys"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ol9gQbAFkh9b"
+ },
+ "source": [
+ "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "id": "sffqFG2EJ8bI",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "7e47aa4d-edca-48f2-df6e-bd7fafabfc9c"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Scrapegraph API key:\n",
+ "··········\n",
+ "Tavily API key:\n",
+ "··········\n",
+ "OpenAI API key:\n",
+ "··········\n"
+ ]
+ }
+ ],
+ "source": [
+ "import getpass\n",
+ "import os\n",
+ "\n",
+ "if not os.environ.get(\"SGAI_API_KEY\"):\n",
+ " os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")\n",
+ "\n",
+ "if not os.environ.get(\"TAVILY_API_KEY\"):\n",
+ " os.environ[\"TAVILY_API_KEY\"] = getpass.getpass(\"Tavily API key:\\n\")\n",
+ "\n",
+ "if not os.environ.get(\"OPENAI_API_KEY\"):\n",
+ " os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API key:\\n\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jnqMB2-xVYQ7"
+ },
+ "source": [
+ "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VZvxbjfXvbgd"
+ },
+ "source": [
+ "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n",
+ "\n",
+ "| \n", + " | title | \n", + "link | \n", + "description | \n", + "
|---|---|---|---|
| 0 | \n", + "Matternet adds ANRA's UTM tech to expand drone... | \n", + "https://www.therobotreport.com/matternet-adds-... | \n", + "This latest partnership follows Matternet’s re... | \n", + "
| 1 | \n", + "Helm.ai upgrades generative AI model to enrich... | \n", + "https://www.therobotreport.com/helm-ai-upgrade... | \n", + "Helm.ai said the new model enables automakers ... | \n", + "
| 2 | \n", + "New research analyzes safety of Waymo robotaxis | \n", + "https://www.therobotreport.com/new-research-an... | \n", + "Waymo shared research with Swiss Re, one of th... | \n", + "
| 3 | \n", + "From AI to humanoids: top robotics trends of 2024 | \n", + "https://www.therobotreport.com/from-ai-to-huma... | \n", + "The Robot Report Podcast reflects on the succe... | \n", + "
| 4 | \n", + "Symbotic acquires OhmniLabs, maker of disinfec... | \n", + "https://www.therobotreport.com/symbotic-buys-h... | \n", + "With the acquisition of OhmniLabs, Symbotic sa... | \n", + "
| 5 | \n", + "Sanctuary AI shows new dexterity with in-hand ... | \n", + "https://www.therobotreport.com/sanctuary-ai-sh... | \n", + "Sanctuary AI showed its latest breakthrough wi... | \n", + "
| 6 | \n", + "Apptronik partners with Google DeepMind to adv... | \n", + "https://www.therobotreport.com/apptronik-partn... | \n", + "Apptronik will combine its iterative design ex... | \n", + "
| 7 | \n", + "Alimak Group, Skyline Robotics create autonomo... | \n", + "https://www.therobotreport.com/alimak-group-sk... | \n", + "Skyline Robotics said the joint system can hel... | \n", + "
| 8 | \n", + "DoorDash partners with Wing to launch drone de... | \n", + "https://www.therobotreport.com/doordash-partne... | \n", + "Beginning today, when certain DoorDash custome... | \n", + "
| 9 | \n", + "Mcity says open-source digital twin enables ch... | \n", + "https://www.therobotreport.com/mcity-open-sour... | \n", + "The Mcity test facility has been open since 20... | \n", + "
| 10 | \n", + "2024: The year humanoids woke up | \n", + "https://www.therobotreport.com/2024-the-year-h... | \n", + "Humanoids empowered by AI are coming, and the ... | \n", + "
| 11 | \n", + "Waymo robotaxis head to Tokyo with the help of... | \n", + "https://www.therobotreport.com/waymo-is-headin... | \n", + "The first all-electric Jaguar I-PACEs for Waym... | \n", + "
| 12 | \n", + "Realbotix earns Amazon development subsidy; pa... | \n", + "https://www.therobotreport.com/realbotix-earns... | \n", + "Realbotix plans to use the funding to directly... | \n", + "
| 13 | \n", + "Eyeonic Trace Laser Line Scanner offers sub-mi... | \n", + "https://www.therobotreport.com/eyeonic-trace-l... | \n", + "Prototype of the Eyeonic Trace Laser Line Scan... | \n", + "
| 14 | \n", + "Slip Robotics picks up $28M for trailer loadin... | \n", + "https://www.therobotreport.com/slip-robotics-p... | \n", + "Slip Robotics plans to use its latest funding ... | \n", + "
| 15 | \n", + "Jetson Orin Nano Super developer kit available... | \n", + "https://www.therobotreport.com/jetson-orin-nan... | \n", + "NVIDIA released Jetson Orin Nano Super Develop... | \n", + "
| 16 | \n", + "Mbodi and T-Robotics are ABB Robotics' AI Star... | \n", + "https://www.therobotreport.com/mbodi-and-t-rob... | \n", + "ABB Robotics is working with Mbodi and T-Robot... | \n", + "
| 17 | \n", + "IEEE Awards announce Daniela Rus as 2025 Ediso... | \n", + "https://www.therobotreport.com/ieee-awards-ann... | \n", + "Currently the director of MIT CSAIL, Daniela R... | \n", + "
| 18 | \n", + "Eureka Robotics raises $10.5M to scale its vis... | \n", + "https://www.therobotreport.com/eureka-robotics... | \n", + "Eureka Robotics provides software and system t... | \n", + "
| 19 | \n", + "Vision-guided cobot automates paint process fo... | \n", + "https://www.therobotreport.com/denso-automates... | \n", + "DENSO deployed a 3D-vision-guided cobot with A... | \n", + "
| 20 | \n", + "Brushed DC motors find use in robot applicatio... | \n", + "https://www.therobotreport.com/brushed-dc-moto... | \n", + "Recent research from Portescap found that brus... | \n", + "
| 21 | \n", + "Diversity and inclusion can accelerate robotic... | \n", + "https://www.therobotreport.com/diversity-and-i... | \n", + "The study outlined seven distinct benefits tha... | \n", + "
| 22 | \n", + "Advanced Precision Strain Wave Gear Offers Tor... | \n", + "https://www.therobotreport.com/advanced-precis... | \n", + "NA | \n", + "
| 23 | \n", + "Innovative motion solutions are supporting the... | \n", + "https://www.therobotreport.com/innovative-moti... | \n", + "NA | \n", + "
| 24 | \n", + "Renishaw and RLS help to drive a robot revolution | \n", + "https://www.therobotreport.com/renishaw-and-rl... | \n", + "NA | \n", + "
| 25 | \n", + "Ask an Expert Podcast: flexible conveyance for... | \n", + "https://www.therobotreport.com/ask-an-expert-f... | \n", + "NA | \n", + "
| 26 | \n", + "Hop Onboard the AMR Revolution: Vision & Local... | \n", + "https://www.therobotreport.com/hop-onboard-the... | \n", + "NA | \n", + "
\n",
+ " \n",
+ "
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️ Extract Wired Science News with langchain-scrapegraph"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"oQa0HZo7nuB_"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install langchain-scrapegraph"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734622417698,"user_tz":-60,"elapsed":2683,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"721836bb-dcd3-4788-d5eb-ff91948d6674"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | category | \n","title | \n","link | \n","author | \n","
|---|---|---|---|---|
| 0 | \n","Science | \n","NASA Postpones Return of Stranded Starliner As... | \n","https://www.wired.com/story/boeing-starliner-a... | \n","Fernanda González | \n","
| 1 | \n","Public Health | \n","CDC Confirms First US Case of Severe Bird Flu | \n","https://www.wired.com/story/cdc-confirms-first... | \n","Emily Mullin | \n","
| 2 | \n","Science | \n","The Study That Called Out Black Plastic Utensi... | \n","https://www.wired.com/story/black-plastic-uten... | \n","Beth Mole, Ars Technica | \n","
| 3 | \n","Health | \n","A Third Person Has Received a Transplant of a ... | \n","https://www.wired.com/story/a-third-person-has... | \n","Emily Mullin | \n","
| 4 | \n","Health | \n","Antibodies Could Soon Help Slow the Aging Process | \n","https://www.wired.com/story/antibodies-could-s... | \n","Andrew Steele | \n","
| 5 | \n","Science | \n","Good at Reading? Your Brain May Be Structured ... | \n","https://www.wired.com/story/good-at-reading-yo... | \n","Mikael Roll | \n","
| 6 | \n","Environment | \n","Mega-Farms Are Driving the Threat of Bird Flu | \n","https://www.wired.com/story/mega-farms-are-dri... | \n","Georgina Gustin | \n","
| 7 | \n","Environment | \n","How Christmas Trees Could Become a Source of L... | \n","https://www.wired.com/story/how-christmas-tree... | \n","Alexa Phillips | \n","
| 8 | \n","Environment | \n","Creating a Global Package to Solve the Problem... | \n","https://www.wired.com/story/global-plastics-tr... | \n","Susan Solomon | \n","
| 9 | \n","Environment | \n","These 3 Things Are Standing in the Way of a Gl... | \n","https://www.wired.com/story/these-3-things-are... | \n","Steve Fletcher and Samuel Winton | \n","
| 10 | \n","Environment | \n","Environmental Sensing Is Here, Tracking Everyt... | \n","https://www.wired.com/story/environmental-sens... | \n","Sabrina Weiss | \n","
| 11 | \n","Climate | \n","Generative AI and Climate Change Are on a Coll... | \n","https://www.wired.com/story/true-cost-generati... | \n","Sasha Luccioni | \n","
| 12 | \n","Climate | \n","Climate Change Is Destroying Monarch Butterfli... | \n","https://www.wired.com/story/global-warming-thr... | \n","Andrea J. Arratibel | \n","
| 13 | \n","Politics | \n","More Humanitarian Organizations Will Harness A... | \n","https://www.wired.com/story/humanitarian-organ... | \n","David Miliband | \n","
| 14 | \n","Environment | \n","Chocolate Has a Sustainability Problem. Scienc... | \n","https://www.wired.com/story/chocolate-has-a-su... | \n","Eve Thomas | \n","
| 15 | \n","Energy | \n","Big Tech Will Scour the Globe in Its Search fo... | \n","https://www.wired.com/story/big-tech-data-cent... | \n","Azeem Azhar | \n","
| 16 | \n","Environment | \n","Humans Will Continue to Live in an Age of Incr... | \n","https://www.wired.com/story/food-production-en... | \n","Vaclav Smil | \n","
| 17 | \n","Energy | \n","A Uranium-Mining Boom Is Sweeping Through Texas | \n","https://www.wired.com/story/a-uranium-mining-b... | \n","Dylan Baddour | \n","
| 18 | \n","Space | \n","The End Is Near for NASA’s Voyager Probes | \n","https://www.wired.com/story/the-end-is-near-fo... | \n","Luca Nardi | \n","
| 19 | \n","Space | \n","The Mystery of How Supermassive Black Holes Merge | \n","https://www.wired.com/story/how-do-merging-sup... | \n","Jonathan O’Callaghan | \n","
| 20 | \n","Space | \n","Starship’s Next Launch Could Be Just Two Weeks... | \n","https://www.wired.com/story/starships-next-lau... | \n","Eric Berger, Ars Technica | \n","
| 21 | \n","Math | \n","The Simple Math Behind Public Key Cryptography | \n","https://www.wired.com/story/how-public-key-cry... | \n","John Pavlus | \n","
| 22 | \n","Math | \n","Everyone Is Capable of Mathematical Thinking--... | \n","https://www.wired.com/story/everyone-is-capabl... | \n","Kelsey Houston-Edwards | \n","
| 23 | \n","Math | \n","The Physics of the Macy’s Thanksgiving Day Par... | \n","https://www.wired.com/story/the-physics-of-the... | \n","Rhett Allain | \n","
| 24 | \n","Math | \n","Mathematicians Just Debunked the ‘Bunkbed Conj... | \n","https://www.wired.com/story/maths-bunkbed-conj... | \n","Joseph Howlett | \n","
| 25 | \n","Biotech | \n","Muscle Implants Could Allow Mind-Controlled Pr... | \n","https://www.wired.com/story/amputees-could-con... | \n","Emily Mullin | \n","
| 26 | \n","Biotech | \n","Combining AI and Crispr Will Be Transformational | \n","https://www.wired.com/story/combining-ai-and-c... | \n","Jennifer Doudna | \n","
| 27 | \n","Biotech | \n","Neuralink Plans to Test Whether Its Brain Impl... | \n","https://www.wired.com/story/neuralink-robotic-... | \n","Emily Mullin | \n","
| 28 | \n","Public Health | \n","A Mysterious Respiratory Disease Has the Democ... | \n","https://www.wired.com/story/drc-mysterious-res... | \n","Marta Musso | \n","
| 29 | \n","Health | \n","Skip the Sea Kelp Supplements | \n","https://www.wired.com/story/pass-on-sea-kelp-s... | \n","Boutayna Chokrane | \n","
| 30 | \n","Sports | \n","Why Soccer Players Are Training in the Dark | \n","https://www.wired.com/story/why-soccer-players... | \n","RM Clark | \n","
| 31 | \n","Environment | \n","A Parasite That Eats Cattle Alive Is Creeping ... | \n","https://www.wired.com/story/a-parasite-that-ea... | \n","Geraldine Castro | \n","
| 32 | \n","Technology | \n","Lasers Are Making It Easier to Find Buried Lan... | \n","https://www.wired.com/story/this-laser-system-... | \n","Ritsuko Kawai | \n","
| 33 | \n","Health | \n","Mark Cuban’s War on Drug Prices: ‘How Much Fuc... | \n","https://www.wired.com/story/big-interview-mark... | \n","Marah Eakin | \n","
| 34 | \n","Environment | \n","Can Artificial Rain, Drones, or Satellites Cle... | \n","https://www.wired.com/story/artificial-rain-dr... | \n","Arunima Kar | \n","
| 35 | \n","Health | \n","These Stem Cell Treatments Are Worth Millions.... | \n","https://www.wired.com/story/stem-cells-cost-ri... | \n","Matt Reynolds | \n","
| 36 | \n","Environment | \n","The $60 Billion Potential Hiding in Your Disca... | \n","https://www.wired.com/story/a-dollar60-billion... | \n","Vince Beiser | \n","
| 37 | \n","Health | \n","Tune In to the Healing Powers of a Decent Play... | \n","https://www.wired.com/story/music-therapy-heal... | \n","Daniel Levitin | \n","
| 38 | \n","Human History | \n","The Whole Story of How Humans Evolved From Gre... | \n","https://www.wired.com/story/the-whole-story-of... | \n","John Gowlett | \n","
| 39 | \n","Health | \n","Why an Offline Nuclear Reactor Led to Thousand... | \n","https://www.wired.com/story/why-an-offline-nuc... | \n","Chris Baraniuk | \n","
\n"," \n","
\n",""]},{"cell_type":"markdown","metadata":{"id":"jEkuKbcRrPcK"},"source":["## 🕷️ Extract Wired Science News with `langchain-scrapegraph` and `langgraph`"]},{"cell_type":"markdown","source":[""],"metadata":{"id":"FxtXj1Qtx3zH"}},{"cell_type":"markdown","metadata":{"id":"IzsyDXEWwPVt"},"source":["### 🔧 Install `dependencies`"]},{"cell_type":"code","execution_count":1,"metadata":{"id":"os_vm0MkIxr9","executionInfo":{"status":"ok","timestamp":1734786986883,"user_tz":-60,"elapsed":13752,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}}},"outputs":[],"source":["%%capture\n","!pip install langgraph langchain-scrapegraph langchain-openai"]},{"cell_type":"markdown","metadata":{"id":"apBsL-L2KzM7"},"source":["### 🔑 Import `ScrapeGraph` and `OpenAI` API keys"]},{"cell_type":"markdown","metadata":{"id":"ol9gQbAFkh9b"},"source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"]},{"cell_type":"code","execution_count":2,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":67210,"status":"ok","timestamp":1734787067908,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"},"user_tz":-60},"id":"sffqFG2EJ8bI","outputId":"8c3dd34f-22d5-4557-c562-97728738726b"},"outputs":[{"name":"stdout","output_type":"stream","text":["Scrapegraph API key:\n","··········\n","OpenAI API key:\n","··········\n"]}],"source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")\n","\n","if not os.environ.get(\"OPENAI_API_KEY\"):\n"," os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API key:\\n\")"]},{"cell_type":"markdown","metadata":{"id":"jnqMB2-xVYQ7"},"source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"]},{"cell_type":"markdown","metadata":{"id":"VZvxbjfXvbgd"},"source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | title | \n","link | \n","author | \n","
|---|---|---|---|
| 0 | \n","December Wildfires Are Now a Thing | \n","https://www.wired.com/story/december-wildfires... | \n","Kylie Mohr | \n","
| 1 | \n","How to Manage Food Anxiety Over the Holidays | \n","https://www.wired.com/story/how-to-cope-with-f... | \n","Alison Fixsen | \n","
| 2 | \n","A Spacecraft Is About to Fly Into the Sun’s At... | \n","https://www.wired.com/story/parker-solar-probe... | \n","Eric Berger, Ars Technica | \n","
| 3 | \n","To Improve Your Gut Microbiome, Spend More Tim... | \n","https://www.wired.com/story/to-improve-your-gu... | \n","Kathy Willis | \n","
| 4 | \n","This Tropical Virus Is Spreading Out of the Am... | \n","https://www.wired.com/story/this-tropical-viru... | \n","Geraldine Castro | \n","
| 5 | \n","CDC Confirms First US Case of Severe Bird Flu | \n","https://www.wired.com/story/cdc-confirms-first... | \n","Emily Mullin | \n","
| 6 | \n","The Study That Called Out Black Plastic Utensi... | \n","https://www.wired.com/story/black-plastic-uten... | \n","Beth Mole, Ars Technica | \n","
| 7 | \n","How Christmas Trees Could Become a Source of L... | \n","https://www.wired.com/story/how-christmas-tree... | \n","Alexa Phillips | \n","
| 8 | \n","Creating a Global Package to Solve the Problem... | \n","https://www.wired.com/story/global-plastics-tr... | \n","Susan Solomon | \n","
| 9 | \n","These 3 Things Are Standing in the Way of a Gl... | \n","https://www.wired.com/story/these-3-things-are... | \n","Steve Fletcher and Samuel Winton | \n","
| 10 | \n","Environmental Sensing Is Here, Tracking Everyt... | \n","https://www.wired.com/story/environmental-sens... | \n","Sabrina Weiss | \n","
| 11 | \n","Generative AI and Climate Change Are on a Coll... | \n","https://www.wired.com/story/true-cost-generati... | \n","Sasha Luccioni | \n","
| 12 | \n","Climate Change Is Destroying Monarch Butterfli... | \n","https://www.wired.com/story/global-warming-thr... | \n","Andrea J. Arratibel | \n","
| 13 | \n","More Humanitarian Organizations Will Harness A... | \n","https://www.wired.com/story/humanitarian-organ... | \n","David Miliband | \n","
| 14 | \n","Chocolate Has a Sustainability Problem. Scienc... | \n","https://www.wired.com/story/chocolate-has-a-su... | \n","Eve Thomas | \n","
| 15 | \n","We’ve Never Been Closer to Finding Life Outsid... | \n","https://www.wired.com/story/james-webb-space-t... | \n","Lisa Kaltenegger | \n","
| 16 | \n","The End Is Near for NASA’s Voyager Probes | \n","https://www.wired.com/story/the-end-is-near-fo... | \n","Luca Nardi | \n","
| 17 | \n","Why Can’t You Switch Seats in an Empty Airplane? | \n","https://www.wired.com/story/why-cant-you-switc... | \n","Rhett Allain | \n","
| 18 | \n","The Simple Math Behind Public Key Cryptography | \n","https://www.wired.com/story/how-public-key-cry... | \n","John Pavlus | \n","
| 19 | \n","Everyone Is Capable of Mathematical Thinking--... | \n","https://www.wired.com/story/everyone-is-capabl... | \n","Kelsey Houston-Edwards | \n","
| 20 | \n","The Physics of the Macy’s Thanksgiving Day Par... | \n","https://www.wired.com/story/the-physics-of-the... | \n","Rhett Allain | \n","
| 21 | \n","A Third Person Has Received a Transplant of a ... | \n","https://www.wired.com/story/a-third-person-has... | \n","Emily Mullin | \n","
| 22 | \n","Muscle Implants Could Allow Mind-Controlled Pr... | \n","https://www.wired.com/story/amputees-could-con... | \n","Emily Mullin | \n","
| 23 | \n","Combining AI and Crispr Will Be Transformational | \n","https://www.wired.com/story/combining-ai-and-c... | \n","Jennifer Doudna | \n","
| 24 | \n","Neuralink Plans to Test Whether Its Brain Impl... | \n","https://www.wired.com/story/neuralink-robotic-... | \n","Emily Mullin | \n","
| 25 | \n","Eight Scientists, a Billion Dollars, and the M... | \n","https://www.wired.com/story/aria-moonshot-darp... | \n","Matt Reynolds | \n","
| 26 | \n","The Atlas Robot Is Dead. Long Live the Atlas R... | \n","https://www.wired.com/story/the-atlas-robot-is... | \n","NA | \n","
| 27 | \n","The Atlas Robot Is Dead. Long Live the Atlas R... | \n","https://www.wired.com/story/the-atlas-robot-is... | \n","Carlton Reid | \n","
| 28 | \n","Meet the Next Generation of Doctors--and Their... | \n","https://www.wired.com/story/next-generation-do... | \n","Neha Mukherjee | \n","
| 29 | \n","AI Is Building Highly Effective Antibodies Tha... | \n","https://www.wired.com/story/labgenius-antibody... | \n","Amit Katwala | \n","
| 30 | \n","An Uncertain Future Requires Uncertain Predict... | \n","https://www.wired.com/story/embrace-uncertaint... | \n","David Spiegelhalter | \n","
| 31 | \n","These Rats Learned to Drive--and They Love It | \n","https://www.wired.com/story/these-rats-learned... | \n","Kelly Lambert | \n","
| 32 | \n","Scientists Are Unlocking the Secrets of Your ‗... | \n","https://www.wired.com/story/cerebellum-brain-m... | \n","R Douglas Fields | \n","
| 33 | \n","Meet the Designer Behind Neuralink’s Surgical ... | \n","https://www.wired.com/story/designer-behind-ne... | \n","Emily Mullin | \n","
| 34 | \n","Antibodies Could Soon Help Slow the Aging Process | \n","https://www.wired.com/story/antibodies-could-s... | \n","Andrew Steele | \n","
| 35 | \n","Good at Reading? Your Brain May Be Structured ... | \n","https://www.wired.com/story/good-at-reading-yo... | \n","Mikael Roll | \n","
| 36 | \n","Mega-Farms Are Driving the Threat of Bird Flu | \n","https://www.wired.com/story/mega-farms-are-dri... | \n","Georgina Gustin | \n","
| 37 | \n","RFK Plans to Take on Big Pharma. It’s Easier S... | \n","https://www.wired.com/story/rfks-plan-to-take-... | \n","Emily Mullin | \n","
| 38 | \n","Designer Babies Are Teenagers Now--and Some of... | \n","https://www.wired.com/story/your-next-job-desi... | \n","Emi Nietfeld | \n","
| 39 | \n","US Meat, Milk Prices Should Spike if Donald Tr... | \n","https://www.wired.com/story/us-meat-milk-price... | \n","Matt Reynolds | \n","
| 40 | \n","An Augmented Reality Program Can Help Patients... | \n","https://www.wired.com/story/lining-up-tech-to-... | \n","Grace Browne | \n","
| 41 | \n","Meet the Plant Hacker Creating Flowers Never S... | \n","https://www.wired.com/story/meet-the-plant-hac... | \n","Matt Reynolds | \n","
| 42 | \n","A Mysterious Respiratory Disease Has the Democ... | \n","https://www.wired.com/story/drc-mysterious-res... | \n","Marta Musso | \n","
| 43 | \n","Skip the Sea Kelp Supplements | \n","https://www.wired.com/story/pass-on-sea-kelp-s... | \n","Boutayna Chokrane | \n","
| 44 | \n","Why Soccer Players Are Training in the Dark | \n","https://www.wired.com/story/why-soccer-players... | \n","RM Clark | \n","
| 45 | \n","A Parasite That Eats Cattle Alive Is Creeping ... | \n","https://www.wired.com/story/a-parasite-that-ea... | \n","Geraldine Castro | \n","
| 46 | \n","Lasers Are Making It Easier to Find Buried Lan... | \n","https://www.wired.com/story/this-laser-system-... | \n","Ritsuko Kawai | \n","
| 47 | \n","Mark Cuban’s War on Drug Prices: ‖How Much Fuc... | \n","https://www.wired.com/story/big-interview-mark... | \n","Marah Eakin | \n","
| 48 | \n","Can Artificial Rain, Drones, or Satellites Cle... | \n","https://www.wired.com/story/artificial-rain-dr... | \n","Arunima Kar | \n","
| 49 | \n","These Stem Cell Treatments Are Worth Millions.... | \n","https://www.wired.com/story/stem-cells-cost-ri... | \n","Matt Reynolds | \n","
| 50 | \n","The Mystery of How Supermassive Black Holes Merge | \n","https://www.wired.com/story/how-do-merging-sup... | \n","Jonathan O’Callaghan | \n","
| 51 | \n","The $60 Billion Potential Hiding in Your Disca... | \n","https://www.wired.com/story/a-dollar60-billion... | \n","Vince Beiser | \n","
| 52 | \n","Tune In to the Healing Powers of a Decent Play... | \n","https://www.wired.com/story/music-therapy-heal... | \n","Daniel Levitin | \n","
| 53 | \n","Returning the Amazon Rainforest to Its True Ca... | \n","https://www.wired.com/story/amazon-rainforest-... | \n","Nemonte Nenquimo and Mitch Anderson | \n","
\n"," \n","
\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jEkuKbcRrPcK"
+ },
+ "source": [
+ "## 🕷️ Extract Wired Science News Info with llama-index and ScrapegraphAI's APIs"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "oQa0HZo7nuB_"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IzsyDXEWwPVt"
+ },
+ "source": [
+ "### 🔧 Install `dependencies`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "os_vm0MkIxr9"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!pip install llama-index-tools-scrapegraphai\n",
+ "!pip install llama-index"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "apBsL-L2KzM7"
+ },
+ "source": [
+ "### 🔑 Import `ScrapeGraph` API key"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ol9gQbAFkh9b"
+ },
+ "source": [
+ "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "sffqFG2EJ8bI"
+ },
+ "outputs": [],
+ "source": [
+ "import getpass\n",
+ "import os\n",
+ "\n",
+ "if not os.environ.get(\"SGAI_API_KEY\"):\n",
+ " os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jnqMB2-xVYQ7"
+ },
+ "source": [
+ "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VZvxbjfXvbgd"
+ },
+ "source": [
+ "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n",
+ "\n",
+ "| \n", + " | category | \n", + "title | \n", + "link | \n", + "author | \n", + "
|---|---|---|---|---|
| 0 | \n", + "Science | \n", + "NASA Postpones Return of Stranded Starliner As... | \n", + "https://www.wired.com/story/boeing-starliner-a... | \n", + "Fernanda González | \n", + "
| 1 | \n", + "Public Health | \n", + "CDC Confirms First US Case of Severe Bird Flu | \n", + "https://www.wired.com/story/cdc-confirms-first... | \n", + "Emily Mullin | \n", + "
| 2 | \n", + "Science | \n", + "The Study That Called Out Black Plastic Utensi... | \n", + "https://www.wired.com/story/black-plastic-uten... | \n", + "Beth Mole, Ars Technica | \n", + "
| 3 | \n", + "Health | \n", + "A Third Person Has Received a Transplant of a ... | \n", + "https://www.wired.com/story/a-third-person-has... | \n", + "Emily Mullin | \n", + "
| 4 | \n", + "Health | \n", + "Antibodies Could Soon Help Slow the Aging Process | \n", + "https://www.wired.com/story/antibodies-could-s... | \n", + "Andrew Steele | \n", + "
| 5 | \n", + "Science | \n", + "Good at Reading? Your Brain May Be Structured ... | \n", + "https://www.wired.com/story/good-at-reading-yo... | \n", + "Mikael Roll | \n", + "
| 6 | \n", + "Environment | \n", + "Mega-Farms Are Driving the Threat of Bird Flu | \n", + "https://www.wired.com/story/mega-farms-are-dri... | \n", + "Georgina Gustin | \n", + "
| 7 | \n", + "Environment | \n", + "How Christmas Trees Could Become a Source of L... | \n", + "https://www.wired.com/story/how-christmas-tree... | \n", + "Alexa Phillips | \n", + "
| 8 | \n", + "Environment | \n", + "Creating a Global Package to Solve the Problem... | \n", + "https://www.wired.com/story/global-plastics-tr... | \n", + "Susan Solomon | \n", + "
| 9 | \n", + "Environment | \n", + "These 3 Things Are Standing in the Way of a Gl... | \n", + "https://www.wired.com/story/these-3-things-are... | \n", + "Steve Fletcher and Samuel Winton | \n", + "
| 10 | \n", + "Environment | \n", + "Environmental Sensing Is Here, Tracking Everyt... | \n", + "https://www.wired.com/story/environmental-sens... | \n", + "Sabrina Weiss | \n", + "
| 11 | \n", + "Climate | \n", + "Generative AI and Climate Change Are on a Coll... | \n", + "https://www.wired.com/story/true-cost-generati... | \n", + "Sasha Luccioni | \n", + "
| 12 | \n", + "Climate | \n", + "Climate Change Is Destroying Monarch Butterfli... | \n", + "https://www.wired.com/story/global-warming-thr... | \n", + "Andrea J. Arratibel | \n", + "
| 13 | \n", + "Politics | \n", + "More Humanitarian Organizations Will Harness A... | \n", + "https://www.wired.com/story/humanitarian-organ... | \n", + "David Miliband | \n", + "
| 14 | \n", + "Environment | \n", + "Chocolate Has a Sustainability Problem. Scienc... | \n", + "https://www.wired.com/story/chocolate-has-a-su... | \n", + "Eve Thomas | \n", + "
| 15 | \n", + "Energy | \n", + "Big Tech Will Scour the Globe in Its Search fo... | \n", + "https://www.wired.com/story/big-tech-data-cent... | \n", + "Azeem Azhar | \n", + "
| 16 | \n", + "Environment | \n", + "Humans Will Continue to Live in an Age of Incr... | \n", + "https://www.wired.com/story/food-production-en... | \n", + "Vaclav Smil | \n", + "
| 17 | \n", + "Energy | \n", + "A Uranium-Mining Boom Is Sweeping Through Texas | \n", + "https://www.wired.com/story/a-uranium-mining-b... | \n", + "Dylan Baddour | \n", + "
| 18 | \n", + "Space | \n", + "The End Is Near for NASA’s Voyager Probes | \n", + "https://www.wired.com/story/the-end-is-near-fo... | \n", + "Luca Nardi | \n", + "
| 19 | \n", + "Space | \n", + "The Mystery of How Supermassive Black Holes Merge | \n", + "https://www.wired.com/story/how-do-merging-sup... | \n", + "Jonathan O’Callaghan | \n", + "
| 20 | \n", + "Space | \n", + "Starship’s Next Launch Could Be Just Two Weeks... | \n", + "https://www.wired.com/story/starships-next-lau... | \n", + "Eric Berger, Ars Technica | \n", + "
| 21 | \n", + "Math | \n", + "The Simple Math Behind Public Key Cryptography | \n", + "https://www.wired.com/story/how-public-key-cry... | \n", + "John Pavlus | \n", + "
| 22 | \n", + "Math | \n", + "Everyone Is Capable of Mathematical Thinking--... | \n", + "https://www.wired.com/story/everyone-is-capabl... | \n", + "Kelsey Houston-Edwards | \n", + "
| 23 | \n", + "Math | \n", + "The Physics of the Macy’s Thanksgiving Day Par... | \n", + "https://www.wired.com/story/the-physics-of-the... | \n", + "Rhett Allain | \n", + "
| 24 | \n", + "Math | \n", + "Mathematicians Just Debunked the ‘Bunkbed Conj... | \n", + "https://www.wired.com/story/maths-bunkbed-conj... | \n", + "Joseph Howlett | \n", + "
| 25 | \n", + "Biotech | \n", + "Muscle Implants Could Allow Mind-Controlled Pr... | \n", + "https://www.wired.com/story/amputees-could-con... | \n", + "Emily Mullin | \n", + "
| 26 | \n", + "Biotech | \n", + "Combining AI and Crispr Will Be Transformational | \n", + "https://www.wired.com/story/combining-ai-and-c... | \n", + "Jennifer Doudna | \n", + "
| 27 | \n", + "Biotech | \n", + "Neuralink Plans to Test Whether Its Brain Impl... | \n", + "https://www.wired.com/story/neuralink-robotic-... | \n", + "Emily Mullin | \n", + "
| 28 | \n", + "Public Health | \n", + "A Mysterious Respiratory Disease Has the Democ... | \n", + "https://www.wired.com/story/drc-mysterious-res... | \n", + "Marta Musso | \n", + "
| 29 | \n", + "Health | \n", + "Skip the Sea Kelp Supplements | \n", + "https://www.wired.com/story/pass-on-sea-kelp-s... | \n", + "Boutayna Chokrane | \n", + "
| 30 | \n", + "Sports | \n", + "Why Soccer Players Are Training in the Dark | \n", + "https://www.wired.com/story/why-soccer-players... | \n", + "RM Clark | \n", + "
| 31 | \n", + "Environment | \n", + "A Parasite That Eats Cattle Alive Is Creeping ... | \n", + "https://www.wired.com/story/a-parasite-that-ea... | \n", + "Geraldine Castro | \n", + "
| 32 | \n", + "Technology | \n", + "Lasers Are Making It Easier to Find Buried Lan... | \n", + "https://www.wired.com/story/this-laser-system-... | \n", + "Ritsuko Kawai | \n", + "
| 33 | \n", + "Health | \n", + "Mark Cuban’s War on Drug Prices: ‘How Much Fuc... | \n", + "https://www.wired.com/story/big-interview-mark... | \n", + "Marah Eakin | \n", + "
| 34 | \n", + "Environment | \n", + "Can Artificial Rain, Drones, or Satellites Cle... | \n", + "https://www.wired.com/story/artificial-rain-dr... | \n", + "Arunima Kar | \n", + "
| 35 | \n", + "Health | \n", + "These Stem Cell Treatments Are Worth Millions.... | \n", + "https://www.wired.com/story/stem-cells-cost-ri... | \n", + "Matt Reynolds | \n", + "
| 36 | \n", + "Environment | \n", + "The $60 Billion Potential Hiding in Your Disca... | \n", + "https://www.wired.com/story/a-dollar60-billion... | \n", + "Vince Beiser | \n", + "
| 37 | \n", + "Health | \n", + "Tune In to the Healing Powers of a Decent Play... | \n", + "https://www.wired.com/story/music-therapy-heal... | \n", + "Daniel Levitin | \n", + "
| 38 | \n", + "Human History | \n", + "The Whole Story of How Humans Evolved From Gre... | \n", + "https://www.wired.com/story/the-whole-story-of... | \n", + "John Gowlett | \n", + "
| 39 | \n", + "Health | \n", + "Why an Offline Nuclear Reactor Led to Thousand... | \n", + "https://www.wired.com/story/why-an-offline-nuc... | \n", + "Chris Baraniuk | \n", + "
\n",
+ " \n",
+ "
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️ Extract Wired Science News with Official Scrapegraph SDK"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"6Rgz5T4eHBVz"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install scrapegraph-py"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734531168564,"user_tz":-60,"elapsed":17535,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"bcdd9ae1-151e-41d6-df36-8835c660460f"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | category | \n","title | \n","link | \n","author | \n","
|---|---|---|---|---|
| 0 | \n","Science | \n","The Study That Called Out Black Plastic Utensi... | \n","https://www.wired.com/story/black-plastic-uten... | \n","Beth Mole, Ars Technica | \n","
| 1 | \n","Environment | \n","Generative AI and Climate Change Are on a Coll... | \n","https://www.wired.com/story/true-cost-generati... | \n","Sasha Luccioni | \n","
| 2 | \n","Xenotransplantation | \n","A Third Person Has Received a Transplant of a ... | \n","https://www.wired.com/story/a-third-person-has... | \n","Emily Mullin | \n","
| 3 | \n","Health | \n","Antibodies Could Soon Help Slow the Aging Process | \n","https://www.wired.com/story/antibodies-could-s... | \n","Andrew Steele | \n","
| 4 | \n","Science | \n","Good at Reading? Your Brain May Be Structured ... | \n","https://www.wired.com/story/good-at-reading-yo... | \n","Mikael Roll | \n","
| 5 | \n","Health | \n","Mega-Farms Are Driving the Threat of Bird Flu | \n","https://www.wired.com/story/mega-farms-are-dri... | \n","Georgina Gustin | \n","
| 6 | \n","Health | \n","RFK Plans to Take on Big Pharma. It’s Easier S... | \n","https://www.wired.com/story/rfks-plan-to-take-... | \n","Emily Mullin | \n","
| 7 | \n","Environment | \n","How Christmas Trees Could Become a Source of L... | \n","https://www.wired.com/story/how-christmas-tree... | \n","Alexa Phillips | \n","
| 8 | \n","Environment | \n","Creating a Global Package to Solve the Problem... | \n","https://www.wired.com/story/global-plastics-tr... | \n","Susan Solomon | \n","
| 9 | \n","Environment | \n","These 3 Things Are Standing in the Way of a Gl... | \n","https://www.wired.com/story/these-3-things-are... | \n","Steve Fletcher and Samuel Winton | \n","
\n"," \n","
+
+
+
+