Merge branch 'main' into feature/toonify-integration

Author: VinciGit00

.agent/system/project_architecture.md

@@ -85,7 +85,14 @@ scrapegraph-sdk/
 - **aiohttp** 3.10+ - Async HTTP client
 - **pydantic** 2.10.2+ - Data validation and modeling
 - **python-dotenv** 1.0.1+ - Environment variable management
-- **beautifulsoup4** 4.12.3+ - HTML parsing (for pagination)
+
+**Optional Dependencies:**
+- **beautifulsoup4** 4.12.3+ - HTML parsing (for HTML validation when using `website_html`)
+  - Install with: `pip install scrapegraph-py[html]`
+- **langchain** 0.3.0+ - Langchain integration for AI workflows
+- **langchain-community** 0.2.11+ - Community integrations for Langchain
+- **langchain-scrapegraph** 0.1.0+ - ScrapeGraph integration for Langchain
+  - Install with: `pip install scrapegraph-py[langchain]`
 
 **Development Tools:**
 - **pytest** 7.4.0+ - Testing framework
@@ -879,12 +886,17 @@ npm publish
 
 ### Python SDK Dependencies
 
-**Runtime:**
+**Core Runtime:**
 - **requests**: Sync HTTP client
 - **aiohttp**: Async HTTP client
 - **pydantic**: Data validation
 - **python-dotenv**: Environment variables
-- **beautifulsoup4**: HTML parsing
+
+**Optional Runtime (install with extras):**
+- **beautifulsoup4**: HTML parsing (required when using `website_html`)
+  - Install with: `pip install scrapegraph-py[html]`
+- **langchain, langchain-community, langchain-scrapegraph**: Langchain integration
+  - Install with: `pip install scrapegraph-py[langchain]`
 
 **Development:**
 - **pytest & plugins**: Testing framework
@@ -918,7 +930,7 @@ Both SDKs depend on the ScrapeGraph AI API:
 | **Architecture** | Class-based (Client, AsyncClient) | Function-based |
 | **Async Support** | ✅ Separate AsyncClient | ✅ All functions async |
 | **Type Safety** | ✅ Pydantic models, mypy | ⚠️ JSDoc comments |
-| **Dependencies** | 5 runtime deps | 0 runtime deps |
+| **Dependencies** | 4 core + 2 optional extras | 0 runtime deps |
 | **Testing** | pytest with mocking | Manual tests |
 | **Documentation** | MkDocs auto-generated | README examples |
 | **Package Size** | ~50KB | ~20KB |

.github/workflows/python-publish.yml

@@ -35,7 +35,7 @@ jobs:
         python -m pip install --upgrade pip
         pip install pytest pytest-asyncio responses
         cd scrapegraph-py
-        pip install -e .
+        pip install -e ".[html]"
 
     - name: Run mocked tests with coverage
       run: |

.github/workflows/test.yml

@@ -34,7 +34,7 @@ jobs:
         python -m pip install --upgrade pip
         pip install pytest pytest-asyncio responses
         cd scrapegraph-py
-        pip install -e .
+        pip install -e ".[html]"
 
     - name: Run mocked tests with coverage
       run: |

.gitignore

@@ -4,5 +4,3 @@
 **/.DS_Store
 *.csv
 venv/
-
-__pycache__/

CLAUDE.md

@@ -44,7 +44,10 @@ scrapegraph-sdk/
 ### Python SDK
 - **Language**: Python 3.10+
 - **Package Manager**: uv (recommended) or pip
-- **Dependencies**: requests, pydantic, python-dotenv, aiohttp, beautifulsoup4
+- **Core Dependencies**: requests, pydantic, python-dotenv, aiohttp
+- **Optional Dependencies**:
+  - `html`: beautifulsoup4 (for HTML validation when using `website_html`)
+  - `langchain`: langchain, langchain-community, langchain-scrapegraph (for Langchain integrations)
 - **Testing**: pytest, pytest-asyncio, pytest-mock, aioresponses
 - **Code Quality**: black, isort, ruff, mypy, pre-commit
 - **Documentation**: mkdocs, mkdocs-material

__pycache__/test_sdk.cpython-312.pyc

@@ -0,0 +1,130 @@
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+lerna-debug.log*
+.pnpm-debug.log*
+
+# Diagnostic reports (https://nodejs.org/api/report.html)
+report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
+
+# Runtime data
+pids
+*.pid
+*.seed
+*.pid.lock
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+*.lcov
+
+# nyc test coverage
+.nyc_output
+
+# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# Bower dependency directory (https://bower.io/)
+bower_components
+
+# node-waf configuration
+.lock-wscript
+
+# Compiled binary addons (https://nodejs.org/api/addons.html)
+build/Release
+
+# Dependency directories
+node_modules/
+jspm_packages/
+
+# Snowpack dependency directory (https://snowpack.dev/)
+web_modules/
+
+# TypeScript cache
+*.tsbuildinfo
+
+# Optional npm cache directory
+.npm
+
+# Optional eslint cache
+.eslintcache
+
+# Optional stylelint cache
+.stylelintcache
+
+# Microbundle cache
+.rpt2_cache/
+.rts2_cache_cjs/
+.rts2_cache_es/
+.rts2_cache_umd/
+
+# Optional REPL history
+.node_repl_history
+
+# Output of 'npm pack'
+*.tgz
+
+# Yarn Integrity file
+.yarn-integrity
+
+# dotenv environment variable files
+.env
+.env.development.local
+.env.test.local
+.env.production.local
+.env.local
+
+# parcel-bundler cache (https://parceljs.org/)
+.cache
+.parcel-cache
+
+# Next.js build output
+.next
+out
+
+# Nuxt.js build / generate output
+.nuxt
+dist
+
+# Gatsby files
+.cache/
+# Comment in the public line in if your project uses Gatsby and not Next.js
+# https://nextjs.org/blog/next-9-1#public-directory-support
+# public
+
+# vuepress build output
+.vuepress/dist
+
+# vuepress v2.x temp and cache directory
+.temp
+.cache
+
+# Docusaurus cache and generated files
+.docusaurus
+
+# Serverless directories
+.serverless/
+
+# FuseBox cache
+.fusebox/
+
+# DynamoDB Local files
+.dynamodb/
+
+# TernJS port file
+.tern-port
+
+# Stores VSCode versions used for testing VSCode extensions
+.vscode-test
+
+# yarn v2
+.yarn/cache
+.yarn/unplugged
+.yarn/build-state.yml
+.yarn/install-state.gz
+.pnp.*

scrapegraph-js/.gitignore

@@ -0,0 +1 @@
+/node_modules

scrapegraph-js/.prettierignore

@@ -0,0 +1,11 @@
+{
+  "semi": true,
+  "singleQuote": true,
+  "trailingComma": "es5",
+  "tabWidth": 2,
+  "useTabs": false,
+  "printWidth": 110,
+  "bracketSpacing": true,
+  "arrowParens": "always",
+  "quoteProps": "preserve"
+}