Skip to content

Conversation

@Vikrant-Khedkar
Copy link
Collaborator

@Vikrant-Khedkar Vikrant-Khedkar commented Nov 14, 2025

Restore the javascript sdk

This reverts commit 3e993b3.


Note

Adds a full JavaScript SDK (scrapegraph-js) with core APIs, mock tooling, examples, tests, and config; includes new features like pagination, cookies, renderHeavyJs/stealth, and HTML/Markdown inputs.

  • SDK: Introduces scrapegraph-js/ with packaging, lint/format config, and documentation.
    • package.json, eslint.config.js, Prettier config, README and feature docs (pagination, agentic scraper).
  • Core APIs (src/):
    • smartScraper (pagination totalPages, cookies, renderHeavyJs, stealth, HTML/Markdown inputs) and status getter.
    • searchScraper (AI extraction or markdown mode via extractionMode, renderHeavyJs, stealth) and status getter.
    • scrape + request getter; markdownify + request getter; sitemap; crawl (AI/markdown modes, sitemap, depth) + getter.
    • agenticScraper + request getter; healthz; scheduledJobs CRUD/trigger/executions.
    • Exports via index.js.
  • Utilities:
    • Mock framework (utils/mockConfig.js, mockResponse.js) and shared error handler.
  • Examples: Comprehensive usage samples for all endpoints (cookies, pagination, markdown mode, sitemap, crawl, agentic, stealth, scheduled jobs).
  • Tests: Validation/behavior tests for new features and endpoints (pagination, HTML/Markdown, renderHeavyJs, sitemap, crawl markdown, search markdown, stealth, healthz, scheduled jobs).
  • Misc: Updates root .gitignore; adds examples/.env.example.

Written by Cursor Bugbot for commit 997ae16. This will update automatically on new commits. Configure here.

@gitguardian
Copy link

gitguardian bot commented Nov 14, 2025

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@github-actions
Copy link

github-actions bot commented Nov 14, 2025

Dependency Review

The following issues were found:

  • ❌ 1 vulnerable package(s)
  • ✅ 0 package(s) with incompatible licenses
  • ✅ 0 package(s) with invalid SPDX license definitions
  • ✅ 0 package(s) with unknown licenses.
  • ⚠️ 7 packages with OpenSSF Scorecard issues.

View full job summary

}

try {
const response = await axios.post(endpoint, payload, { headers });
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Crawl: Extraction mode option is non-functional.

The crawl function accepts an extractionMode option according to the JSDoc comments and is used in multiple examples, but this parameter is never destructured from the options object or added to the API payload. The documentation and examples show extractionMode controlling whether AI extraction (true) or markdown conversion (false) is used, but this setting is never actually sent to the API, causing the function to ignore this important configuration option.

Fix in Cursor Fix in Web

@VinciGit00 VinciGit00 merged commit 002e4ab into main Nov 14, 2025
4 of 9 checks passed
@github-actions
Copy link

🎉 This PR is included in version 1.41.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

@VinciGit00 VinciGit00 deleted the revert/restore-js-sdk-3e993b3 branch November 14, 2025 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants