xk6-crawler

Example

import { Crawler } from "k6/x/crawler";

export default function () {
  const c = new Crawler({ max_depth: 2 });

  c.onHTML("a[href]", (e) => {
    if (e.attr("href").startsWith("/")) {
      e.request.visit(e.attr("href"));
    }
  });

  c.onResponse((r) => {
    console.log(r.status_code, r.request.url);
  });

  c.visit("https://grafana.com");
}
Copy

The crawler is programmed using so-called callback functions. In the example above, two callback functions are registered.

With Crawler.onHTML, you can register a callback function that will run on HTML elements that match the selector parameter. The callback function parameter is the matching HTMLElement.

Normally, the Crawler.onHTML callback function is responsible for collecting the URLs to be crawled and placing them in a queue. This is usually done using the Request.visit function of the Request object belonging to the HTMLElement.

  c.onHTML("a[href]", (e) => {
    // ...
    e.request.visit(e.attr("href"));
  });
Copy

With the Crawler.onRequest callback function, you can customize the HTTP request before it is executed. The parameter of the callback function is the Request object.

  c.onRequest((r) => {
    // ...
  });
Copy

The Crawler.onResponse callback function is called after the response has been received. The callback function parameter is the Response object.

  c.onResponse((r) => {
    // ...
  });
Copy

The Crawler.visit function can be used to initiate the download of a given URL and the execution of registered callback functions.

  c.visit("https://grafana.com");
Copy