VulnIQ web processor, like Googlebot, fetches and processes urls referenced from other data and indexes web page contents. You can use VulnIQ full text search engine to search in processed web pages.
VulnIQ web processor can also capture screenshots or pdf exports of web pages. When running private instances, VulnIQ customers can configure the web processor as they like. They can even use it to process web pages for other purposes as it is implemented as a micro-service.