We are about to switch to a new forum software. Until then we have removed the registration on this forum.
Hello, I am interested in making a web scraper that will not just search certain websites and URLs, but the entire internet for keywords, and then save the websites they find the keywords on into a Microsoft Access database. How would I do this?
Answers
The program doesn't have to be in processing either; it can be done in any text editor
That is called a "search engine", and it is really hard. Alternately, your code would need to use an existing search engine -- probably with an API key. You wouldn't want either the search crawler of the scraper / downloaded to be written in processing -- honestly, you don't want to write technologies like that yourself at all, as they are usually incredibly complex if they work well.
If you really do want to create your own, look at Apache Lucene, Nutch, Solr, ElasticSearch etc. There are tons of examples. But if this is a simple project you would probably be happier using an API to an existing search engine.