Howdy, Stranger!
We are about to switch to a new forum software. Until then we have removed the registration on this forum.
Categories
- All Categories 25.7K
- Announcements & Guidelines 13
- Common Questions 30
- Using Processing 22.1K
- Programming Questions 12.2K
- Questions about Code 6.4K
- How To... 4.2K
- Hello Processing 72
- GLSL / Shaders 292
- Library Questions 4K
- Hardware, Integration & Other Languages 2.7K
- Kinect 668
- Arduino 1K
- Raspberry PI 188
- Questions about Modes 2K
- Android Mode 1.3K
- JavaScript Mode 413
- Python Mode 205
- Questions about Tools 100
- Espanol 5
- Developing Processing 548
- Create & Announce Libraries 211
- Create & Announce Modes 19
- Create & Announce Tools 29
- Summer of Code 2018 93
- Rails Girls Summer of Code 2017 3
- Summer of Code 2017 49
- Summer of Code 2016 4
- Summer of Code 2015 40
- Summer of Code 2014 22
- p5.js 1.6K
- p5.js Programming Questions 947
- p5.js Library Questions 315
- p5.js Development Questions 31
- General 1.4K
- Events & Opportunities 288
- General Discussion 365
In this Discussion
- jeremydouglass March 2017
- troym811 March 2017
Scraping
Hello, I am interested in making a web scraper that will not just search certain websites and URLs, but the entire internet for keywords, and then save the websites they find the keywords on into a Microsoft Access database. How would I do this?
Answers
The program doesn't have to be in processing either; it can be done in any text editor
That is called a "search engine", and it is really hard. Alternately, your code would need to use an existing search engine -- probably with an API key. You wouldn't want either the search crawler of the scraper / downloaded to be written in processing -- honestly, you don't want to write technologies like that yourself at all, as they are usually incredibly complex if they work well.
If you really do want to create your own, look at Apache Lucene, Nutch, Solr, ElasticSearch etc. There are tons of examples. But if this is a simple project you would probably be happier using an API to an existing search engine.