Panda-> It retrieves the source code of a given URL using a get_sourcecode(url) from Module ' func' and then passes it to juicer(class) for extracting the useful information and after the work is completed, reports to the panda_manger (boss of all the pandas) and again assigns a new job to panda if any.So you would find lots of classes in the project. I have basically divided every task into small parts so that the work could become easy. It took me 3 days because every day I started from the beginning because I was not satisfied with the performance of the crawler or there was some problem. that was the time when I finally slept comfortably and full of satisfaction. After 3 sleepless nights, on the 3 rd day at 6:00 a.m., I was ready with my search engine working with 100 URLs in database. there was a thunder storm of ideas in my mind.
It seemed like an auto complete rather than a search engine, but later that night it was 2:00 am, and I couldn't sleep at all because of that auto complete feature with which I was too impressed. It all started when my friend showed me his search engine with 4 URLs in an XML file. Now these keywords are stored in a database and then used to find the relevant URL for given keywords. It is clear how tag cloud can highlight the key words which could describe a given URL.
It is not a fool proof method, but can work for sites with lots of words in it, like a blog or article or a discussion forum, etc. Like most search engines, this one also has a crawler whose basic aim is to retrieve the source code of a given URL and then break the content into words with which we can create an array of tag words which will represent the content of the site. (This is because of some unidentified problem in conversion of relative to absolute URL.) You must use it to crawl thousands of URLs because you may find that it crawls the same URL for the last 100 times.
0 kommentar(er)
