[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Web search tools
You can try UdmSearch : http://search.mnogo.ru/
> Perhaps you could help me:
>
> I'm looking for a simple web search tool or spider to map all the links
> from a certain URL starting point(s) to a certain depth, and print or
> log the URLs it encounters. I've been using "wget -r -l 5" so far but it
> cannot parallelize its requests, visits the same URLs several times and
> cannot be told to refrain from downloading files unnecessarily, which
> makes it far too slow. Does anyone have experience with or knows a tool
> that does have these features (especially parallelizing requests)?
>
> TIA,
> Adi Stav
>
=================================================================
To unsubscribe, send mail to linux-il-request@linux.org.il with
the word "unsubscribe" in the message body, e.g., run the command
echo unsubscribe | mail linux-il-request@linux.org.il