[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Web search tools
check
swish-e (have problems with hebrew characters)
ht-dig (never tried)
Adi Stav wrote:
> Perhaps you could help me:
>
> I'm looking for a simple web search tool or spider to map all the links
> from a certain URL starting point(s) to a certain depth, and print or
> log the URLs it encounters. I've been using "wget -r -l 5" so far but it
> cannot parallelize its requests, visits the same URLs several times and
> cannot be told to refrain from downloading files unnecessarily, which
> makes it far too slow. Does anyone have experience with or knows a tool
> that does have these features (especially parallelizing requests)?
>
> TIA,
> Adi Stav
>
> =================================================================
> To unsubscribe, send mail to linux-il-request@linux.org.il with
> the word "unsubscribe" in the message body, e.g., run the command
> echo unsubscribe | mail linux-il-request@linux.org.il
--
--------------------------
Canaan Surfing Ltd.
Internet Service Providers
Ben-Nes Michael - Manager
Tel: 972-6-6925757
Fax: 972-6-6925858
http://www.canaan.co.il
--------------------------
=================================================================
To unsubscribe, send mail to linux-il-request@linux.org.il with
the word "unsubscribe" in the message body, e.g., run the command
echo unsubscribe | mail linux-il-request@linux.org.il