Advertisement
Help Keep Boards Alive. Support us by going ad free today. See here: https://subscriptions.boards.ie/.
https://www.boards.ie/group/1878-subscribers-forum

Private Group for paid up members of Boards.ie. Join the club.
Hi all, please see this major site announcement: https://www.boards.ie/discussion/2058427594/boards-ie-2026

Is it possible for me to see all the links of pages on a website?

  • 29-07-2020 01:54PM
    #1
    kala85
    Registered Users, Registered Users 2 Posts: 1,908 ✭✭✭


    I am looking at a company website , which shares specific links with certain people or organisations.

    The links addresses shared dont follow a certain pattern.

    Is there a program that I can use to scrape the server of this website and find out a list of links on the website.

    Some of these pages are embedded with content as well to an external webpage and I am just wondering if it is possible to find these webpages.


Comments

  • #2
    Baz_
    Registered Users, Registered Users 2 Posts: 2,660 ✭✭✭Baz_


    kala85 wrote: »

    Your question seems shadily unspecific, nevertheless, this should help https://scrapy.org/

    Note, though, that this would have been easily found with a web search...


  • #3
    kala85
    Registered Users, Registered Users 2 Posts: 1,908 ✭✭✭kala85


    Baz_ wrote: »
    Your question seems shadily unspecific, nevertheless, this should help https://scrapy.org/

    Note, though, that this would have been easily found with a web search...

    Did a search but didn't come up with that website.

    Do I have to write code or is the code there already.


  • #4
    28064212
    Registered Users, Registered Users 2 Posts: 11,105 ✭✭✭✭28064212


    kala85 wrote: »
    Did a search but didn't come up with that website.

    Do I have to write code or is the code there already.
    You'd have to write some code. However, it likely wouldn't help with what you're looking for. Web spiders work by picking one page on a website, then following all the links on that page to all the new pages, then all the links on the new pages to more new pages etc. That doesn't help if there aren't any links to your pages from any other pages on the site. If that's the case, there aren't really any practical ways to build a list of the pages you're looking for

    Boardsie Enhancement Suite - a browser extension to make using Boards on desktop a better experience (includes full-width display, keyboard shortcuts, dark mode, and more). Now available through your browser's extension store.

    Firefox: https://addons.mozilla.org/addon/boardsie-enhancement-suite/

    Chrome/Edge/Opera: https://chromewebstore.google.com/detail/boardsie-enhancement-suit/bbgnmnfagihoohjkofdnofcfmkpdmmce



  • #5
    ressem
    Registered Users, Registered Users 2, Paid Member Posts: 2,427 ✭✭✭ressem


    Depends on the website and whether it needs authentication, or makes life difficult for bots.

    You could try wget (available on linux, or windows using Cygwin)

    wget -r https://www.test.mysite
    will try to create a mirror of the content of the website's html.

    wget -r -spider https://www.test.mysite
    will crawl through the website making a note of the links it finds on the console.
    (handy for finding broken links if you no better tool available)

    and
    wget -r -spider --span-hosts https://www.test.mysite
    should follow for external links.


Advertisement