Page only loads when scrolling down

I need to crawl the entire page for elements, but only some of it loads until I scroll down. Is their anyway to counteract this? I can workaround this by running scroll to bottom functions, but it would be nice if their was a cleaner way of doing this.

1 answer

  • answered 2017-06-17 19:55 archae0pteryx

    here's the code to accomplish this. You can either plug it straight into the console or inject it into your crawler

    window.scrollTo(0,document.body.scrollHeight);
    

    you can also wrap this in a setInterval function to increment over time. Heres an example of this:

    setInterval(function() { 
    document.body.scrollTop = document.body.scrollHeight;
    }, 50);
    

    EDIT: sorry, i missed the last part of your question where you mentioned that you know you can do this. What kind of crawler are you working with. It would help in answering your question. Look at this example: Scroll Automatically to the Bottom of the Page