Around Paywalls? Probably Not Spot On
February 27, 2016
I read “How Google’s Web Crawler Bypasses Paywalls.” I am not confident the write up is spot in. You may find the information useful in your own efforts to do the Connotate-type or Kimono-type thing.
The outfit with the paywall tunnel, according to the write up, is Alphabet’s Google unit. Talk about the tail wagging the dog.
The write up points out that the method uses Referer and User –Agent headers.
The approach is detailed in the article via code snippets. It’s in the cards, so have at it.
Oh, there may be other methods in play, but I will leave you to your experimentation.
Stephen E Arnold, February 23, 2016
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
