urlopen#

The urllib.request module provides the most basic method to construct HTTP requests. It can simulate a browser’s request initiation process, and it also handles authentication, redirection, browser cookies, and other content.

Take the Python official site as an example; we’ll fetch this page:

1
import urllib.request
2

3
response = urllib.request.urlopen('<https://www.python.org>')
4
print(response.read().decode('utf-8'))
5

6
print(type(response))

An HTTPResponse object mainly contains methods such as read, readinto, getheader, getheaders, fileno, and attributes like msg, version, status, reason, debuglevel, and closed.

After obtaining this object, assign it to a variable named response, and you can call these methods and attributes to obtain a return information series.

1
import urllib.request
2

3
response = urllib.request.urlopen('<https://www.python.org>')
4
print(response.status)
5
print(response.getheaders())
6
print(response.getheader('Server'))
7

8
print(type(response))
9
# <class 'http.client.HTTPResponse'>

It can be seen that the return is an HTTPResponse type object, mainly containing methods such as read, readinto, getheader, getheaders, fileno, and attributes like msg, version, status, reason, debuglevel, and closed.

After obtaining this object, assign it to a variable named response, and you can call these methods and attributes to obtain a series of information about the return result.

For example, calling read can obtain the web page content, and accessing the status attribute gives the status code of the response, where 200 indicates success and 404 indicates not found, etc.

Using the most basic urlopen method, you can complete the simplest GET request for a web page.

If you want to pass parameters in the link, how can you implement that? First, look at the urlopen method API:

urllib.request.urlopen(url, data=None, [timeout,]*, cafile=None, capath=None, cadefault=False, context=None)

You can see that besides the first argument which can pass a URL, we can also pass other contents, such as data (payload), timeout (timeout) and so on.

• data parameter The data parameter is optional. If you want to add this parameter, you must convert the parameter to a bytes type using the bytes method. Also, if this parameter is passed, the request method is no longer GET but POST.

  1
  import urllib.parse
  2
  import urllib.request
  3
  
  4
  data = bytes(urllib.parse.urlencode({'word': 'hello'}), encoding='utf8')
  5
  response = urllib.request.urlopen('<http://httpbin.org/post>', data=data)
  6
  print(response.read())

  Here, we pass a parameter word with the value hello. It needs to be encoded into bytes. The bytes method requires the first argument to be a string, and you should use urllib.parse.urlencode to convert the parameter dictionary into a string; the second argument specifies the encoding, here utf8.

  The target site here is httpbin.org, which provides HTTP request testing. Our request URL is http://httpbin.org/post, a URL that can be used to test POST requests. It outputs some information about the Request, including the data parameter we passed.

  The result is as follows:

  1
  {
  2
      "args": {},
  3
      "data": "",
  4
      "files": {},
  5
      "form": {
  6
          "word": "hello"
  7
      },
  8
      "headers": {
  9
          "Accept-Encoding": "identity",
  10
          "Content-Length": "10",
  11
          "Content-Type": "application/x-www-form-urlencoded",
  12
          "Host": "httpbin.org",
  13
          "User-Agent": "Python-urllib/3.5"
  14
      },
  15
      "json": null,
  16
      "origin": "123.124.23.253",
  17
      "url":"<http://httpbin.org/post>"
  18
  }

  As you can see, the parameter we passed appears in the form field, indicating that we simulated a form submission using the POST method to transmit data.

• timeout parameter

  The timeout parameter is used to set a timeout in seconds. It means that if the request exceeds this time without a response, an exception will be raised. If this parameter is not specified, a global default time will be used. It supports HTTP, HTTPS, and FTP requests.

  1
  import urllib.request
  2
  
  3
  response = urllib.request.urlopen('<http://httpbin.org/get>', timeout=1)
  4
  print(response.read())

  Here the timeout is set to 1 second. After 1 second, the server still hasn’t responded, so an URLError exception is raised. This exception comes from the urllib.error module, and the reason is a timeout.

  Therefore, you can control a webpage’s crawl by setting this timeout. This can be implemented with try-except as follows:

  1
  import socket
  2
  import urllib.request
  3
  import urllib.error
  4
  
  5
  try:
  6
      response = urllib.request.urlopen('<http://httpbin.org/get>', timeout=0.1)
  7
  except urllib.error.URLError as e:
  8
      if isinstance(e.reason, socket.timeout):
  9
          print('TIME OUT')

  Here we request the test URL http://httpbin.org/get with a timeout of 0.1 seconds, and then catch the URLError exception, checking that the reason is socket.timeout, meaning a timeout error occurred.

  Timeout handling via the timeout parameter can be very useful.

• Other parameters

  In addition to data and timeout, there are also context, cafile, capath, cadefault, etc. The context must be an ssl.SSLContext object to specify SSL settings. Besides, cafile and capath specify CA certificates and their paths, which are useful when requesting HTTPS links. The cadefault parameter has been deprecated and its default is False.

  We have explained how to use the urlopen method above. With this fundamental method, we can complete simple requests and webpage crawling. For more detailed information, refer to the official documentation: https://docs.python.org/3/library/urllib.request.html.

Request#

We know that using the urlopen method can initiate the most basic request, but these simple parameters are not enough to build a complete request. If you need to add information such as headers in the request, you can use the more powerful Request class to construct it.

First, let’s feel the usage of Request with an example:

1
import urllib.request
2

3
request = urllib.request.Request('<https://python.org>')
4
response = urllib.request.urlopen(request)
5
print(response.read().decode('utf-8'))

We can see that we still use the urlopen method to send this request; the only difference is that this time the argument is not a URL but a Request object. By constructing this data structure, we can separate the request into an object and configure parameters more richly and flexibly.

Its constructor is as follows:

class urllib.request.Request(url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None)

1. The first parameter url is the request URL and is required; the others are optional.
2. The second parameter data, if provided, must be of type bytes. If it is a dictionary, you can first use urllib.parse.urlencode() to encode it.
3. The third parameter headers is a dictionary representing the request headers. You can construct it directly via the headers parameter when creating the request, or you can add headers later by calling add_header() on the request instance. Adding request headers most commonly via modifying the User-Agent to masquerade as a browser. The default User-Agent is Python-urllib; you can modify it to masquerade as a browser. For example, to masquerade as Firefox, you can set it to: Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127 Firefox/
4. The fourth parameter origin_req_host refers to the host name or IP address of the requester.
5. The fifth parameter unverifiable indicates whether this request is verifiable; the default is False, meaning the user does not have sufficient permission to choose the outcome of receiving this request. For example, if we request an image within an HTML document but we do not have permission to automatically fetch images, the unverifiable value would be True.
6. The sixth parameter method is a string used to indicate the method used by the request, such as GET, POST, PUT, etc.

1
from urllib import request, parse
2

3
url = '<http://httpbin.org/post>'
4
headers = {'User-Agent': 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)',
5
    'Host': 'httpbin.org'
6
}
7
dict = {'name': 'Germey'}
8
data = bytes(parse.urlencode(dict), encoding='utf8')
9
req = request.Request(url=url, data=data, headers=headers, method='POST')
10
response = request.urlopen(req)
11
print(response.read().decode('utf-8'))

Here we construct a request with four parameters, where url is the request URL, headers specify User-Agent and Host, data is converted to bytes via urlencode and bytes, and the request method is set to POST.

The following shows the result:

1
{
2
    "args": {},
3
    "data": "",
4
    "files": {},
5
    "form": {
6
        "name": "Germey"
7
    },
8
    "headers": {
9
        "Accept-Encoding": "identity",
10
        "Content-Length": "11",
11
        "Content-Type": "application/x-www-form-urlencoded",
12
        "Host": "httpbin.org",
13
        "User-Agent": "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)"
14
    },
15
    "json": null,
16
    "origin": "219.224.169.11",
17
    "url":"<http://httpbin.org/post>"
18
}

From the result, we can see that we successfully set data, headers, and method.

Headers can also be added using the add_header method:

1
req = request.Request(url=url, data=data, method='POST')
2
req.add_header('User-Agent', 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)')

This allows for easier request construction and sending.

Advanced Usage#

In the previous process, although we could construct requests, for more advanced operations (such as cookies handling, proxy settings, etc.), what should we do?

Next, we need more powerful tools: Handlers. In short, you can think of them as various processors—some for login validation, some for handling cookies, some for managing proxies. With them, you can almost do everything in HTTP requests.

First, let’s introduce the BaseHandler class inside the urllib.request module. It is the parent class of all other Handlers and provides the most basic methods, such as default_open, protocol_request, etc.

Next, there are various Handler subclasses that inherit from BaseHandler, for example:

• HTTPDefaultErrorHandler: handles HTTP response errors; errors will raise HTTPError exceptions.
• HTTPRedirectHandler: handles redirection.
• HTTPCookieProcessor: handles Cookies.
• ProxyHandler: configures proxies; the default proxy is none.
• HTTPPasswordMgr: manages passwords; maintains a table of username/password.
• HTTPBasicAuthHandler: handles authentication; if a link requires authentication, this can resolve it.

There are other Handler classes as well; for details, refer to the official documentation: https://docs.python.org/3/library/urllib.request.html#urllib.request.BaseHandler

Another important class is OpenerDirector, which we can call Opener. We have used urlopen before; in fact, it is an Opener provided by urllib for us.

Why introduce Opener? Because it enables more advanced functionality.

An Opener can use the open method; the return type is the same as urlopen, and it can be built using Handlers.

• Authentication

  Some sites present a login prompt on open, requiring you to enter a username and password; you can view the page only after successful authentication.

  To request such a page, you can complete it with HTTPBasicAuthHandler; the relevant code is:

  1
  from urllib.request import HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler, build_opener
  2
  from urllib.error import URLError
  3
  
  4
  username = 'username'
  5
  password = 'password'
  6
  url = '<http://localhost:5000/>'
  7
  
  8
  p = HTTPPasswordMgrWithDefaultRealm()
  9
  p.add_password(None, url, username, password)
  10
  auth_handler = HTTPBasicAuthHandler(p)
  11
  opener = build_opener(auth_handler)
  12
  
  13
  try:
  14
      result = opener.open(url)
  15
      html = result.read().decode('utf-8')
  16
      print(html)
  17
  except URLError as e:
  18
      print(e.reason)

  Here we first instantiate an HTTPBasicAuthHandler object, whose parameter is an HTTPPasswordMgrWithDefaultRealm object. It uses the add_password method to add the username and password, thereby creating a Handler to perform the authentication.

  Next, use this Handler and build_opener to construct an Opener; this Opener acts as if authentication has already succeeded when sending requests. Then, use the Opener’s open method to open the link, completing the authentication. The result is the page source after authentication.

• Proxies

  Some sites monitor access counts from a given IP. If the count is too high, access may be blocked. We can configure proxies and switch IPs periodically to continue crawling even if one IP is blocked.

  Proxy settings are also implemented via Handlers. Here is how to use ProxyHandler:

  1
  from urllib.error import URLError
  2
  from urllib.request import ProxyHandler, build_opener
  3
  
  4
  proxy_handler = ProxyHandler({
  5
      'http': '<http://127.0.0.1>：9743',
  6
      'https': '<https://127.0.0.1:9743>'
  7
  })
  8
  opener = build_opener(proxy_handler)
  9
  try:
  10
      response = opener.open('<https://www.baidu.com>')
  11
      print(response.read().decode('utf-8'))
  12
  except URLError as e:
  13
      print(e.reason)

  Here we set up a local proxy running on port 9743. ProxyHandler’s argument is a dictionary where the keys are protocol types (e.g., HTTP or HTTPS) and the values are proxy URLs; you can add multiple proxies.

  Then, using this Handler and build_opener, construct an Opener and send requests.

• Cookies

  Handling cookies requires the corresponding Handler.

  1
  import http.cookiejar, urllib.request
  2
  
  3
  cookie = http.cookiejar.CookieJar()
  4
  handler = urllib.request.HTTPCookieProcessor(cookie)
  5
  opener = urllib.request.build_opener(handler)
  6
  response = opener.open('<http://www.baidu.com>')
  7
  for item in cookie:
  8
      print(item.name+"="+item.value)

  First, declare a CookieJar object. Next, use HTTPCookieProcessor to build a Handler, and finally use build_opener to construct an Opener and call open.

  You can see that it prints the name and value of each cookie.

  Cookies can also be saved to a file format. Cookies are stored as text as well.

  1
  filename = 'cookies.txt'
  2
  cookie = http.cookiejar.MozillaCookieJar(filename)
  3
  handler = urllib.request.HTTPCookieProcessor(cookie)
  4
  opener = urllib.request.build_opener(handler)
  5
  response = opener.open('<http://www.baidu.com>')
  6
  cookie.save(ignore_discard=True, ignore_expires=True)

  Here, CookieJar needs to be MozillaCookieJar, which is used when generating files. It handles cookies and file-related events, such as reading and saving cookies, and can save cookies in Mozilla-style browser format.

  LWPCookieJar can also read and save cookies, but its format differs from MozillaCookieJar and saves cookies in the libwww-perl (LWP) format.

  To save as an LWP cookies file, declare it as: cookie = http.cookiejar.LWPCookieJar(filename)

  After generating the cookies file, how do you read it from the file and use it?

  1
  cookie = http.cookiejar.LWPCookieJar()
  2
  cookie.load('cookies.txt', ignore_discard=True, ignore_expires=True)
  3
  handler = urllib.request.HTTPCookieProcessor(cookie)
  4
  opener = urllib.request.build_opener(handler)
  5
  response = opener.open('<http://www.baidu.com>')
  6
  print(response.read().decode('utf-8'))

  You can see that this uses load to read the local Cookies file and obtains its contents. The prerequisite is that you first generate a LWPCookieJar-formatted cookie and save it to a file, then read the cookies and use the same method to build a Handler and an Opener to complete the operation.

  If the page returns normally, you will see the source of the Baidu page.

  The above methods allow you to implement most request feature configurations.

  This is the basic usage of the request module in urllib. For more features, refer to the official documentation: https://docs.python.org/3/library/urllib.request.html#basehandler-objects

URLError#

URLError is a class from urllib’s error module; it inherits from OSError and is the base class for error exceptions produced by the request module. It has an attribute reason that returns the cause of the error.

1
from urllib import request, error
2
try:
3
    response = request.urlopen('<https://cuiqingcai.com/index.htm>')
4
except error.URLError as e:
5
    print(e.reason)

If we open a non-existent page, an error would normally be raised, but in this case we catch URLError and print the reason, which would be:

Not Found

The program does not crash, and the error is handled gracefully.

HTTPError#

It is a subclass of URLError, specifically for HTTP request errors, such as authentication failures. It has three attributes:

• code: the HTTP status code, e.g., 404 for not found, 500 for server error.
• reason: like the parent, the error reason.
• headers: the response headers.

1
from urllib import request,error
2
try:
3
    response = request.urlopen('<https://cuiqingcai.com/index.htm>')
4
except error.HTTPError as e:
5
    print(e.reason, e.code, e.headers, sep='\n')

The result is:

Not Found 404 Server: nginx/1.4.6 (Ubuntu) Date: Wed, 03 Aug 2016 08:54<22> GMT Content-Type: text/html; charset=UTF-8 Transfer-Encoding: chunked Connection: close X-Powered-By: PHP/5.5.9-1ubuntu4.14 Vary: Cookie Expires: Wed, 11 Jan 1984 05:00<00> GMT Cache-Control: no-cache, must-revalidate, max-age=0 Pragma: no-cache Link: https://cuiqingcai.com/wp-json/; rel=“https://api.w.org/”

Here we catch HTTPError and print reason, code, and headers.

Since URLError is the parent of HTTPError, a better approach is to catch the subclass first, then the parent:

1
from urllib import request, error
2

3
try:
4
    response = request.urlopen('<https://cuiqingcai.com/index.htm>')
5
except error.HTTPError as e:
6
    print(e.reason, e.code, e.headers, sep='\n')
7
except error.URLError as e:
8
    print(e.reason)
9
else:
10
    print('Request Successfully')

This way, you can first catch HTTPError and obtain its status code, reason, and headers. If it’s not an HTTPError, you’ll catch URLError and print the reason. Finally, use an else to handle normal logic. This is a good exception-handling pattern.

Sometimes, the reason attribute may not be a string; it can be an object as well.

1
import socket
2
import urllib.request
3
import urllib.error
4

5
try:
6
    response = urllib.request.urlopen('<https://www.baidu.com>', timeout=0.01)
7
except urllib.error.URLError as e:
8
    print(type(e.reason))
9
    if isinstance(e.reason, socket.timeout):
10
        print('TIME OUT')

Here we directly set a timeout to force a timeout exception. The result is:

<class ‘socket.timeout’> TIME OUT

As you can see, the reason attribute is the socket.timeout class. So you can use isinstance to check its type for more precise error handling.

urlparse#

This method can identify and split a URL

1
from urllib.parse import urlparse
2

3
result = urlparse('<http://www.baidu.com/index.html;user?id=5#comment>')
4
print(type(result), result)

<class 'urllib.parse.ParseResult'> ParseResult(scheme='http', netloc='www.baidu.com', path='/index.html', params='user', query='id=5', fragment='comment')

You’ll see that the result is a ParseResult object containing six parts: scheme, netloc, path, params, query, and fragment.

It shows that urlparse splits into six parts. Generally, there are specific delimiters in parsing. For example, the part before :// is the scheme (the protocol); the first / is the netloc (domain); the part after is the path; the semicolon leads to params; the question mark indicates query; the hash denotes the fragment.

Thus, the standard URL format is:

scheme://netloc/path;params?query#fragment

A standard URL adheres to this rule, and urlparse can split it.

1
# API configuration
2
urllib.parse.urlparse(urlstring, scheme='', allow_fragments=True)

It has three parameters:

• urlstring: required; the URL to be parsed.
• scheme: the default protocol (e.g., http or https). The scheme parameter only takes effect if the URL does not contain a scheme. If the URL contains a scheme, the parsed scheme is returned.
• allow_fragments: whether to ignore the fragment. If set to False, the fragment portion will be ignored and parsed as part of path, parameters, or query, and fragment becomes empty.

urlunparse#

With urlparse, there is the opposite method urlunparse. It accepts an iterable object, but its length must be 6; otherwise you’ll get an error about insufficient or excessive arguments.

1
from urllib.parse import urlunparse
2

3
data = ['http', 'www.baidu.com', 'index.html', 'user', 'a=6', 'comment']
4
print(urlunparse(data))

Here, data is a list. Of course, you can also use other types, such as a tuple or a specific data structure.

http://www.baidu.com/index.html;user?a=6#comment

urlsplit#

This method is very similar to urlparse, except it does not parse the params section separately; it returns five results. In the above example, params are merged into path.

1
from urllib.parse import urlsplit
2

3
result = urlsplit('<http://www.baidu.com/index.html;user?id=5#comment>')
4
print(result)

The result is SplitResult, which is a tuple-like object; you can access its values via attributes or indices.

urlunsplit#

Similar to urlunparse, it also combines URL parts into a complete URL. The parameter is an iterable object, for example a list or tuple; the only difference is the length must be 5.

1
from urllib.parse import urlunsplit
2

3
data = ['http', 'www.baidu.com', 'index.html', 'a=6', 'comment']
4
print(urlunsplit(data))

http://www.baidu.com/index.html?a=6#comment

urljoin#

With urlunparse and urlunsplit, we can merge links, but a base_url must be provided with scheme, netloc, and path. Another method to generate links is urljoin. You can provide a base_url as the first argument and the new link as the second; the method analyzes base_url’s scheme, netloc, and path, fills in the missing parts of the new link, and returns the result.

1
from urllib.parse import urljoin
2

3
print(urljoin('<http://www.baidu.com>', 'FAQ.html'))
4
print(urljoin('<http://www.baidu.com>', '<https://cuiqingcai.com/FAQ.html>'))
5
print(urljoin('<http://www.baidu.com/about.html>', '<https://cuiqingcai.com/FAQ.html>'))
6
print(urljoin('<http://www.baidu.com/about.html>', '<https://cuiqingcai.com/FAQ.html?question=2>'))
7
print(urljoin('<http://www.baidu.com?wd=abc>', '<https://cuiqingcai.com/index.php>'))
8
print(urljoin('<http://www.baidu.com>', '?category=2#comment'))
9
print(urljoin('www.baidu.com', '?category=2#comment'))
10
print(urljoin('www.baidu.com#comment', '?category=2'))

The results are:

1
<http://www.baidu.com/FAQ.html>
2
<https://cuiqingcai.com/FAQ.html>
3
<https://cuiqingcai.com/FAQ.html>
4
<https://cuiqingcai.com/FAQ.html?question=2>
5
<https://cuiqingcai.com/index.php>
6
<http://www.baidu.com?category=2#comment>
7
www.baidu.com?category=2#comment
8
www.baidu.com?category=2

You can see base_url provides three parts: scheme, netloc, and path. If these three parts do not exist in the new link, they will be filled in; if the new link already has them, those parts from the new link are used. The params, query, and fragment in base_url do not apply.

With urljoin, you can easily perform link parsing, concatenation, and generation.

urlencode#

Here we introduce another commonly used method—urlencode—which is very useful for constructing GET request parameters.

1
from urllib.parse import urlencode
2

3
params = {
4
    'name': 'germey',
5
    'age': 22
6
}
7
base_url = '<http://www.baidu.com>?'
8
url = base_url + urlencode(params)
9
print(url)

Here we first declare a dictionary to represent the parameters, then call urlencode to serialize them into a GET parameter string.

http://www.baidu.com?name=germey&age=22

As you can see, the parameters have been converted from a dictionary to GET parameters. This method is very common. For convenience, you may represent parameters as a dictionary beforehand; to convert to URL parameters, simply call this method.

parse_qs#

Once serialized, there is necessarily deserialization. If we have a string of GET request parameters, using parse_qs we can convert it back to a dictionary:

1
from urllib.parse import parse_qs
2

3
query = 'name=germey&age=22'
4
print(parse_qs(query))

parse_qsl#

There is also a parse_qsl method, which converts parameters into a list of tuples.

1
from urllib.parse import parse_qsl
2

3
query = 'name=germey&age=22'
4
print(parse_qsl(query))

quote#

This method can convert content into URL-encoded format. When a URL contains non-ASCII characters, this can cause garbling; this method converts Chinese characters into URL encoding.

1
from urllib.parse import quote
2

3
keyword = ' 壁纸 '
4
url = '<https://www.baidu.com/s?wd=>' + quote(keyword)
5
print(url)

Here we declare a Chinese search term and URL-encode it with quote, yielding:

https://www.baidu.com/s?wd=% E5% A3% 81% E7% BA% B8

unquote#

With quote comes unquote, which decodes URL-encoded content:

1
from urllib.parse import unquote
2

3
url = '<https://www.baidu.com/s?wd=%> E5% A3% E7% BA% B8'
4
print(unquote(url))

This is the URL-encoded result obtained above; unquote decodes it to:

https://www.baidu.com/s?wd = 壁纸

As you can see, unquote makes decoding easy.

Robots Protocol#

The Robots Protocol, also called the crawler protocol or robot protocol, is short for the Robots Exclusion Protocol. It tells crawlers and search engines which pages may be crawled and which should not be crawled. It is typically a text file named robots.txt, usually placed in the site’s root directory.

When a search crawler visits a site, it first checks whether a robots.txt file exists in the site’s root directory. If it exists, the crawler crawls according to the scope defined there. If not, the crawler will access all directly accessible pages.

Here is a sample robots.txt:

1
User-agent: *
2
Disallow: /
3
Allow: /public/3

This restricts all crawlers to crawl only the public directory. Save the above content as robots.txt in the site’s root directory, alongside the site’s entry files (such as index.php, index.html, index.jsp, etc.). The User-agent describes the crawler’s name; setting it to * means the protocol applies to any crawler. For example:

User-agent: Baiduspider

This means the rules apply to Baidu’s crawler. If there are multiple User-agent lines, multiple crawlers will be affected, but at least one should be specified.

Disallow specifies the directories not allowed to crawl (in the above example, / means disallow crawling all pages). Allow is usually used together with Disallow to carve out exceptions. Now we set Allow to /public/, meaning all pages are not crawlable except the public directory.

Let’s look at a few more examples.

1
<!-- Code below forbids all crawlers from accessing any directory: -->
2
User-agent: *
3
Disallow: /
4
<!-- Code below allows all crawlers to access any directory: -->
5
User-agent: *
6
Disallow:
7
<!-- Also, leaving robots.txt empty is allowed. -->
8
<!-- Code below forbids all crawlers from accessing certain website directories: -->
9
User-agent: *
10
Disallow: /private/
11
Disallow: /tmp/
12
<!-- Code below allows only one crawler to access: -->
13
User-agent: WebCrawler
14
Disallow:
15
User-agent: *
16
Disallow: /

These are common patterns for robots.txt.

Crawler Names#

You might wonder where crawler names come from; they have fixed names, for example, Baidu’s crawler is called BaiduSpider.

robotparser#

After understanding the Robots protocol, we can use robotparser to parse robots.txt. This module provides the RobotFileParser class, which can determine whether a given crawler has permission to crawl a website’s pages based on the site’s robots.txt.

The class is simple to use: simply pass the robots.txt URL to the constructor. First, here is its declaration:

urllib.robotparser.RobotFileParser(url='')

Of course, you can also instantiate it without passing a URL, defaulting to empty, and then use set_url() to set it.

Here are the commonly used methods of this class:

• set_url: sets the robots.txt URL. If the link is provided when creating the RobotFileParser, you don’t need to call this method.
• read: reads robots.txt and analyzes it. Note that this method performs a read and analyze operation. If you don’t call this method, subsequent checks will return False, so be sure to call it. This method does not return content, but performs the read operation.
• parse: parses the contents of robots.txt. The argument is some lines from robots.txt; it will analyze these contents according to robots.txt syntax.
• can_fetch: this method takes two parameters: the first is User-agent, the second is the URL to fetch. It returns True if the search engine is allowed to crawl the URL, otherwise False.
• mtime: returns the last time robots.txt was fetched and analyzed; useful for long-running crawlers.
• modified: similarly updates the time to now for long-running crawlers.

Let’s see an example:

1
from urllib.robotparser import RobotFileParser
2
rp = RobotFileParser()
3
rp.set_url('<http://www.jianshu.com/robots.txt>')
4
rp.read()
5
print(rp.can_fetch('*', '<http://www.jianshu.com/p/b67554025d7d>'))
6
print(rp.can_fetch('*', "<http://www.jianshu.com/search?q=python&page=1&type=collections>"))

Here we use JianShu (jianshu) as an example. First we create a RobotFileParser object, set the robots.txt link with set_url. Of course, without this method you can also set it directly at declaration: rp = RobotFileParser('<http://www.jianshu.com/robots.txt>'). Then we use can_fetch to determine if a page can be crawled. The output is:

1
True
2
False

We can also use the parse method to read and analyze, as follows:

1
from urllib.robotparser import RobotFileParser
2
from urllib.request import urlopen
3
rp = RobotFileParser()
4
rp.parse(urlopen('<http://www.jianshu.com/robots.txt').read().decode('utf-8').split('\\n>'))
5
print(rp.can_fetch('*', '<http://www.jianshu.com/p/b67554025d7d>'))
6
print(rp.can_fetch('*', "<http://www.jianshu.com/search?q=python&page=1&type=collections>"))

GET Requests#

One of the most common HTTP requests is GET. Here we’ll detail how to construct GET requests with requests.

Basic example

First, build a simple GET request to http://httpbin.org/get. That site will return information about the request if the client uses GET:

1
import requests
2

3
r = requests.get('<http://httpbin.org/get>')
4
print(r.text)

The result:

1
{"args": {},
2
"headers": {
3
    "Accept": "*/*",
4
    "Accept-Encoding": "gzip, deflate",
5
    "Host": "httpbin.org",
6
    "User-Agent": "python-requests/2.10.0"
7
},
8
"origin": "122.4.215.33",
9
"url": "<http://httpbin.org/get>"
10
}

We can see that the GET request succeeded and the response includes the request headers, URL, IP, etc.

If you want to attach extra information to GET requests, you can use the params parameter to store them:

1
import requests
2

3
data = {
4
    'name': 'germey',
5
    'age': 22
6
}
7
r = requests.get("<http://httpbin.org/get>", params=data)
8
print(r.text)

The request URL will be automatically constructed as: http://httpbin.org/get?age=22&name=germey.

Additionally, the response is a string that is actually in JSON format. If you want to parse it directly into a dictionary, you can call the json method. For example:

1
import requests
2

3
r = requests.get("<http://httpbin.org/get>")
4
print(type(r.text))
5
print(r.json())
6
print(type(r.json()))

Calling json converts the JSON-formatted string into a dictionary.

Note that if the response isn’t JSON, parsing will raise json.decoder.JSONDecodeError.

• Fetching a webpage

The above URL returns JSON-formatted content. If you fetch a regular webpage, you’ll get the page content. For example, here is how to fetch Zhihu’s Explore page:

1
import requests
2
import re
3

4
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
5
}
6
r = requests.get("<https://www.zhihu.com/explore>", headers=headers)
7
pattern = re.compile('explore-feed.*?question_link.*?>(.*?)</a>', re.S)
8
titles = re.findall(pattern, r.text)
9
print(titles)

Here we include headers, which contain the User-Agent header to masquerade as a browser. Without this, Zhihu may block Crawling.

• Fetching binary data

In the above example, Zhihu’s page is HTML. If you want to fetch images, audio, video, etc., what to do? Images, audio, and video are binary data and require saving in binary format with appropriate encoding.

For example, GitHub’s site favicon:

1
import requests
2

3
r = requests.get("<https://github.com/favicon.ico>")
4
print(r.text)
5
print(r.content)

This fetches the site icon displayed in a browser tab.

Then save the image:

1
import requests
2

3
r = requests.get("<https://github.com/favicon.ico>")
4
with open('favicon.ico', 'wb') as f:
5
    f.write(r.content)

Here we used the open method to write in binary mode to a file, which creates a favicon.ico file when done.

Similarly, audio and video can be obtained this way.

• Adding headers

As with urllib.request, you can also pass header information via the headers parameter.

1
import requests
2

3
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
4
}
5
r = requests.get("<https://www.zhihu.com/explore>", headers=headers)
6
print(r.text)

You can add other fields in the headers parameter as well.

4. POST Requests

Earlier we learned the basics of GET requests; another common method is POST.

1
import requests
2

3
data = {'name': 'germey', 'age': '22'}
4
r = requests.post("<http://httpbin.org/post>", data=data)
5
print(r.text)

5. Response

After sending a request, you naturally receive a response. In the examples above, we used text and content to obtain the response, and there are many other attributes and methods for obtaining additional information such as status code, headers, cookies, etc.

1
import requests
2

3
r = requests.get('<http://www.jianshu.com>')
4
print(type(r.status_code), r.status_code)
5
print(type(r.headers), r.headers)
6
print(type(r.cookies), r.cookies)
7
print(type(r.url), r.url)
8
print(type(r.history), r.history)

Here we print the status_code (status), headers, cookies, url, and history to show their types.

The status code is commonly used to determine if the request was successful, and requests also provides a built-in status code lookup object, requests.codes.

1
import requests
2

3
r = requests.get('<http://www.jianshu.com>')
4
exit() if not r.status_code == requests.codes.ok else print('Request Successfully')

This compares the status code to the built-in success code to ensure a proper response; here, requests.codes.ok gives the success status code 200.

The following is a list of status codes and their query values:

1
# Informational status codes
2
100: ('continue',),
3
101: ('switching_protocols',),
4
102: ('processing',),
5
103: ('checkpoint',),
6
122: ('uri_too_long', 'request_uri_too_long'),
7

8
# Successful status codes
9
200: ('ok', 'okay', 'all_ok', 'all_okay', 'all_good', '\\\\o/', '✓'),
10
201: ('created',),
11
202: ('accepted',),
12
203: ('non_authoritative_info', 'non_authoritative_information'),
13
204: ('no_content',),
14
205: ('reset_content', 'reset'),
15
206: ('partial_content', 'partial'),
16
207: ('multi_status', 'multiple_status', 'multi_stati', 'multiple_stati'),
17
208: ('already_reported',),
18
226: ('im_used',),
19

20
# Redirection status codes
21
300: ('multiple_choices',),
22
301: ('moved_permanently', 'moved', '\\\\o-'),
23
302: ('found',),
24
303: ('see_other', 'other'),
25
304: ('not_modified',),
26
305: ('use_proxy',),
27
306: ('switch_proxy',),
28
307: ('temporary_redirect', 'temporary_moved', 'temporary'),
29
308: ('permanent_redirect',
30
        'resume_incomplete', 'resume',), # These 2 to be removed in 3.0
31

32
# Client error status codes
33
400: ('bad_request', 'bad'),
34
401: ('unauthorized',),
35
402: ('payment_required', 'payment'),
36
403: ('forbidden',),
37
404: ('not_found', '-o-'),
38
405: ('method_not_allowed', 'not_allowed'),
39
406: ('not_acceptable',),
40
407: ('proxy_authentication_required', 'proxy_auth', 'proxy_authentication'),
41
408: ('request_timeout', 'timeout'),
42
409: ('conflict',),
43
410: ('gone',),
44
411: ('length_required',),
45
412: ('precondition_failed', 'precondition'),
46
413: ('request_entity_too_large',),
47
414: ('request_uri_too_large',),
48
415: ('unsupported_media_type', 'unsupported_media', 'media_type'),
49
416: ('requested_range_not_satisfiable', 'requested_range', 'range_not_satisfiable'),
50
417: ('expectation_failed',),
51
418: ('im_a_teapot', 'teapot', 'i_am_a_teapot'),
52
421: ('misdirected_request',),
53
422: ('unprocessable_entity', 'unprocessable'),
54
423: ('locked',),
55
424: ('failed_dependency', 'dependency'),
56
425: ('unordered_collection', 'unordered'),
57
426: ('upgrade_required', 'upgrade'),
58
428: ('precondition_required', 'precondition'),
59
429: ('too_many_requests', 'too_many'),
60
431: ('header_fields_too_large', 'fields_too_large'),
61
444: ('no_response', 'none'),
62
449: ('retry_with', 'retry'),
63
450: ('blocked_by_windows_parental_controls', 'parental_controls'),
64
451: ('unavailable_for_legal_reasons', 'legal_reasons'),
65
499: ('client_closed_request',),
66

67
# Server error status codes
68
500: ('internal_server_error', 'server_error', '/o\\\\', '✗'),
69
501: ('not_implemented',),
70
502: ('bad_gateway',),
71
503: ('service_unavailable', 'unavailable'),
72
504: ('gateway_timeout',),
73
505: ('http_version_not_supported', 'http_version'),
74
506: ('variant_also_negotiates',),
75
507: ('insufficient_storage',),
76
509: ('bandwidth_limit_exceeded', 'bandwidth'),
77
510: ('not_extended',),
78
511: ('network_authentication_required', 'network_auth', 'network_authentication')

For example, if you want to check whether the result is 404, you can compare with requests.codes.not_found.