Python urllib.error.httperror: http error 403: forbidden

The urllib.error.httperror: http error 403: forbidden occurs when you try to scrap a webpage using urllib.request module and the mod_security blocks the request. There are several reasons why you get this error. Let’s take a look at each of the use cases in detail.

How to fix urllib.error.httperror: http error 403: forbidden?

Usually, the websites are protected with App Gateway, WAF rules, etc., which monitor whether the requests are from the actual users or triggered through the automated bot system. The mod_security or the WAF rule will block these requests treating them as spider/bot requests. These security features are the most standard ones to prevent DDOS attacks on the server.

Now coming back to the error when you make a request to any site using urllib.request basically, you will not set any user-agents and headers and by default the urllib sets something like python urllib/3.3.0, which is easily detected by the mod_security.

The mod_security is usually configured in such a way that if any requests happen without a valid user-agent header(browser user-agent), the mod_security will block the request and return the urllib.error.httperror: http error 403: forbidden

Example of 403 forbidden error

from urllib.request import Request, urlopen

req = Request('http://www.cmegroup.com/')
webpage = urlopen(req).read()

Output

  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
urllib.error.HTTPError: HTTP Error 403: Forbidden
PS C:\Projects\Tryouts> from urllib.request import Request, urlopen

The easy way to resolve the error is by passing a valid user-agent as a header parameter, as shown below. 

from urllib.request import Request, urlopen

req = Request('https://www.yahoo.com', headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()

Alternatively, you can even set a timeout if you are not getting the response from the website. Python will raise a socket exception if the website doesn’t respond within the mentioned timeout period.

from urllib.request import Request, urlopen

req = Request('http://www.cmegroup.com/', headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req,timeout=10).read()

In some cases, like getting a real-time bitcoin or stock market value, you will send requests every second, and the servers can block if there are too many requests coming from the same IP address and throws 403 security error.

If you get this error because of too many requests, consider adding delay between each request to resolve the error.

Leave a Reply

Your email address will not be published.

Sign Up for Our Newsletters

Get notified of the best deals on our WordPress themes.

You May Also Like
Python String Isspace()

Python String isspace()

Table of Contents Hide What are whitespace characters in Python?isspace() Syntaxisspace() Parametersisspace() Return valueExample 1: Working with isspace() method Example 2: Program to check if the string has only whitespace…
View Post
Python String Center()

Python String center()

Table of Contents Hide center() Syntaxcenter() Parameter center() Return ValueExample 1: center() Method With Default fillcharExample 2: center() Method With * fillchar Example 3: Returns original string if the width is…
View Post
Convert String To Float In Python

Convert String to Float in Python

Table of Contents Hide Convert string to float in Pythonfloat() functionExample 1 – Convert string to floatExample 2 – Default float conversion without any input parameterExample 3 – Convert a…
View Post
Python Check If String Is Empty Or Not

Check if string is empty or not in Python

Table of Contents Hide Python program to check if string is empty or notUsing len() funtionUsing not operatorUsing not+str.strip() method Using not + str.isspace method In this article, you will learn…
View Post
Python String Islower()

Python String islower()

Table of Contents Hide islower() Syntaxislower() Parameterislower() Return ValueExample 1: Demonstrating the working of islower() methodExample 2: Practical use case of islower() in a program Python String islower() method is a…
View Post
Nxnxn Matrix Python

nxnxn matrix python

Table of Contents Hide What is NxNxN?How to Create NxNxN Matrix in Python?Create NxN Matrix in Python 3 with Non Duplicating numbersCreate NxNxN matrix in Python using numpy In this…
View Post