the peculiar case of urllib2 in python2

It’s been a while since I got my hands on python and I came across with an interesting issue of an internal check today, when upgrading an old Proxy Virtual Machine.

Certificate Verify Failed

Suddenly the internal python check was reporting

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)

Python Version

python version before the upgrade was Python 2.7.12 , now is Python 2.7.17

According to the online python documentation:

Changed in version 2.7.9: cafile, capath, cadefault, and context were added.

So something has changed but in both versions the verification for self-signed CA should have the same behavior!

Going down the rabbit hole, seems the most relevant info is in here PEP-0476 but in both cases you have to import SSL and this is not 100% the case !

Nevertheless the default behavior now is to verify the certificate in HTTPS and that is what we need to do!

urllib2 proxy

One of requirements is to check that the proxy is working. URLLIB2 needs a ProxyHandler to do so.

proxy_url = "http://{proxy}/".format(**cfg)
proxy_handler = urllib2.ProxyHandler({ 'http' : proxy_url })

after that, we need to pass the proxy_handler to a build_opener and get the response

opener = urllib2.build_opener(proxy_handler)
response = opener.open(cfg["url"])
return response

This will check an internal URL through our proxy but it will fail when trying to verify the self-signed certificate.

SSL

To enable SSL functionality to python2 we need to import the ssl module

import ssl

after that it is simple a case of creating a SSLContext to open the https url.

To create an SSL Context, there is a useful function for common purposes

ssl.create_default_context(purpose=Purpose.SERVER_AUTH, cafile=None, capath=None, cadata=None)¶

In our case we can pass our internal self-signed CA file as an argument

ctx = ssl.create_default_context(cafile="/etc/certs/ca.pem")

perfect.

Build an HTTPS handler

As we are using an build_opener to pass the proxy handler, but we also need to create an additional handler for the SSL (transport layer) in urllib2.

handler = urllib2.HTTPSHandler(context=ctx)

okay, our handler has the SSL Context with the CA file in place.

Build Opener

it is time to stitch everything together

## Create SSLContext with CA file to pass to handler
ctx = ssl.create_default_context(cafile="/etc/csi.certs/vm2csi/ca.pem")

## Create an HTTPS Handler with our custom SSL Context
handler = urllib2.HTTPSHandler(context=ctx)

## Create a Proxy Handler
proxy_handler = urllib2.ProxyHandler({ 'http' : proxy_url })

## Stitch together the proxy and https handlers to build an opener
opener = urllib2.build_opener(proxy_handler,handler)

## Get the response
response = opener.open(cfg["url"])
return response

and now we have a proper internal check!