ComputersConsumer ElectronicsCell PhonesHome Theater & AudioGraphic Design & Video EditingInternetIndustrial Technology

Intercepting HTTPS Traffic Using the Squid Proxy Service in pfSense

Updated on May 23, 2016
skear profile image

Sam works as a Network Analyst for an algorithmic trading firm. He obtained his Bachelors Degree in Information Technology from UMKC.

Source

A Brief Introduction to Squid

Squid has become one of the most popular packages for pfSense firewalls and it's not hard to see why.

Squid proxy servers can improve network performance by keeping a local cache of commonly accessed web pages, images and other files. Additionally Squid can monitor traffic and keep a log of which web pages users on your network are viewing.

In order to fully take advantage of the benefits of Squid I recommend taking the time to properly enable HTTPS interception.

This guide assumes you already have a pfSense firewall up and running. If you don't have one yet you can easily install pfSense on an old computer you probably already have on hand.

Why You Should Enable HTTPS Interception in Squid

By default Squid proxy servers cannot monitor encrypted HTTPS traffic. Squid simply establishes a TCP connection to the destination server and responds to the client with an HTTP 200 response to indicate the connection was established.

Once this encrypted tunnel has been established Squid passes the packets between the client and the server but has no longer has any visibility to the traffic since it is protected by SSL encryption.

Over the last few years many popular web sites including Google, Youtube, Reddit and Facebook have started enabling HTTPS encryption by default. This means that without configuring HTTPS interception Squid proxies have limited filtering, monitoring and logging capabilities.

Fortunately Squid supports man in the middle SSL filtering which will allow you to more effectively monitor the traffic passing through the proxy server.

Encrypted HTTPS traffic can be inspected using the SSL interception feature in Squid.
Encrypted HTTPS traffic can be inspected using the SSL interception feature in Squid. | Source

Step 1: Install the Squid3 Package

To get started install the Squid3 package using the pfSense package manager (System \ Packages).

After locating Squid3 from the package list click the plus button on the right side of the package to start the package installation.

The package manager will automatically download and install the Squid3 PBI.

Installing the Squid3 package from the pfSense package manager.
Installing the Squid3 package from the pfSense package manager.
Squid3 package installation progress.
Squid3 package installation progress.

Step 2: Configure the Squid General Settings

After the Squid package has been installed the general settings must be configured. The settings page can be found in Services \ Squid Proxy Server.

Configure the following options:

  1. Enable Squid Proxy - Checked
  2. Keep Settings/Data - Checked
  3. Proxy Interface(s) - Select LAN and Loopback

The rest of the settings in the general settings area can be left on the default settings.

Squid3 General Settings Page in pfSense
Squid3 General Settings Page in pfSense

Step 3: Configure Transparent Proxy Settings

Just below general settings you'll find the transparent proxy settings. In most cases you will probably want to enable transparent proxy mode. When this mode is enabled the firewall will automatically redirect all incoming web traffic to the Squid proxy server.

With transparent mode enabled it is not necessary to configure the clients web browser to use the proxy. In most cases the client will not even notice their traffic is passing through the proxy.

To enable the transparent proxy configure the following settings:

  1. Transparent HTTP Proxy - Checked
  2. Transparent Proxy Interface(s) - LAN

The rest of the settings in this section can be left on their default settings unless you wish to configure specific addresses to bypass the proxy.

Scroll down to the bottom of the page and click save to apply the settings.

If you decide not to enable transparent mode you will need to configure the web browser settings for each client you wish to specifically use the proxy server.

Squid3 transparent proxy settings in pfSense.
Squid3 transparent proxy settings in pfSense.

At this point you have configured a basic Squid proxy running in transparent mode. Before proceeding further I recommend testing web browsing functionality from a client computer to make sure everything is working normally.

Troubleshoot and resolve any problems with the basic proxy functionality before proceeding to enable SSL interception.

Step 4: Configure a Certificate Authority

A certificate authority must be configured in pfSense before HTTPS interception can be enabled in Squid. The CA will be used to generate new client SSL certificates on the fly for automatically encrypting and decrypting web traffic.

  1. Access the cert manager in the System \ Cert Manager menu.
  2. Click the plus button to create a new certificate authority.

Configure the following settings for the new certificate authority.

  1. Descriptive Name - Choose a name for your CA. Keep in mind that this will be displayed on the client certificate visible to the clients.
  2. Method - Select 'Create an Internal Certificate Authority' from the dropdown menu.
  3. Key length - I recommend 2048 for maximum compatibility but you can use 4096 bits for maximum security.
  4. Digest Algorithm - Use SHA256 or higher. Like the previous setting you need to balance security with device compatibility.
  5. Lifetime - Set this to 3650 days (10 years)
  6. Distinguished name - Fill out all of the fields in the section (Country, State, etc). These will all be visible in the certificates seen by the clients.

Click the save button to finish creating the CA.

Creating a new certificate authority in pfSense.
Creating a new certificate authority in pfSense.

Step 5: Export the CA Certificate

After creating the new CA the CA certificate must be exported. This certificate will need to be installed on any client machine that will be using the proxy server.

From the CA manager page click the export CA cert button to download the certificate. This will download a copy of the CA certificate in .crt file format.

Exporting the CA certificate from the pfSense certificate authority manager.
Exporting the CA certificate from the pfSense certificate authority manager.

Step 6: Install the CA Cert to the Client Computers

In order to prevent the web browsers on client computers from showing certificate errors the CA certificate from the pfSense CA must be installed on all client computers that will be using the proxy server.

Skipping this step will result in clients receiving browser security errors and can cause various HTTPS connection problems.

The certificate must be placed in the Trusted Root Certificate Authorities store to prevent browser errors.

If you only have a small number of computers on your network then it will probably be easiest to manually import the certificate on each computer.

For larger networks you should consider setting up Microsoft Active Directory Certificate Services. AD integrated certificate authorities can automatically push out a root certificate to hosts which are members of the domain.

Importing the Certificate on Windows 7

To import the certificate to a computer running Windows 7 double click on the .crt file to open the certificate dialog box.

  1. Click the install certificate button to start the import wizard.
  2. Click next on the first page of the certificate import wizard.
  3. Select the option place all certificates in the following store.
  4. Click on the browse button and select Trusted Root Certification Authorities.
  5. Click next, then click finish on the import confirmation page.
  6. When prompted click yes to acknowledge the security warning.

You should see a message indicating the certificate import was successfully completed.

Click thumbnail to view full-size
Windows 7 certificate properties dialogWindows 7 certificate installation wizardImporting a certificate into the Trusted CA store on Windows 7Successfully importing a certificate in Windows 7
Windows 7 certificate properties dialog
Windows 7 certificate properties dialog
Windows 7 certificate installation wizard
Windows 7 certificate installation wizard
Importing a certificate into the Trusted CA store on Windows 7
Importing a certificate into the Trusted CA store on Windows 7
Successfully importing a certificate in Windows 7
Successfully importing a certificate in Windows 7

Importing the Certificate on Mac OS X

Follow the steps below to manually import the certificate on Mac OS X.

  1. Open the Keychain Access application - Use the spotlight search to easily find this app.
  2. Click the lock symbol to unlock the key chain for changes.
  3. Open the File menu and select Import Items.
  4. Select the CA certificate exported from pfSense. (At this point you should see the certificate in the keychain with the message "This root certificate is not trusted")
  5. Double click the certificate and expand the trust section of the dialog box. In the first dropdown box called "When using this certificate" select always trust.
  6. Close the dialog boxes and exit the keychain access application.

Click thumbnail to view full-size
Imported a CA certificate in Mac OS XMarking a certificate as "Always Trusted" in Mac OS X
Imported a CA certificate in Mac OS X
Imported a CA certificate in Mac OS X
Marking a certificate as "Always Trusted" in Mac OS X
Marking a certificate as "Always Trusted" in Mac OS X

Step 7: Enable SSL Man in the Middle Filtering

After loading the certificate to the client computers you are ready to enable SSL filtering in Squid. Access the squid settings page (Services \ Squid Proxy Server) and configure the settings below.

  1. HTTPS/SSL Interception - Checked
  2. SSL Intercept Interface(s) - Select LAN
  3. CA - Select the certificate authority created in step 4

Click save at the bottom of the page to apply the settings.

Squid3 SSL man in the middle settings in pfSense
Squid3 SSL man in the middle settings in pfSense

Step 8: Testing SSL Interception

The best practice after enabling SSL interception is to confirm that it is working as intended. Follow these steps to verify HTTPS connections are being decrypted by the proxy.

  1. Visit a site which uses HTTPS such as Reddit from a client computer behind the proxy.
  2. View the certificate information presented by the web browser. In Chrome this can be done by clicking the lock symbol on the address bar.
  3. Confirm the certificate issuer information matches the information you entered when creating the certificate authority in step 4.


Known Issues in Squid 3.4

The pfSense package manager currently contains Squid version 3.4 which has a known issue where it incorrectly generates SHA1 certificates instead of SHA256.

Since SHA1 is a weak algorithm many browsers will show errors when they encounter certificates signed using this algorithm.

To fix this I recommend manually upgrading to Squid version 3.5.3 which does not have this problem.

Upgrading to Squid 3.5.3

The upgrade instructions are slightly different depending on whether you are running the 32-bit or 64-bit version of pfSense.

To determine which version you have open the pfSense dashboard and check the version section of the system information dashboard widget. If you see AMD64 then follow the 64-bit instructions. If you see i386, then use the 32-bit instructions.

The commands can be run through an SSH terminal, or the web based terminal (Diagnostics \ Command Prompt)

64-Bit (AMD64) Instructions

  1. Download the PBI by running the command: fetch https://files.pfsense.org/packages/10/All/squid-3.5.3-amd64.pbi
  2. Install the package by running: pbi_add --no-checksig -f squid-3.5.3-amd64.pbi
  3. Run the commands below to create the correct directory structure
cd /usr/pbi/squid-amd64/
rm -rf /usr/pbi/squid-amd64/etc
ln -s /usr/pbi/squid-amd64/local/etc .
ln -s /usr/pbi/squid-amd64/local/lib .
ln -s /usr/pbi/squid-amd64/local/libexec .
ln -s /usr/pbi/squid-amd64/local/share .
ln -s /usr/pbi/squid-amd64/bin sbin


Reboot pfSense after running the above commands (Diagnostics \ Reboot).

32-Bit (i386) Instructions

  1. Download the PBI by running the command: fetch https://files.pfsense.org/packages/10/All/squid-3.5.3-i386.pbi
  2. Install the package by running: pbi_add --no-checksig -f squid-3.5.3-i386.pbi
  3. Run the commands below to create the correct directory structure
cd /usr/pbi/squid-i386/
rm -rf /usr/pbi/squid-i386/etc
ln -s /usr/pbi/squid-i386/local/etc .
ln -s /usr/pbi/squid-i386/local/lib .
ln -s /usr/pbi/squid-i386/local/libexec .
ln -s /usr/pbi/squid-i386/local/share .
ln -s /usr/pbi/squid-i386/bin sbin


Reboot pfSense after running the above commands (Diagnostics \ Reboot).

Verifying the Installation of Squid 3.5.3

After rebooting pfSense start a new SSH session (or use the web terminal) to verify the updated package was correctly installed.

When you run the command below you should see version 3.5.3 listed in the output.

/usr/local/sbin/squid -v

Verify Squid 3.5.3 has been correctly installed.
Verify Squid 3.5.3 has been correctly installed.

Completion

Congratulations, if you completed all of the steps above you have successfully configured Squid to intercept encrypted HTTPS traffic. You should immediately notice HTTPS requests being logged in the Squid access logs.

To take full advantage of the new access logs being collected I recommend installing a Squid log analyzer such as Lightsquid.

© 2016 Sam Kear

Comments

    0 of 8192 characters used
    Post Comment

    • profile image

      sachin19 2 days ago

      Hello Sir,

      how can i block facebook/youtube pages ( Dynamic content filtering) in my entire Network ( AS Internet service provider).

    • profile image

      Franco 5 weeks ago

      I have had similar issues that is stated below, and while playing around a bit, i found the solution that worked for me.

      With bank sites you visit and get errors: go to squid proxy server - SSL Man in the middle. Set the SSL/MITM Mode to "splice all". Then go to remote cert checks and high-lite "Accept remote server certificate with errors". This worked and the banks and other ssl sites can be visited without any errors.

      To clarify, i had issues with the blacklist option. I re-installed squidgaurd, and did not do the blacklist option again. I manually went to target categories and did my blacklisting there.

      I know some might have better solutions, so, if you don't like what i have done that works for me, please do not take this as the best and final solution.

    • profile image

      Wild Wings 5 weeks ago

      For David,

      The reason this exists is because users are now bypassing security settings and policies by using https sites for common social media or other blocked/ policy prohibited sites. Most companies and government agencies have network acceptable use policies that clearly indicate when using their assets you are subject to monitoring and their acceptable use policies. If it is not work related then don't use company assets plain and simple. Go home use your personal computer if you don't like it. The short of it is by bypassing company security measures that filter and block you from visiting sites that could put the company in a position where they would be subject to litigation you can and should be fired. This is is one more step to ensure that said filtering can still be accomplished and protect the company.

    • profile image

      Aubrey-mw 4 months ago

      Would you please write the same with reference to pfsense 2.3.4-RELEASE (amd64)

    • profile image

      Kirby 4 months ago

      Hi, I am using the latest version of pfsense squid and squidguard. I enabled the SSL Man In the Middle Filtering in Squid proxy server to block https sites. However, when enabled I can't connect to certain messenger apps (ie. slack, bbm, skype). Any ideas why connection to these apps is always retrying?

    • profile image

      Sergs 5 months ago

      Hi. Is there any way that I can cache https without installing certificates on each clients on my network?? i have my latest version of pfsense 2.3.4-RELEASE (amd64)

      built on Wed May 03 15:13:29 CDT 2017

      FreeBSD 10.3-RELEASE-p19 installed on my pfsense box.

      Thank you

    • profile image

      Sergs 5 months ago

      Hi. Is there any way that I can cache https without installing certificates on each clients on my network?? i have my latest version of pfsense 2.3.4-RELEASE (amd64)

      built on Wed May 03 15:13:29 CDT 2017

      FreeBSD 10.3-RELEASE-p19 installed on my pfsense box.

      Thank you

    • profile image

      Deepak 6 months ago

      Hello Sam,

      This guide worked for me in pf sense 2.3.2 thank you.

      I want to access from home network to office network using vpn please suggest.

    • profile image

      Sheffy 6 months ago

      Can I use Let's encrypt' CA to intercept HTTPS traffic?

    • profile image

      Claudio 7 months ago

      Hi, I want to grant access only to some site, this is what I've done:

      I put in the white list a list of site to be allowed, for example

      google.com

      utorrent.com

      and in the black list I put . to block all others.

      Normal HTTP site are allowed or blocked correctly as they in the WL or not, while HTTPS sites are all permitted, why?

      How can I block HTTPS sites that are not in the WL?

      I also tried to put ^. in the BL but nothing changed.

      Thanks,

      Claudio.

    • profile image

      Waldo Pulanco 7 months ago

      Hi! It works like a charm, but the problem is very slow to load a page when the protocol is https. how do I overcome this problem?

      thanks in advance!!

    • profile image

      Patricio 8 months ago

      Hi, I implemented step by step and it works perfect, except when I want to enter a web of banks, they reject the certificate and do not let me log in. Could you help me ?

    • profile image

      Pfsense Authentication with OpenLDAP 8 months ago

      Sir

      I am new with PFSense, i have installed and configured pfsense and it is working fine.

      i have setup openLDAP server to create users and want to authenticate those users too with PFsense, so that i can provide internet to the user on the basis of userid authentication.

      But i am unable to achieve my goal please guide me how to do the same.

      my PFsense firewall ip is 192.168.1.1

      my OpenLDAP server ip is 192.168.1.4

      Thanks in advance

    • profile image

      David 9 months ago

      ** Installing the certificate on the client machines isn't necessary for deception to take place but the client browsers will display certificate warnings on any https websites. Because of these warnings it's really only feasible if you deploy the certificates to the client systems.

      Set up a forged cert server and spoof the trust settings in client machines to eavesdrop on private traffic. That is the very _essence_ of deception. I find it very unlikely that someone doing this would inform users ahead of time that their encrypted https communications have been exposed and are being monitored.

      ** Fortunately Squid supports man in the middle SSL filtering

      More commonly referred to as a "man-in-the-middle attack," and with good reason.

      Certificate Authorities exist for a purpose. Once that purpose has been subverted, trust is lost and everyone loses. If it's OK for you to eavesdrop on your users, it's OK for your upstream provider to eavesdrop on you, right?

      Offhand, I cannot think of any honest or ethical application for this. You are purposefully crippling https, and then modifying client trust mechanisms in such a way that it will not be immediately apparent to the end user. If you want to block access to sites like reddit and youtube, simply do so. Otherwise, you are violating your users' reasonable expectations of privacy. Depending on the venue, you are also violating law.

      One more reason that everyone should be using a good, reliable VPN service all the time.

      ** Sam works as a Network Analyst for an algorithmic trading firm

      Wow. I'm glad I don't work there.

      ** I’ve visited many of the major financial datacenters in North America to install new equipment and solve problems

      Oh, shit. Welp. We're screwed.

    • profile image

      umair975 9 months ago

      On PFSense 2.3.2... enabled SSL Interception with transparent proxy and Internal CA.. getting certificate error..

      My aim is to block social networking sites https via SquidGuard.

      any suggestions?

    • profile image

      Boyom Rodrigue 10 months ago

      Hello,

      I followed your guide carefully and I succeeded successfully.

      Nevertheless I have a problem. When I activate the https filtering in squid, the skype application of the stations on my network no longer connects to skype. Have you encountered this problem? If so, how did you resolve it?

    • skear profile image
      Author

      Sam Kear 10 months ago from Kansas City

      @Carlos

      HTTPS decryption will still work without the certificate installed on the end devices but they will see a certificate error on encrypted pages they visit. Best practice is to install the certificate to the devices.

      This can be a challenge if you're dealing with a public or guest network. In order to install it on the cell phones you could implement a mobile device management system such as XenMobile or possibly push the certificate using a captive portal.

      Another possibility might be to direct users to a link to download and install the certificate through a splash page.

    • profile image

      Carlos 10 months ago

      Is there any way to intercept https with transparent proxy with out to use a CA?. How I do to install a CA in each cell phone conected to my wifi lan?

    • profile image

      ERNEST 11 months ago

      hi sam

      what version of PF you mount this...i has been problems with your config... hhtp is ok but https doesn open and CA is ok but still not open when i enable SSL _tick_

      i have another question how filter by groups af acl

      the black list into squid is working but for all into the LAN

      if i want to by pass that restriction i puti on in SQUID

      im tried to use squidguard but doesnt work

      could explain me how to do that

      eaxample .- if i deny lego.com into squid = ok but for all....how by pass when a ip add (pc or user ) wants to pass trow this site

      ok i think into squid doesnt possible because apply for all lan ( value given into squid )

      2.- if i dont put it nothing into blacklist into squid

      i have the idea squidguard get into work , right ?? in that supouse case, i generate ACL commun or ACL particular ...obviosly i generate Targets

      but this is not working at all...

      could you explainme how to do that

    • DelfinDelfin profile image

      DelfinDelfin 11 months ago

      Hi Sam

      How can I install the certificate on Ubuntu?. I tried this: http://superuser.com/questions/437330/how-do-you-a...

      But it didn't work. Thanks in advance. Great post by the way

    • profile image

      konti 12 months ago

      use mail.google.com :)

    • profile image

      Nel 12 months ago

      gmail.com can't get through......

    • skear profile image
      Author

      Sam Kear 12 months ago from Kansas City

      Mohan,

      Installing the certificate on the client machines isn't necessary for deception to take place but the client browsers will display certificate warnings on any https web sites. Because of these warnings it's really only feasible if you deploy the certificates to the client systems.

    • profile image

      Mohan 12 months ago

      Hi,

      is there any possibility to without import CA certificate at client machine.

      Thanks

      mohanrao83@gmail.com

    • profile image

      Agi 13 months ago

      Hi !

      i have your solution fully working, but i'm not able to integrate with squidguard url filtering.

      When i enable squidguard the http site are working, but all httpS sites are not displayed and the browser and the url bar report a thing like : https://http.* and ovviously no site displayed.

      Any suggestion ?

    • profile image

      moivg79 14 months ago

      I have pfsense 2.3 and squid 3.5.19 and I have completed all of the steps above but ssl webs show certificate error. Can you help me?

      Thank you.

    • profile image

      Alonso 15 months ago

      On PfSense 2.3.1 Rp5 the last Squid versión available is 3.5.19. Even you suggest to install squid-3.5.3-amd64.pbi, but I can't install it because in PfSense 2.3.1-RELEASE the command pbi_add has been deprecated. Now seems that pkg command must to be used but "pkg add squid-3.5.3-amd64.pbi" is not enought. Do you know some trick to have installed the fixed Squid versión on PfSense 2.3.1 R p5?

    • profile image

      Mike 15 months ago

      Thanks a lot. This guide worked great!