How to Access Localhost from the Internet

by Michael Cropper | Apr 9, 2016 | Developer | 0 comments

When developing basic websites, never need to be able to access your local machine form the internet. Whatever language or platform you are using, you’ll likely have your local development environment at something like http://localhost:8080 or similar. This allows you to test the code you are writing with ease without keep uploading it to your web server which is a more time consuming process.

When you start getting involved with building more advanced web applications, opening up your localhost environment to the internet is such a valuable tool, I can’t stress this enough. Particularly when working with technologies like webhooks, payment gateways and other integrated technologies. This is required because these websites need to send data back to your application outside of the initial request, and they cannot send this data to your local machine without being able to access it. You see, your local machine is protected by multiple firewalls, within your router as a hardware firewall and also with the firewall software running on your machine. So even if the outside website did know your IP address to send the data to, they wouldn’t be able to get through your firewalls anyway.

This is why you need to be able to access localhost from the internet which is only possible with specialist software designed to do exactly this. There are several different tools available to allow you to do this, personally I find the ease of use of Ngrok making it the only choice for me. To understand how Ngrok works, a couple of images shown on their website helps explain this in more detail as this is quite a technical topic to get your head around if you haven’t used before;

How to Access Localhost form the Internet with Ngrok

Image source

How to Send Data from Webhooks to Localhost with Ngrok

Image source

Ngrok have a list of platforms that it works on and this can be set up in as little as 30 seconds, it is seriously awesome. So anyone wanting to know how to access localhost from the internet or send webhooks data to your local machine, then take a look at Ngrok. They have free versions of the software and paid subscriptions which are dirt cheap.

Debugging Email Deliverability Issues between cPanel, WHM, IMAP and Exchange

by Michael Cropper | Apr 4, 2016 | Developer | 4 comments

If you have ever worked with websites, web servers, cPanel, WHM and Exchange, you will know that there is never a situation that is ever the same. With default configurations for installations, changing settings and automatic updates, it’s easy for systems to sometimes get tangled in a few knots. Thankfully, there are tools available to debug things to identify specifically what is going on, below is a process which will hopefully save you hours of time investigating issues like this to quickly identify what is happening when your emails are going astray.

Sending & Receiving Emails

Ok, so if we look at the underlying technologies about sending & receiving emails first, this helps to understand where to first look for problems when they do occur. Let’s break this down into the two main categories of IMAP and Exchange as they both behave significantly differently. For Exchange, whether you are running a private Exchange server or whether you are on the Microsoft Hosted Exchange platform, this doesn’t make a huge deal of difference, so this information is going to be generic to the two with the understanding that there are always the odd discrepancies depending on configurations etc.

I’m going to assume that you have already set your MX Records correctly at the DNS. Make sure you have an SPF Record / TXT record with SPF details in if you are sending from IMAP or you will get blacklisted by Microsoft Exchange and will need to un-blacklist yourself.

Microsoft Exchange – Sending & Receiving Emails

Whenever you send emails from name@website.com from Microsoft Exchange, these emails are automatically handled by Exchange itself. Meaning that Microsoft Exchange respects the rules of the internet in relation to the DNS MX Records. So when you send emails to hello@another-website.com, Microsoft Exchange looks up the MX records for another-website.com and simply sends the message on its merry way to be picked up by the mail exchanger. Simple.

Whenever you receive emails into name@website.com, likewise, Microsoft simply looks within the Exchange settings for the website.com domain and decides which account this belongs in, if this is an invalid account or if this needs to be forwarded on to a distribution group or similar. Fairly straight forward.

Overall, Microsoft Exchange does just work and rarely is the problem at Microsoft’s end.

IMAP, cPanel, WHM and Web Servers – Sending & Receiving Emails

In comparison to Microsoft Exchange above which is essentially a single piece of software, sending and receiving emails with IMAP is a whole different ball game. It is one of the reasons why we always recommend companies use Microsoft Exchange, not only because it is a much more powerful system, but because it also costs you more for us to support you with IMAP related issues as they take 10x longer to get to the bottom of. Just use Exchange where possible.

Never the less, many businesses use a mixture of technologies for a variety of reasons, so it’s important to be able to understand how things fit together and most importantly how things can be quickly debugged and tweaked. So let’s look at how these different technologies fit together.

WHM is an administrator piece of software for managing web servers, much like an administrator account on your personal computer which allows you to install software, configure settings, create additional users etc. The difference being is that it allows you ultimate control of functionality through an easy to use interface and most importantly debugging tools for email delivery from and to your web server.

cPanel is designed for individual users and accounts. It is the equivalent of having an administrator account on your home computer (i.e. WHM) and then an account each for Mum, Dad, Son, Daughter (i.e. cPanel accounts). It is designed to allow each user / account to manage their own items on the web server in a safe and secure way without being able to interfere with other people’s data.

IMAP is a technology which transcends both WHM, cPanel and often desktop applications which makes things a little more complex when debugging issues. Domains are assigned to individual cPanel accounts, such as website.com and my-other-website.com. And individual emails for name@website.com are configured within the individual cPanel accounts, even though the IMAP technology means that it is the actual server sending and receiving emails and deciding how to deal with email routing.

Websites also send emails from scripts and code which has been developed which introduces another layer of complexity as from an underlying technology point of view, the settings which are configured mean that emails from websites may or may not be delivered, even if they appear to be sent when you send a message on the contact form on the website for example. Always test your contact forms on a regular basis.

Whenever you receive email to your IMAP email address for name@website.com, ‘the internet’ routes this email from the sender, via the DNS MX record settings, to your web server where your emails live. The software running on your web server then ultimately decides which account this should end up in. Receiving emails is generally ok as long as the accounts are set up.

Whenever you send email from your IMAP email address for name@website.com, this is when things get a little different. This is because there are many technologies interacting together, particularly when you are using a desktop email application to send emails or even a website to send emails. There are other things to consider particularly when web servers, websites and email applications may have been configured a specific way which may in fact mean that emails are actually sent through SMTP opposed to IMAP.

Debugging Email Deliverability Problems

Ok, so now we’ve had a look at the many moving parts behind the scenes when it comes to emails, let’s look at how you go about debugging email deliverability problems and where to start. I’m going to stick with IMAP here as Exchange is an enormous topic on its own and takes quite a bit of learning. Once Exchange is configured, it just works. Get in touch if you are having issues related to Microsoft Exchange configuration and I’m sure we can help walk you through what needs to be done.

DNS Configuration

This has been mentioned before, but it’s worth reiterating. Make sure your DNS is configured correctly and is pointing to the right place it should be. This relates specifically to your MX records, and TXT, A, SPF, or SRV records which may need to be created depending on your email technology in the background. Speak with your web hosting company for the actual settings here.

Review What is Actually Happening

Next, review what is actually happening to try and determine where the problem may lie. Review;

• Can you send emails via webmail?
• Can you receive emails via webmail?
• Can you send emails via your desktop application?
• Can you receive emails via your desktop application?
• Is your website sending emails?
• Are you receiving emails sent by your website?
• Can you send emails to specific domains?
• Can you receive emails from specific domains?
• Can you send emails to specific domains when sending from another specific domain?
• Can you receive emails to a specific domain when sending from another specific domain?
• Can you successfully authenticate with the server when trying to send and receive emails?
• Have you correctly configured your email settings and port settings?
• Have you waited 24 hours for DNS propagation if you have just changed your DNS settings?
• Are there any service interruptions with your web server?

The more you can narrow down exactly what is happening here, the greater the clues are to what is actually going on. Yes, your emails don’t work, but why, what is the specific issue we are facing. Ultimately, when emails are sent, they should be received at their intended destination. When things don’t quite go to plan here, it means that something in the process is doing something different which is generally down to some settings somewhere which need to be tweaked.

Tracking Received Email Deliverability

Receiving emails into IMAP is much easier to debug than sending emails, so it’s always best to start here. If your DNS records are set correctly, then you should be receiving emails by default. As long as you have an email account set up within cPanel, your emails should be arriving successfully. Always check your junk folders and make sure that you web server isn’t automatically junking / deleting emails that it believes are junk. I have seen this happen in the past with over active junk email settings. Likewise, keep an eye out for any email forwarders that may have been set up and are taking priority when the email arrives and sending it off somewhere else before it lands in your inbox.

Generally speaking though, if you aren’t receiving emails it is due to your DNS records being incorrect or the email address you are sending to blocking emails from your server as they look like spam. Make sure you are sending these test emails from an email account that you know already works. It helps to have a few email accounts such as Hotmail / Outlook / Gmail etc. so that you can test this from various sources. If you aren’t receiving emails from your other email accounts that you know are working correctly, then re-check your DNS settings.

Tracking Sent Email Deliverability

Now onto tracking your sent email deliverability issues. This depends on where you are sending emails from in the first place. If you are sending emails from your website and they aren’t being received, this again can be for many reasons. It could be related to how your web server is configured for when it is sending emails to @website.com, it could be trying to deliver your emails locally to an email account hosted on the web server and in which case if you are running your emails remotely through Microsoft Exchange, then you may need to update these settings. It could also be the way in which you are sending emails as many web hosting companies actively turn off specific functionality on web servers which allow code to send emails as a security measure. Speak with your web hosting company about this, it is the only way to know for sure.

If you are having issues sending emails via webmail or via a desktop application, first try to verify that this isn’t an issue which relates to your desktop application communicating with your webserver. Try and aim for a single point of failure to test if this point is failing or not and work your way through each one by one, I’m talking about testing specifically, you should never have a single point of failure for anything in business – if you do this is a whole other discussion to be had. Assuming you have tested this, next step is to look within WHM to see where your emails are actually going when they are being sent from one of the accounts on IMAP. There is debugging tool within WHM under Email > Mail Delivery Reports which allows you to see the email you just sent, where did it end up. Often this relates to an issue with local / remote / automatic mail delivery. Check what is happening here, if the recipient isn’t who you intended, then you need to configure your settings within cPanel accordingly so that cPanel isn’t overriding any settings when it shouldn’t be doing.

Outliers and Things to Remember

The above is a rough guide and doesn’t cover everything you could ever check for. For example, one project we recently worked on had a unique issue which doesn’t crop up very often whereby sent emails were getting lost in the ether due to configuration settings within cPanel which means that only a tiny amount of emails were getting lost, everything else was absolutely fine. The situation was that within cPanel, multiple domains were configured including website1.com and website2.com. One website was running Microsoft Exchange, the other IMAP. Exchange was working perfect. IMAP was working perfect….almost. The issue cropped up when website2.com on IMAP wanted to send emails to website1.com on Exchange. Because both domains were configured under the same cPanel account, MX Entry settings within cPanel meant that emails sent from website2.com on IMAP were attempted to being delivered locally due the website1.com domain having an automatic configuration in place which decided where emails should be delivered. Changing this to a Remote location resolved this issue, meaning that cPanel no longer overwrote the default behaviour of the DNS and things continued to function normally.

There are always things to bear in mind and check.

Summary

When using older technology such as IMAP instead of the far superior Microsoft Exchange, debugging email deliverability issues can be tough if you don’t know your way around your web server, the software that is running and particularly on cheap web hosting packages where you may not even have access to many of the tools outlined above. It is so important to make sure you have good web hosting in place for your business to allow you to debug problems like this quickly and continue working away. The last thing you want is weeks of delays trying to get to the bottom of these types of issues as you could be missing vital orders and customer queries.

Should you need any support managing your email technologies, get in touch and we can happily support you in this area. The reality is that all web servers are configured differently, have many technologies running and various settings based on your individual requirements. Cheaper web hosting is more of a challenge to debug as you simply often don’t have access to the tools you need and can only deal with the issues by speaking with the web hosting company yourself or letting someone like ourselves do this on your behalf so they don’t try and bamboozle you with technical jargon with the aim of fobbing you off. It’s always good to work with companies who understand these technologies inside and out. Get in touch for more information.

How to Prevent Emails from Web Applications and Websites Landing in the Junk Folder

by Michael Cropper | Feb 13, 2016 | Developer | 1 comment

To set the scene. Firstly, you should never be hosting emails on your web server, it is extremely bad practice these days for a variety of reasons, read the following for why; Really Simple Guide to Business Email Addresses, Really Simple Guide to Web Servers, Really Simply Guide to Web Server Security, The Importance of Decoupling Your Digital Services and most importantly, just use Microsoft Exchange for your email system.

Ok, so now we’ve got that out of the way, let’s look at some of the other practicalities of web application development. Most, if not all, web applications require you to send emails in some form to a user. Whether this is a subscriber, a member, an administrator or someone else. Which is why it is important to make sure these emails are being received in the best possible way by those people and not getting missed in the junk folder, or worse, automatically deleted which some email systems actually do.

Sending emails from your web application is not always the most straight forward task and hugely depends on your web server configuration, the underlying technology and more. So we aren’t going to cover how to do this within this blog post, instead we are going to assume that you have managed to implement this and are now stuck wondering why your emails are ending up in your and your customers junk folders. The answer to this is often simple, it’s because your email looks like spam, even though it isn’t.

Introducing Sender Policy Framework (SPF Records)

The Sender Policy Framework is an open standard which specifies a technical method to prevent sender address forgery. To put this into perspective, it is unbelievable simple to send a spoof email from your.name@your-website.com. Sending spoof emails was something I was playing around with my friends when still in high school at aged 14, sending spoof emails to each other for a bit of fun and a joke.

Because it is so simple to send spoof emails, most email providers (Microsoft Exchange, Hotmail, Outlook, Gmail, Yahoo etc.) will often automatically classify emails as spam if they aren’t identified as a legitimate email. A legitimate email in this sense is an email that has been sent from a web server which is handling the Mail Exchanger technologies.

To step back a little. The Mail Exchanger, is a record which is set at the DNS level which is referred to as the MX record. What this record does is that whenever someone sends an email to your.name@your-website.com, the underlying technologies of the internet route this email to the server which is handling your emails, the server which is identified within your MX Record at your DNS. The server then does what it needs to do when receiving this email so you can view it.

For example, when hosting your email on your web server with IMAP (as explained before, you shouldn’t be doing this), your MX record will likely be set to something along the lines of ‘mail.your-domain.com’. When using Microsoft Exchange, your MX record will be something along the lines of ‘your-domain-com.mail.protection.outlook.com’.

What this means is that if an email is sent from @your-domain.com which hasn’t come from one of the IP addresses attached to your-domain-com.mail.protection.outlook.com, then this looks to have been sent by someone who is not authorised to send emails from your domain name, and hence why this email will then end up in the spam box.

With Microsoft Exchange specifically to use this as an example, you will also have an additional SPF record attached to your DNS as a TXT type which will look something along the lines of, ‘v=spf1 include:spf.protection.outlook.com -all’, which translates as, the IP addresses associated with the domain spf.protection.outlook.com are allowed to send emails from the @your-domain.com email address, and deny everything else.

This is perfect for standard use, but doesn’t work so well when trying to send emails from a web application. Which is why we need to configure the DNS records to make the web server which is sending emails from your web application a valid sender too.

How to Configure your DNS with SPF Records

To do this, we simply add the IP address of your web server into the DNS TXT SPF record as follows;

v=spf1 include:spf.protection.outlook.com ip4:123.456.123.456 –all’

Your records will likely be different, so please don’t just copy and paste the above into your DNS. Make sure you adapt this to your individual needs. That is it. Now when your web application sends emails to your users, they will arrive in their main inbox instead of their junk folder. Simple.

A few handy tools for diagnosing DNS propagation along with SPF testing include;

• http://www.dnswatch.info/
• https://www.whatsmydns.net/
• http://www.kitterman.com/spf/validate.html

How to Configure Let’s Encrypt on CentOS6 with cPanel and WHM

by Michael Cropper | Feb 7, 2016 | Developer | 7 comments

Quick reference guide for how to implement. Let’s Encrypt is a new free certificate authority, allowing anyone and everyone to encrypt communications between users and the web server with ease. For many businesses, cost is always a concern, so saving several hundred pounds for a basic SSL Certificate often means that most websites aren’t encrypted. This no longer needs to be the case and it would be recommended to implement SSL certificates on every website. Yes…we’re working on getting around to it on ours 🙂

We recently implemented Let’s Encrypt on a new project, Tendo Jobs and I was quite surprised how relatively straight forward this was to do. It wasn’t a completely painless experience, but it was reasonably straight forward. For someone who manages a good number of websites, the cost savings annually by implementing Let’s Encrypt on all websites that we manage and are involved with is enormous. Looking forward to getting this implemented on more websites.

Disclaimer as always, make sure you know what you’re doing before jumping in and just following these guidelines below. Every web server setup and configuration is completely different. So what is outlined below may or may not work for you, but hopefully either way this will give you a guide to be able to adjust accordingly for your own web server.

How to Set up Lets Encrypt

So, let’s get straight into this.

• Reference: https://community.letsencrypt.org/t/quick-start-guide/1631
• Run command, yum install epel-release, to install the EPEL Package, http://fedoraproject.org/wiki/EPEL. Extra Packages for Enterprise Linux, lots of extra goodies, some of which are required.
• Run command, sudo yum install git-all, to install GIT, https://git-scm.com/book/en/v2/Getting-Started-Installing-Git
• Clone the GIT repository for Let’s Encrypt with the command, git clone https://github.com/letsencrypt/letsencrypt, http://letsencrypt.readthedocs.org/en/latest/using.html#id22
• For cPanel servers, need to run a separate script, hence the next few steps
• Install Mercurial with the command, yum install mercurial, http://webplay.pro/linux/how-to-install-mercurial-on-centos.html. This is Mercurial, https://www.mercurial-scm.org/
• Run the install script command, hg clone https://bitbucket.org/webstandardcss/lets-encrypt-for-cpanel-centos-6.x /usr/local/sbin/letsencrypt && ln -s /usr/local/sbin/letsencrypt/letsencrypt-cpanel* /usr/local/sbin/ && /usr/local/sbin/letsencrypt/letsencrypt-cpanel-install.sh, https://bitbucket.org/webstandardcss/lets-encrypt-for-cpanel-centos-6.x
• Run the command to verify the details have been installed correctly, ls -ald /usr/local/sbin/letsencrypt* /root/{installssl.pl,letsencrypt} /etc/letsencrypt/live/bundle.txt /usr/local/sbin/userdomains && head -n12 /etc/letsencrypt/live/bundle.txt /root/installssl.pl /usr/local/sbin/userdomains && echo “You can check these files and directory listings to ensure that Let’s Encrypt is successfully installed.”
• Generate an SSL certificate with the commands;
  1. cd /root/letsencrypt
  2. ./letsencrypt-auto –text –agree-tos –michael.cropper@contradodigital.com certonly –renew-by-default –webroot –webroot-path /home/{YOUR ACCOUNT HERE}/public_html/ -d tendojobs.com -d www.tendojobs.com
  3. Note: Make sure you change the domains in the above, your email address and the {YOUR ACCOUNT HERE} would be replaced with /yourusername/ without the brackets.
  4. Reference: https://forums.cpanel.net/threads/how-to-installing-ssl-from-lets-encrypt.513621/
• Run the script with the commands;
  1. cd /root/
  2. chmod +x installssl.pl
  3. ./installssl.pl tendojobs.com
  4. Again, change your domain name above
• Set up a CRON Job within cPanel as follows, which runs every 2 months;
  1. 0 0 */60 * * /root/.local/share/letsencrypt/bin/letsencrypt –text certonly –renew-by-default –webroot –webroot-path /home/{YOUR ACCOUNT HERE}/public_html/ -d tendojobs.com -d www.tendojobs.com; /root/installssl.pl tendojobs.com
• For reference, The SSL certificate is placed in /etc/letsencrypt/live/bundle.txt when installing Let’s Encrypt.
• Done!

Note on adding CRON job to cPanel, this is within cPanel WHM, not a cPanel user account. cPanel user accounts don’t have root privileges so a CRON job from within here won’t work. To edit the CRON job at the root level, first SSH into your server, then run the following command to edit the main CRON job file;

crontab -e

Add the CRON job details to this file at the bottom. Save the file. Then restart the CRON deamon with the following command;

service crond restart

It is recommended to have a 2 month renewal time at first as this gives you 4 weeks to sort this out before your certificate expires. Thankfully you should receive an email from your CRON service if this happens and you will also receive an email from Let’s Encrypt when the certificate is about to expire so there are double safe guards in place to do this.

On-Going Automatic Renewal & Manually SSL Certificate Installation

Important to note that when you automatically renew your Let’s Encrypt certificates, they won’t be automatically installed. The installssl.pl script doesn’t seem to handle the installation of the certificate. Instead, you may need to update the renewed certificates within the user cPanel account for the domain manually. To do this, open cPanel and view the SSL/TSL settings page, update the currently installed (and about to expire) SSL certificate and enter in the new details. The details for the new certificate will need to be obtained via logging into the ROOT server via SSH and viewing the updated SSL certificate details in the folder, /etc/letsencrypt/live/yourdomain.com where you can use the command pico cert.pem and pico privkey.pem to view the details you need to copy over to cPanel. It’s decoding the SSL certificates in these two files to make sure the dates have been updated, you can use a tool such as an SSL Certificate Decoder to decode the certificate. If the certificate is still showing the old details, then you may need to run the command letsencrypt-auto renew which will update the certificates.

Hope this is useful for your setup. Any questions, leave a comment.