Photobox Downloader Updated

I have updated my Photobox Downloader application with a bunch of new features and fixes. Photobox recently implemented some heavy throttling for photos download, this update addresses that and more.

 

Change log for 0.3.2:

  • New retry logic will retry failed (timeout) downloads automatically
  • Can now skip already downloaded files (helps greatly with interrupted downloads)
  • New debug mode (pass “-d”) gives extensive logging
  • Less concurrent photo downloads, to avoid throttling
  • Improved the documentation
  • Updated example
  • Albums with slashes are handled better

Update by running:

  $ npm update -g photobox-downloader

The source code for API usage is available on GitHub project repository.


API Gateway testing permissions tip

Just a tip, in order for API Gateway test / sandbox area to be able to execute (invoke) a Lambda function that was generated by CloudFormation, you need to explicitly grant the Sandbox permission in your CloudFormation file. As it is not documented and there is currently no way to “export” a manually created API as CloudFormation file, it is easy to overlook/miss. The simple solution for this is to add a new Lambda permission, with the “stage name” set to “null”.

Here is a complete example of a Lambda Permission Resource in CloudFormation:

        "ApiGatewaySandboxPermission" : {
            "Type" : "AWS::Lambda::Permission",
            "Properties" : {
                "FunctionName" : { "Ref" : "MyFunctionAlias" },
                "Action" : "lambda:InvokeFunction",
                "Principal" : "apigateway.amazonaws.com",
                "SourceArn" : { "Fn::Join": [ "", [
                    "arn:aws:execute-api:",
                    { "Ref" : "AWS::Region" }, ":",
                    { "Ref" : "AWS::MyAccountId" }, ":",
                    { "Ref" : "MyRestApiId"}, "/",
                    "null",
                    "/GET/*"
                ] ] }
            }
        },
 The interesting part here are the last few lines. This grants the SandBox (Stage Name is “null”) to invoke all GET based methods, starting at the root (/*) of your API, tweak path as needed.
                    "null",
                    "/GET/*"
 Hope this helps.

Nginx Proxy Pass, resolving “No required SSL certificate was sent”

If you are using Nginx as a reverse proxy and trying to inject client certificates you may run into a Server 400 “No required SSL certificate was sent” error. I spent a few hours debugging this issue and thought I’d share my findings. The problem is fairly subtle and easy to over look/miss, numerous other people have ran into it. The problem is that the backend server is using SNI (Server Name Indication, basically allows multiple SSL/TLS certs on a single IP). You must explicitly tell Nginx to pass forward the domain name in the TLS handshake, so that the final destination (your backend) knows which SSL/TLS cert to serve.

Following the Nginx proxy documentation, you would set the required directives and expect it to work, so your configuration might look something like:

location / {
        proxy_pass                 https://api-backend.somesite.com/;
        proxy_ssl_certificate      /etc/nginx/conf.d/ssl/client.crt;
        proxy_ssl_certificate_key  /etc/nginx/conf.d/ssl/client.key;
}


The solution is to just add and extra directive to enable SNI , the directive is called “proxy_ssl_server_name“. A working example would be:
location / {
        proxy_pass                 https://api-backend.somesite.com/;
        proxy_ssl_server_name      on;
        proxy_ssl_certificate      /etc/nginx/conf.d/ssl/client.crt;
        proxy_ssl_certificate_key  /etc/nginx/conf.d/ssl/client.key;
}


 Restart Nginx and test with your browser again, all should be working!

Another year, another SSL cert, another bullet dodged!

Just renewed my SSL certificate thanks to StartSSL (using their very nice and fairly straight forward automated system), it got me thinking on all the issues that I have encountered and heard about in regards to SSL certificate and domain renewal.

In a previous job there was an incident where a SSL certificate expired, this went unnoticed (it was a analytics service, not a core service) for 5 weeks, not too bad right? Well, unfortunately due to an additional oversight in the client code, if a request failed, it would retry the request after a time out of 0.5 seconds, indefinitely!. This lead to hundreds of thousands of clients hammering the (auto-scaling) load balancer with requests that were dropped as SSL verification failed. The end result, a large bill that could have been avoided. This issue is hardly unique, even the largest cloud providers have been hit by similar issues (see Windows Azure Service Disruption from Expired Certificate or Google.com domain getting transferred or Microsoft losing its Hotmail domain).

It is hard to have a universal list of best practices as it is dependent on the size of the organisation, but some good ideas:

  • Avoid single point of failure, never have an SSL cert (or domain name) associated with an individual developers email address. Instead use a specific mailing list.
  • Monitoring, even secondary services (that are considered ‘best effort’), need active monitoring.
    • Monitoring results should be sent to mailing list not a specific individual
    • When setting up monitoring, don’t forget about monitoring costs as well as uptime!
  • Consolidate all domain and SSL certs with a single trusted provider
  • Purchase with a Company credit card with auto renew

What best practices would you recommend? What bullets have you dodged (or got hit by!)? Any services out there that you use to avoid issues?

 


Setup OpenVPN using OpenWRT

Configuring OpenVPN to work on OpenWRT is relatively easy and straight forward, just not very well documented. This is an in depth, step-by-step guide to configure OpenVPN (VPN provider Private Internet Access – commonly called PIA) on an OpenWRT router.

Assumptions

  • You already have OpenWRT installed on your router
  • You know how to connect to your router via SSH and Web panel
  • Router is connecting to another device (Modem, other router, direct to ISP) that is supplying internet access

Lets being…

This tutorial will cover:

  • Installing and configuring OpenVPN
  • Configuring a network interface
  • Setting up some firewall rules & DNS Leak protection
  • Verify everything works

 

  1. First step, open a SSH connection to your router, login as root. You should see something like Figure 1 below.
    Fig 1

    Figure 1 – SSH Login

     

  2. Next we have to update packages and installed some required libraries, enter the following commands in the terminal:
  3. Next, download the OpenVPN config files from PIA – https://www.privateinternetaccess.com/openvpn/openvpn.zip somewhere to your local machine. Extract all the files from the zip file. You are only interested in 2 files (ca.crt and crl.pem, we will get back to them later). You can safely delete the *.ovpn files.
  4. Now open your broswer and go to your router web panel, by default this should be: http://192.168.1.1
  5. Once logged in you should notice a new menu item called Services, goto it and click the OpenVPN option, see Figure 2 below.
    Figure 2 - OpenVPN menu

    Figure 2 – OpenVPN menu

     

  6. Time to add our new configuration. At the bottom, in the text field, enter a new name “pia_client”, select “Simple client configuration for a routed point-to-point VPN” and click Add button (Figure 3)
    Figure 3 - Create config

    Figure 3 – Create config

     

  7. You will immediately be taken to the config page, click the link “Switch to advanced configuration”
    Figure 4 - Advanced Menu

    Figure 4 – Advanced Menu

     

  8. All settings on the Service page should be fine. Click the “Networking” link at the top.
  9. See Figure 5 below for how the settings should look like. A few notes:
    • If there is a line missing, use the “Additional Field” drop down at the bottom, select the missing field and press Add button
    • Ensure that “dev” is set to “tun” and not “tap”
    • If there is a field called “ifconfig” with an IP address, remove the address (i.e. make field blank)
    Figure 5 - Networking Setup

    Figure 5 – Networking configuration

     

  10. Click blue Save button on the bottom
  11. Now click on the “VPN” link to change to the VPN tab. As in Networking, there will be some fields missing, use the “Additional Field” drop down at the bottom again to add them. A few notes:
    • “auth_user_pass” field value should be “/etc/openvpn/userpass.txt” (It doesn’t exist yet, but we will get back to it in a few minutes)
    • The “remote” field should be the hostname of which ever exit node you want to use – see PIA Networking page for a complete list.
      Figure 6 - VPN configuration

      Figure 6 – VPN configuration

       

  12. Now click on the “Cryptography”. As  before, use the “Additional Field” drop down at the bottom to add missing fields.
    • IMPORTANT: for the “ca” field, you will need to browse to the location of the ca.crt file from the openvpn.zip you downloaded in step 3.
    • The “crl_verify” path should be set to “/etc/openvpn/crl.pem”
      Figure 7 - Cryptography configuration

      Figure 7 – Cryptography configuration

       

  13. We have the VPN configuration done now, but we still need to configure the interface as well as the Firewall.
  14. From the Menu at the top select Networking -> Interfaces.
  15. Click the “Add new interface…” button.
    • Name: “PIA_VPN”(IMPORTANT: Name must be exactly this)
    • Protocol of the new interface: Unmanaged
    • Cover the following interface: Custom Interface: tun0
      Figure 8 - Create Interface

      Figure 8 – Create Interface

       

  16. Enter in the details and click the Save button.
  17. For the final few steps, we will switch back to SSH.
  18. Next we have to create a file that will store your PIA username and password. It is just a simple text file, with first line username and second line your password. Then we will chmod it to set correct permissions.
    Create username and password file

    Figure 9 – Create username and password file

  19. Now have to add the crl.pem file (from the openvpn.zip), just open it in a text editor like notepad and copy the contents
    Figure 9 - Create CRL file

    Figure 10 – Create CRL file

     

  20. Now we need to setup some firewall rules to forward the VPN traffic
  21. Almost done!
  22. In order to protect against DNS Leaks, we need to update the DHCP server to supply the PIA DNS servers instead of your ISP’s DNS.
  23. From the main menu, goto: Network -> Interfaces -> LAN -> DHCP Server (below the “Common Configuration” section) -> Advanced Settingss. In the “DHCP-Options” field enter the value: “6,209.222.18.222,209.222.18.218”.
  24. Click “Save & Supply”
    Figure 10 - Interfaces - > LAN

    Figure 11 – Interfaces – > LAN

    Figure 11 - DNS settings

    Figure 12 – DNS settings

     

  25. All done! Now we can start the VPN connection.
  26. Goto: Services -> OpenVPN, check the Enabled checkbox beside our”pia_client”, then press the Start button, your VPN should now start up.
    Fig 12 - VPN Started

    Fig 13 – VPN Started

     

Verify it works…

  • To verify your traffic is going over VPN you can use the PIA What is My IP tool

    Figure 13 - Successs! VPN working

    Figure 14 – Successs! VPN working

  • If it isn’t working then you may have missed a step. Try going to Status -> System Log in the main menu, it may contain useful information.
  • To verify your DNS is not leaking use something like DNS Leak site (you may have to release & renew your DHCP IP before this will work)

Congratulations, your VPN tunnel is now setup!


Fix ‘node-gyp rebuild’ error on windows

While playing around with Flux & React I ran into some issues using a yoeman flux generator. It kept failing on “node-gyp rebuild”. If you do any development on Windows you’ve likely run into issues with node-gyp before. The core of the problem is that node-gyp is no longer being actively developed  and so it has some old dependencies that a modern development env might not have.

node-gyp rebuild failed

node-gyp rebuild failed

How to fix?

  1. Goto Control Panel -> Programs and Features and uninstall “Microsoft Visual C++2010 x64 Redistributable” and “Microsoft Visual C++ 2010 x86 Redistributable” (if present)cpluplus_redist
  2. Download and install Python 2.7.3 (if you have Python 3.x already installed, just leave it, both can coexist)
  3. Visual C++ 2010 Express or Visual Studio 2010
  4. Windows SDK 7.1
  5. Visual Studio 2010 SP1
  6. Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1

IMPORTANT: The order of steps above is important!

Now open a command window/console and enter the following commands

npm config set python /Python27/python.exe --global
npm config set msvs_version 2010 --global

Final Step

Finally goto Start -> All Programs -> Microsoft Windows SDK v7.1 -> Windows SDK 7.1 Command Prompt

sdk-cmd

From this command window the `node-gyp rebuild` command will work.

NOTE: You only need to use the Windows SDK 7.1 Command Prompt when running `npm install`, once installed can go back to normal command window.


Download all your Photobox Photos

I got a few emails from people asking how to use the original Photobox Downloader. The initial idea of the app was to provide a clean API to allow people to interact with their Photobox account, however it seems that people just want a quick and easy to download their photos! With the new V2 version, you can much easier download your photos, no editing of files needed!

Installation & Usage

$ npm install -g photobox-downloader
$ mkdir albums
$ pb-dl

The new version will now prompt you for the 3 required items:

  1. The domain that your photos are on (www.photobox.ie, www.photobox.co.uk, etc…)
  2. The directory where to store the files (this directory must exist already!)
  3. The Authentication cookie value (see below for more detailed instructions)

photobox-downloader-v2

Once you enter this info it will immediately start downloading all your photos.

How to get authentication cookie value?

When you log into your account on Photobox, Photobox sets an authentication cookie, if you know how to view cookies, look for the pbx_www_photobox_xx (xx depends on where you are logging into) cookie, otherwise you can just log into your Photobox account, open the Developer Toolbar (press F12 on your keyboard), goto the “Console” tab and paste the following command into the input area and press enter, it will output your authentication cookie value.

document.cookie.split(';').forEach(function(item){if(item.match('pbx_www_photobox')!==null){console.log('Auth cookie:',item.split('=')[1])}});

photobox-downloader-v2-console

Further instructions on the GitHub Photobox Download project page.


Generate Random Serial Numbers

On a project recently I needed to generate a large number of random serial keys, I couldn’t find any a simple/easy way to handle this, so here is my implementation. You specify the length you want and it will automatically insert a separator every 4 characters.

The second parameter, separator, is optional. It defaults to ‘-‘.


Photobox photo downloader/backup tool

Photobox Downloader V2 is now available, even easier to download all your photos. Blog post below refers to the original (V1) version.

During the week I found out that my wife had a bunch (over 2.4K!) of photos on PhotoBox.ie that were not part of our family archive. This weekend I dug around PhotoBox website and could no way to download an entire album, it was only possible to download photos individually! With 2.4K+ of photos, that is a lot of clicking!

As Larry Wall said the, a good developer, is a lazy developer! Thus I created a NodeJS module (photobox-downloader) that makes it easy to access an download individual photos or an entire album (or everything!) easily. With the module I was able to backup all of my wife’s photos in about 5 minutes, no clicking needed!

Photobox Downloader will auto scan your album list and then can automatically downloaded every photo. Optionally you can download only specific files or albums.

Built with Request for making the HTTP requests, Cheerio for parsing of the HTML, Async for managing the asynchronous control flow, as well as Progress for showing a fancy progress bar.

I have open sourced the code and it is available on GitHub, I also published it as a NodeJS module, so you can find it in the NPM registry.

Installation

$ npm install photobox-downloader

Usage

Screenshot

photobox-downloader

 

If you run into any issues running the module just give me a shout!