How to fix ProFTP Handshake Trouble in Plesk

I fixed a problem this morning which wouldn’t let the latest version of FileZilla v3.10.1.1 connect to one of my client’s servers anymore.

This had not been a problem in the past.

The connection itself worked, but FileZilla failed due to a problem with the TLS Certificate. Here’s the error:

Status: Initializing TLS...
Error:  Received TLS alert from the server: Handshake failed (40)
Error:  Could not connect to server

Turns out that FileZilla have made a few changes and deprecated the insecure RC4 algorithm in FTP over TLS. Since ProFTP didn’t know the path to the server certificates, TLS failed and hence no connection was possible.

Thankfully there was an easy fix for this, courtesy of this Parallels Knowledge Base article: http://kb.sp.parallels.com/en/2207

To add the default Plesk certificates to the server, all I had to do was tweak the ProFTP config file at /etc/proftpd.conf and add the following at the bottom:

  
    TLSEngine on
    TLSLog /var/log/tls.log
    TLSProtocol SSLv23 

    # Are clients required to use FTP over TLS?
    TLSRequired off

    # Server's certificate
    TLSRSACertificateFile /usr/local/psa/admin/conf/httpsd.pem
    TLSRSACertificateKeyFile /usr/local/psa/admin/conf/httpsd.pem

    # Authenticate clients that want to use FTP over TLS?
    TLSVerifyClient off

    # Allow SSL/TLS renegotiations when the client requests them, but
    # do not force the renegotations.  Some clients do not support
    # SSL/TLS renegotiations; when mod_tls forces a renegotiation, these
    # clients will close the data connection, or there will be a timeout
    # on an idle data connection.
    TLSRenegotiate required off
  

In this example the Server Certificate section contains the default path to Plesk’s certificates, but feel free to substitute them if yours are stored elsewhere.

There’s no need to restart xinetd because ProFTP creates a new process for every new connection, which will then include the new configuration. NOw FileZilla can connect without a hitch, only displaying the new Server Certificate the first time it is encountered:

Note that this issue no longer occurs with newer installations of Plesk. This particular instance of Plesk has seen many updates since version 10.4, hence the tweak was necessary.

• http://kb.sp.parallels.com/en/2207
• https://filezilla-project.org
• http://superuser.com/questions/874981/recieved-tls-alert-from-the-server-handshake-failed-40