hMailServer: auto-ban can be bypassed with 4-byte characters in usernames

The auto-ban of Progressive Robot hMailServer 6.2.0 through 6.3.3, with a MySQL or MariaDB database, can be kept from banning an address after repeated failed logons, which allows a remote unauthenticated attacker to go on guessing account passwords from a single address without being banned. The server makes its MySQL tables in MySQL's utf8 character set, which holds no character outside the Basic Multilingual Plane, sends its statements as UTF-8 over a utf8mb4 connection and leaves the database's default strict mode in place. From 6.2.0 it decodes the user name of a SASL PLAIN logon as UTF-8, so the name can hold such a character. When the failed logons counted from one address reach MaxInvalidLogonAttempts (3 by default), the server clears that address's count, saves an IP range named "Auto-ban: " followed by the user name of the logon that reached the limit, and drops the connection. If such a character is in the part of the user name that the range's name keeps (the name is cut to 100 characters), the database refuses the range: no range is saved, the address is not refused, and, its count having already been cleared, it is given the next full set of attempts. An attacker can in this way spend all but the last attempt of each set on password guesses, and make the last under such a name. On Windows, POP3 AUTH PLAIN reaches it on a server as installed, and IMAP AUTHENTICATE PLAIN where IMAP SASL authentication is turned on (off by default); on Windows and on Linux (from 6.3.0), SMTP AUTH PLAIN reaches it where plain-text authentication is allowed (off by default), and ManageSieve and CardDAV where they are turned on (off by default). Each failed save is reported in the server's error log. A database whose strict mode has been turned off stores the character as a question mark and makes the first such ban, but refuses every later ban named after the same user name, or after one holding other such characters in their place, as a duplicate while the first stands, since the server cannot look up a name holding such a character there.