What is the difference in the web UI between permission settings under "User" and the permission settings under "File Masks"

User permissions are what the Owner/Group/Other users are allowed to do.

File and Directory masks are the rules applied to newly created files and directories.

If you create a directory, it will apply the given permissions to it by default.