Between version 5.3.0.13 and the latest Lansweeper version, 6.0.0.65, there have been quite a lot of changes to the database. For the helpdesk module and knowledgebase feature many "htbl" tables were added. Some tables were also added to accommodate new scanned data and some slight changes were made in existing tables. Unfortunately though we don't keep a detailed schema change log, so we can't specify exactly which changes were made to the database.
You could install the latest Lansweeper version on a test machine and pull all the tables, and pull all the tables from your 5.3 installation and then run them through a diffchecker for instance. That should provide you with the specifics.
Regarding custom reports, no changes will happen to these upon upgrading, these views will remain the same.
Regarding the changes to functionalities between the two version, you can consult our change log
here. The greatest amount of changes will be between the last 5.x version and the first publicly released 6.x version (6.0.0.19).