Yeah, I think there is a lot of potential for code analysis. There's a limited cross section of ways malware can do interesting things, but many permutations of ways to do that.

So look for the interesting things, like:

• accessing other programs' address spaces
• reading/writing files
• deleting/moving files
• sending/receiving network traffic
• os system calls and console commands
• interacting with hardware
• spawning new processes
• displaying things on the screen
• accessing timing information

Obviously there's legitimate uses for each of these, so that's just the first step.

Next, analyze the data that is being used for that:

• what's the source?
• what's the destination?
• what kind of transformations are being applied to the data?

Then you can watch out for things like:

• is it systematically going through directories and doing some operation to all files? (Maybe ransomware, data scrubbing, or just maliciously deleting stuff?)
• is it grabbing data from somewhere and sending it somewhere else on the internet? (Stealing data?)
• is it using timing information to build data? (Timing attacks to figure out kernel data that should be hidden?)
• is it changing OS settings/setup?

Then generate a report of everything it is doing and see if it aligns with what the code is supposed to do. Or you could even build some kind of permissions system around that with more sophistication than the basic "can this app access files? How about the internet?"

Computer programs can be complex, but are ultimately made up of a series of simple operations and it's possible to build an interpreter that can do those operations and then follow everything through to see exactly what is included in the massive amount of data it sends over the network so that you can tell your file sharing program is also for some reason sending /etc/passwords to a random address or listening for something to access a sequence of closed ports and then will do x, y, z, if that ever happens. Back doors could be obvious with the right analysis tools, especially if it's being built from source code (though I believe it's still possible with binaries, just maybe a bit harder).