It certainly can make a difference, for some of the reasons that you've alluded to. If we break hosting's influence on ranking down to its 3 core factors, we have:
Security
Location (which ties into:)
Speed
Security is pretty basic - the more at risk your site is to being hacked, the bigger the risk of that hack happening and a drop in rankings occurring (which can then be very hard to regain). If you're managing the server yourself, make sure you take all of the necessary steps. If you're using managed solutions, vet the provider as much as possible.
Location - this is a two-fold factor. There is some correlation (albeit it is not a big one) that if you had two equal websites, one in the UK and one in the US, and you're trying to rank in Google UK, the site hosted in the UK might rank a bit better. It's not huge, but worth keeping in mind. What is more of a factor in server location is the actual location of the server itself and where your typical users (or target users) are based in the world. The closer your users are to your server's datacentre, the faster the server response is likely to be. Faster websites are happy websites. That leads us to the main point on:
Speed. As a (very general) rule, the more allocated RAM and bandwidth, plus the greater server's processing capabilties, the faster it is likely to be. Typically, this means that dedicated servers (where it's you and only you taking 100% of the server's resources) will perform better than a dedicated VPS (100% of the resource, just less resource), shared VPS, or shared hosting. The bigger your site becomes and the bigger bandwidth footprint it creates, as it were, the more you'll need a server that can handle that. There is a good correlation between site speed and organic rankings. If the server is slowing you down, it could be holding you down as well.
But servers are just the start - there are a number of server and site configurations that can have a big impact on site speed - stuff like a Content Delivery Network (CDN), Gzip compression, image size. Swing over to GTMetrix and add in a URL and you'll get a speed analysis, plus tips on how you can improve your speed score and overall site speed.
Hope this helps.
