if this is for kubernetes, you could look into node-exporter. It deploys as daemonset and should automatically start on new nodes. Prometheus scrapes from node-exporter and gives you good visibility into node resources and follows them as they come up and go down
as for monitoring of containers and pods, look into cadvisor, it is embedded in the kubelet
All of this works with standard Prometheus pull. Prometheus guys do not recommend using push at any scale because it is expensive and complex.
Good overview:
Visualisation and alerts can be done with Grafana. There is also Prometheus alert manager.
All of this is very mature and well maintained. Do not reinvent the wheel
--vk