Most "varmint rifles" are heavy assed target guns trhat are a bear to lug around.
I'd choose a Savage 116 fsack (fluted bull barrel) "weather warrior" with tasco 3-9 40mm scope in .243 for "walking around" I use this rig for antelope/deer/elk in 30-06. I can shoot out to about 350 accurately without a bipod with this rifle. Use a GI leather sniper sling and practice breath control. You may want a higher power scope for the longer range stuff.. but at that range you should have a spotter too.
Another ideal rifle is that new CZ 527 in 243 or 223.. lighter profile barrel and blue steel but it sure is pretty. It's clip fed so you could carry a variety of ammo loads.. ie thgis is a wood chuck bullet vs this is a coyote killer.
An inexpensive option is an H&R varminter single shot in 223 or 243 Great bang for the buck. Top in off with a tasco 4-16X50 and you can really get out there, and even with a bipod attached this is a cherry rig for walking and shooting all day.
Just be forewarned those light wieght 22/6mm bullets are more prone to wind drift than a .308 round is.
Rob