Chests, like barrels, contain items, and in most cases can be broken by hitting them. They have the appearance of crates, however.
Some items chests may contain include (but are not limited to) Meat, Bread, Grapes (best used for planting to make more grapes) Fruit juice, First aid kits, Iron, and cookies.
It is usually best to open crates as soon as you find one. As there is usually a large amount of valuable loot inside.
They are sometimes locked, that's when the picklock comes in handy.