Whenever you pick something up, it goes into your inventory.

To view your inventory press I.

Here you can use items, drop them or delete them.

To delete an item drag the item to the trash at the bottom of the inventory window.
If you have alot of stuff in your inventory, you will see arrows on either side of the trash can at the bottom of inventory window.

Click on them to turn the page.

You can drag and drop items between pages by dragging the item onto the arrow of the page you wish to put the item on.

For example I've moved a defense scribe from inventory page 1 to page 2.