With the device, device set, or individual devices selected, click the Delete Selection button  in the upper-right corner of the Device Sets panel

Right-click your selection and choose Delete.

With a device or device set selected, click Delete on your keyboard.