Magic damage

Magic damage (or Spell damage) is one of the two types of damage in League of Legends, the other one being Physical Damage. This one is dealed by most Champions' spells, though some of them deal physical damage.

Statistics influencing Magic Damage
The magic damage of a spell can be boosted with Ability Power. A spell's total damage is calculated by the damage gained from AP + it's basic damage. Every ability dealing Magic Damage has an Ability Power - Coefficient, which determines how strong the AP effects the damage dealed. The formula to calculate an ability's bonus damage is AP * AP-Coefficient.

The champion that is taking the Magic Damage can reduce it with Magic Resistance, all Magic Resistance points are added to calculate a damage reduction percentage. The defender's Magic Resistance can be decreased by your Magic Penetration.

Calculation example
Putting everything together, you receive the general formula to calculate Magic Damage of an ability:

MagDmg = (BasicDmg + AP*APCoefficient) * (1 - ((MagRes-MagPen) / (100+MagRes-MagPen)))

Lets say your ability a base damage of 150, your AP is 170, and the AP-Coefficient of the ability is 0.5. You have 20 Magic Penetration and the opponent has 34 Magic Resistance.

Using the general formula, you will hit your opponent for (150+(170*0.5)*(1-((34-20)/(100+34-20)))=206.14 = 206 Magic Damage.