Minecraft Server Error Bad Packet id 131?

I recently made a mod pack for me and my friend and created a server. However every time either of us places a block, we get kicked for the reason:

java.io.IOException: Bad Packet is 131
at ey.a<SourceFile:185>
at co.i<SourceFile:249>
at co.c<SourceFile:17>
at cp.run<SourceFile:94>

After getting kicked, we can join instantly back and the block we placed is still there, this happens with both vanilla and mod blocks. Also, when right clicking to eat food it crashes so it appears to be every time we interact with items in our hot-bar (either placing blocks or eating food).

The mods used are both the same for the client side and the server side as I believe they are all universal (please correct me if i'm wrong). We can both do whatever we want on single player worlds but when we join my server we can't place blocks or eat or else we get kicked so i'm guessing its something to do with the server.

I'm using forge version "forge-1.6.4-9.11.1.965" and these are the mods I'm using:

Rei's Minimap v3.4
FurnitureMod v3.2.7
Damage Indicatorsv2.9.2.3
appeng-rv14-finale3
Auto Utilities v1.0.1
BiblioCraft v1.5.5
Binnie Mods v1.8.0
Buildcraft v4.2.2
ChickCunks (1.6.4)
CodeChickenCore (1.6.4)
CoFH Core v2.0.0.2
Computer Craft v1.58
DartCraft Beta v0.2.15
denLib v3.1.35
DenPipes v2.1.18
Enchanting Plus v1.16.5
Ender Storage Mod (1.6.4)
Extra Utilities v.1.0.2
Factorization v0.8.29
Gravity Gun v2.0.0
Torched v2.0.0
Forestry v2.3.1.1
Galacticraft v2.0.8.908
Galacticraft-Planets v2.0.8.908
Hats v2.1.1
HatStand v2.1.0
IndustrialCraft 2 v2.0.395-experimental
InfiniBows v1.2.0.14
Ironchest v5.4.1.702
LogisticsPipes v0.7.4.dev.106
MicdoodleCore v2.0.8.908
MineFactoryReloaded v2.7.5-352
MiscPeripherals v3.4b2
ModularPowersuits v0.10.0-91
NeiAddons v1.10.2.59
NEIPlugins v1.1.0.6
NetherOres v2.2.1-14
NotEnoughItems v1.61.5
Numina v0.1.2-55 (1.6.2)
OpenModsLib v0.1
OpenPeripheralCore v0.3.0
PluginsforForestry v3.2.30
PortalGun v2.0.2
PowerCrystalsCore v1.1.8-9
PowerSuitAddons v877
qCraft v1.1
RailCraft v8.3.2.0
Slick-Util
SoulShards v2.0.15
Switches v1.3.0.25
Sync v2.1.1
TConstruct v1.5.2.6
ThermalExpansion v3.0.0.2
Twilight Forest v1.20.3
iChunUtil v2.4.0
Vending v1.1.1
WirelessRedstoneCore v1.7.1.4

Any help is appreciated.

Remove the mods.