Hey thanks, i fixed it, however I don't know why my coords are different than what you mentioned they should be. My upper left is not 0,0 its instead 0,768 (height of window). I just made mouseY = Math.abs(mouseY - 768) and it works now.
is it possible one of these methods messed up my coords, i don't actually know what they do i think i just copied them from somewhere.
GL11.glMatrixMode(GL11.GL_PROJECTION);
GL11.glLoadIdentity();
GLU.gluOrtho2D(0, 1024, 0, 768);
GL11.glMatrixMode(GL11.GL_MODELVIEW);
GL11.glLoadIdentity();
GL11.glViewport(0, 0, 1024, 768);