The operation must depend only on the cosets and not on the representatives; that happens exactly when \(gN=Ng\), i.e. when \(N\) is normal. The other conditions are neither necessary nor sufficient.
Given a normal subgroup \(N\) of a group \(G\), the quotient group \(G/N\) is the set of cosets of \(N\) with the operation induced by multiplying representatives.