I installed speakers on my 97 Prelude, and now I can get power to my deck even without turning on ignition?

I have a JVC deck that circuit city installed. I replaced the stock honda speakers with pioneers. There were four wires coming into a two prong connector to the old speakers. Since I had to connect this to two wires i just crimped the two wires of each color together and then to the two leads coming out of the speaker. Should I be afraid that this will drain the battery? How can I fix it?