No one would say you should but it's not that you couldn't!
Size is a big issue and I understand. I run a digital camera site and the absolutely most frequent complaint about any DSLRs by far is that they too big and end up staying at home. You can work on discipline to take it with you and enjoy top image quality and crucially fast autofocus speed. Plus, as someone suggested, you can get a smaller lens.
Mirrorless systems have gone a long way since their introduction. Some approach the speed of DSLRs in good light and yet are much smaller. The latest generation from Olympus even have comparable image quality: OM-D E-M5, E-PL5 and E-PM2, from largest to smallest. When light levels drop, so does their AF performance. They still do well but do not match top DSLRs. You can somewhat close the gap by using small bright lenses though.
One of those will certainly do if you compare this to leaving your DSLR at home! Nikon also has a series of mirrorless which ultra-fast AF in good light but they seriously drop in AF speed when light is low. Pentax mirrorless are much slower. I have not tried Canon's. Sony has mirrorless systems too but some are as big as a DSLR. The smaller NEX series now has phase-detect on sensor on the latest models but I am not sure you it performs. Panasonic AF is fast but the last ones I tried were slower in low-light than the Olympus ones.
You haven't said about what else you take pictures of. If it were that you did not really need a DSLR for anything else, switching is a great choice. You can always keep more than one camera. Family photos is the reason I kept a small camera around and it predates mirrorless! It is less intimidating and I get more candid photos from the young ones with the small cameras. They do great for small prints and emailing but wouldn't hold up to a large wall print.