Yes, unfortunately, physics dictates that faster/brighter/larger aperture lenses will tend to be bigger. And the Panasonic 20mm f1.7 is really the best compromise between small size and large aperture. (Full disclosure: I own a 20mm f1.7.)
>"The RX100 is another consideration but from most of the reviews which I have read it seems that in low light conditions the LX7 is the one to beat."
Not according to Gordon's review (linked to previously):
The LX7 holds its own up to around 400 ISO but beyond that quickly falls behind the ... 20.2 Megapixel RX100. Looking at the 800 and 1600 ISO crops the LX7 displays a higher level of noise and less image detail.
Plus, the RX100 is even a little smaller than the LX7.
Having said that, my personal experience with my Olympus E-PL5 + Pany 20mm f1.7 lens vs my Sony RX100 (zoomed out and wide open) indicate that the Oly/Pany combo offers better image quality in a package that's not a whole lot bigger than the Sony (and slightly smaller than the GX1 would be). And if you don't need the E-PL5's tiltable LCD (which would seem to be the case since neither the LX7's or GX1's LCD can tilt), you can get the same sensor for less money in the slightly smaller yet E-PM2.
In short, I think you would be best served by either an RX100 or E-PM2 + 20mm f1.7 - Mark