> > > > > > the cmos is said to operate at 8 fps … this would make my argument > > > > > with the ai wrong unless (e.g. amortized with many parallel things or > > > > > something maybe), because normally one matrix op relies on output of > > > > > last. Maybe that is what it was trying to get at. In the microscopy > > > > > papers they imply cmos is faster than this — but in the fpm papers I > > > > > was excited about recently they usually do low resolution. > > > > > > > > > > It sounds like it may be possible but if so I haven’t sorted out how > > > > > quite yet, to do fast tokens. Maybe by engaging the internals of the > > > > > cmos somehow? > > > > > > > > > > I sure didn’t connect that! I was expecting faster cmos > > > > > > > > yeah the Sony IMX500 used in the raspberry pi camera can do 12MP/60fps > > > > or 1080p at 240fps [1] > > > > 1: https://developer.sony.com/imx500/imx500-key-specifications > > > > > > Oh bit the numbers from perplexity were for a 3,000mp chip 0.o sounds > > > expensive > > It was 410 mp/ 8 fps. 3,000mp/s . > The page perplexity cited says it also does 100MP @ 24 fps and > implying 4x underlying pixel depth from binning. (Note though that 24 > < 8 x 4, although it may get 32 fps with binning disabled, I dunno. > Either way 75% is not much if one is trying to get to milliseconds or > microseconds or whichever it was > > > > > Oh! But these chips generally offer higher frame rates at lower > > resolutions, so they _already_ can exchange between space and time. It > > should be fine to reorder matrices to be more serial.
So you’d need a very high frame rate. Not sure how many tensors are in 405b, but you’d need a frame for almost every one. So if there are 800, then 800fps would net 1 token/sec . One could consider using many cheap high-frame-rate chips possibly with low resolution, and clocking them in an offset manner. The “photonic multiplication” would be limited by the speed of the lcd (although these could also be offset). A slow lcd might divide the 16k integration factor, slowing the whole system. There are cmos with 700,000fps [2]. The company selling them also sells photonic chips [3]. With the IMX500 running at 240fps, one might consider figuring a way to have 16 of them and getting 3k fps, this might give 4 tokens/sec or so. The high cmos frame rates are accomplished by windowing part of the visual field. This windowing is likely possibly with the canon chip and the imx500 too, at least with the underlying chip. This might let them accomplish the theoretical speed (not totally sure if i have that right but it seems quite possible). Then emitter speed vs 16k integration speed up would come into play. 2: https://www.bi-pol.com/high-speed-cmos-image-sensors/ 3: https://www.bi-pol.com/photonic-integrated-circuits/
