Hello FreeCalypso community, A couple of days ago I finally performed a long-delayed experiment: I tried increasing the bypass capacitor values on the V-FLASH and V-SRAM power nets to see if doing so would make the sleep mode bug go away while keeping the high-capacity flash+pSRAM chip S71PL129NC0HFW4B. The result: upping these capacitor values did not change the behaviour in any way, and the following scenario still consistently produces an aberrant self-reboot:
* Put FC Magnetite l1reconst firmware in the flash (must be flashed, not fc-xram, see below); * Enable small sleep (AT%SLEEP=1) or all sleep (AT%SLEEP=4); * Try to bring up the SIM with AT+CFUN=1 - bam, the board reboots. With the power nets ruled out, I went thinking what else could possibly be causing this sleep-triggered misbehaviour to occur with our high- capacity Spansion MCP (copied from Pirelli DP-L10) but not with the smaller Samsung MCP used by Openmoko, and I got a new working hypothesis. The Calypso chip has a poorly documented output called FDP. TI's CAL000 document gives it a one-line description of "Flash deep low-power", but there is no further explanation anywhere that I could find. TI's Leonardo reference schematics connect this Calypso output to the flash chip's reset input, and this same connection is present in Openmoko's PCB. We then copied it for the FCDEV3B. I have no proof yet, but my current working hypothesis is that Calypso's FDP output goes low (corresponding to flash reset asserted) not only in deep sleep, but also in small and big sleep modes which involve the ARM7 being stopped, but not the VC(TC)XO. The proper scientific way to test this hypothesis would be to probe this signal with an oscilloscope on a live board and see what happens in different sleep modes, but it would be difficult to do this experiment on an FCDEV3B or a GTA02 because in our (previously OM's) PCB the signal in question goes from one BGA to another (from the Calypso to the memory chip) without any accessible test points. However, on Pirelli's PCB the signal that comes from Calypso's FDP output goes to an unpopulated resistor footprint that can be used as a test point, hence I plan on using a decased Pirelli motherboard for this experiment - but it will take me some time to make the necessary arrangements. So here is my current working hypothesis as to the sleep mode bug: if we suppose that FDP does go low during small sleep, then if the running firmware is flashed (not RAM-based) and has the FFFF:FB10 register configured so that interrupt vectors go to flash rather than Calypso's internal ROM (vectors in flash is the configuration used in TI's original firmwares and in FC Magnetite), then the interrupt event that causes wakeup from sleep is immediately followed by the Calypso's memory interface (MEMIF) doing a read from address 0x0000_0018, which is the ARM IRQ vector location. The flash chip sees an asserted reset signal during the sleep, then upon wakeup it sees this reset being negated, immediately followed by a read cycle. Now suppose that the reset pulse seen by the flash chip is too short because the sleep duration was very short, or perhaps the time between FDP going high (reset negation) and the immediate read cycle for the IRQ vector is shorter than the minimum required by the flash chip. Under such circumstances it is plausible that the flash chip may ignore the read cycle or return the wrong data. Because this read is for the IRQ vector which contains an ARM branch instruction, if the Calypso reads wrong bits from the data bus on this cycle, the flow of control will go haywire, and a reboot can plausibly occur if the control falls into the RESET entry code nearby. I have no way of telling if TI's approach of connecting the FDP output to the flash chip's reset input was ever a good idea for any flash chip, but it appears that the high-capacity Spansion MCP which we copied from the Pirelli may be less forgiving of such abuse than its predecessors used by TI or the Samsung MCP used by Openmoko, and while this arrangement worked fine for OM and possibly on TI's Leonardo boards (of which I have never seen one with my own eyes), the newer Spansion flash chip gets unhappy with the peculiar reset timing it gets subjected to. If this hypothesis is correct, then it would follow that the very strange behaviour that was seen on only one FCDEV3B board whereby the board would boot from flash only in mode 0 but not in mode 1 probably has the same root cause as the sleep mode bug, and it just happened that on that one board the flash chip is a bit more sensitive and gets unhappy with the watchdog reset that happens as part of flash boot mode 1. I can only assume that TI connected Calypso's FDP output to the flash chip's reset input on the reasoning that having the reset input asserted ought to put the flash chip in its lowest current draw state, and like I already said, I don't know if this reasoning was ever correct for any flash chip - maybe it was, maybe it wasn't. But the datasheet for Samsung K5A3280 (the closest I could find to Openmoko's K5A3281CTM) gives exactly the same Icc (current from Vcc) spec both for the reset held down state and for the regular standby state (no reset asserted, but no chip select activity), hence with this chip having or not having the reset input asserted during sleep ought to make no difference. And with Spansion high-capacity flash it is the opposite according to the datasheet we have: Icc in the reset held down state is higher than Icc in regular standby! At this point the logical thing to do was to see how this flash reset signal is connected on the Pirelli DP-L10, so I traced it using steve-m's PCB grind-down scans. And lo and behold: on the Pirelli phone the reset input to this same Spansion flash chip is NOT connected to the FDP output from the Calypso! Instead it is connected to a pair of 0402 resistor footprints, making it configurable: there is one resistor that, if populated, would connect the flash reset line to FDP - but that resistor is not populated - and the other resistor, which is populated, connects it to Calypso's TCXOEN output. TCXOEN enables or disables the 26 MHz oscillator (and on the Pirelli it actually is an external VCTCXO), and it goes low during deep sleep - but not during small or big sleep. Wakeup from deep sleep has programmable timings (unlike wakeup from small sleep), and I can only reason that Foxconn/Pirelli must have configured these timings so that the flash chip's reset timing requirements are satisfied. Another experiment I did was to try running a RAM build of the same Magnetite l1reconst fw that consistently triggers the sleep mode bug when flashed, and guess what: when I run the fw in RAM, I can enable all sleep modes with AT%SLEEP=4, then do AT+CFUN=1 and the latter command completes without rebooting! I then connected to the GSM network with AT+COPS=0, and I saw the fw go into both big and deep sleep while camped on a cell and listening for paging. This was my first time ever seeing deep sleep on an FCDEV3B with L1 in I_MODE (idle, camped on a cell) as opposed to CS_MODE0 (the power-up state before any AT commands when no network bring-up is attempted), and I am pleased to report that we do NOT have the kind of issues with deep sleep which Openmoko experienced as their infamous bug #1024. My next experiment was to hack together a firmware image that boots and runs out of flash, but routes the ARM7 interrupt and exception vectors through Calypso's internal ROM and RAM just like we do with our run-from-RAM fw builds. I was hoping that this change would make the sleep mode problem go away by inserting an additional delay between the wakeup event and the first subsequent flash access, and thus give us a workable software solution for allowing sleep modes in flashed fw on our current FCDEV3B hw, but no such luck: the exact behaviour changed a little, but the same AT+CFUN=1 operation that always caused a self-reboot before still causes some kind of crash or hang. I am now wondering if this sleep mode bug can be fixed in the hw by disconnecting the flash chip's reset input from Calypso's FDP output and simply tying it high instead. There will be no more external power-on reset to the flash chip with this arrangement, but I reason that the flash chip probably does not need such an external reset and can instead ensure the correct power-up state on its own just from the Vcc ramp-up - after all, many flash chips I have worked with don't have an external reset pin at all. I plan to proceed as follows: 1. Do some experiments on a decased Pirelli motherboard to test my hypothesis regarding Calypso's FDP output going low in all sleep modes. 2. If my FDP hypothesis proves correct, try a very delicate surgical experiment on an FCDEV3B: remove the flash+pSRAM chip (BGA rework station to the rescue), cut the trace between the D5 ball pad (flash reset) and the microvia it goes to, make a solder bridge or some other hacky connection to the C5 ball pad next to it (WP#/ACC input, connected to flash Vcc through a pull-up resistor), and then put a new S71PL129NC0HFW4B flash+pSRAM chip back on (BGA rework station once again). The effect will be to disconnect the flash chip's reset input from FDP and tie it high instead. See if this hw change makes the sleep mode bug go away. 3. If the previous step succeeds, make the same change in the Altium PCB layout, make new gerbers with this change, and fab a new batch of boards (call them FCDEV3Bv2) with this change incorporated. The last step will probably cost me somewhere in the 3-4 kUSD range, hence the preceding steps to make very sure that my proposed change will do the trick. Hasta la Victoria, Siempre, Mychaela aka The Mother _______________________________________________ Community mailing list Community@freecalypso.org https://www.freecalypso.org/mailman/listinfo/community