Ever feel like you’re riding the Struggle Bus when using the esp8266?
tl;dr
Try a different USB serial adapter.
The Debug
This is the first part of a series on the insanely stupid amount of debugging I had to endure, in order to get started with this magnificent chip.
A lot of people seem to have problems programming it, myself included. There are a lot of documented cases of weird errors showing up, such as a flash failing at 7 or 8%, every time [1] [2] :
While I was getting up to speed on the esp8266, I built at least three programming circuits, and each one failed to upload a program correctly to the chip.
This was extremely irritating and I wasted a ridiculous amount of time trying to figure out what’s going on, whether I was at fault, or the circuit was at fault, or the uploader program was at fault.
I tried adding decoupling capacitors to the 3.3v → ground lines, I tried tying up the RESET and GPIO2 lines.
I tried the NodeMCU flasher, esptool.py, and esptool-ck, and none of them worked.
I tried on a Windows machine, on a Mac, and on a Linux machine, again nothing worked.
I tried attaching jumper cables directly between the programming pins and an FTDI serial adapter, and it still froze on upload.
I tried adding a separate 2A regulated power supply, this didn’t help either.
I even tried buying an EEPROM programming clip and seeing if I could write the EEPROM that way, no luck.
The esp8266 has become my white whale, as they say. And I will see it at the gates of Hell, waiting for me with a espcomm_send_command(FLASH_DOWNLOAD_DATA) failed message.
Or with obtuse responses like:
$ ./esptool.py --baud 115200 --port /dev/cu.usbserial-A50285BI write_flash 0x00000 nodemcu_integer_0.9.6-dev_20150704.bin Connecting... Erasing flash... Took 2.21s to erase flash block Writing at 0x00008000... (7 %) A fatal error occurred: Invalid head of packet $ ./esptool.py --baud 115200 --port /dev/cu.usbserial-A50285BI write_flash 0x00000 nodemcu_integer_0.9.6-dev_20150704.bin Connecting... Erasing flash... Took 2.32s to erase flash block Writing at 0x00005000... (4 %) A fatal error occurred: Invalid head of packet $ ./esptool.py --baud 115200 --port /dev/cu.usbserial-A50285BI write_flash 0x00000 nodemcu_integer_0.9.6-dev_20150704.bin Connecting... Erasing flash... Took 2.28s to erase flash block Writing at 0x00003c00... (3 %) A fatal error occurred: Invalid head of packet $ ./esptool.py --baud 57600 --port /dev/cu.usbserial-A50285BI write_flash 0x00000 nodemcu_integer_0.9.6-dev_20150704.bin Connecting... Erasing flash... Took 2.28s to erase flash block Writing at 0x00000000... (0 %) A fatal error occurred: Invalid head of packet $ ./esptool.py --baud 57600 --port /dev/tty.usbserial-A50285BI write_flash 0x00000 nodemcu_integer_0.9.6-dev_20150704.bin Connecting... Erasing flash... Took 2.17s to erase flash block Writing at 0x00000000... (0 %) A fatal error occurred: Invalid head of packet $ ./esptool.py --baud 230400 --port /dev/tty.usbserial-A50285BI write_flash 0x00000 nodemcu_integer_0.9.6-dev_20150704.bin Connecting... Erasing flash... Took 2.24s to erase flash block Writing at 0x00008000... (7 %) A fatal error occurred: Invalid head of packet $ ./esptool.py --baud 460800 --port /dev/tty.usbserial-A50285BI write_flash 0x00000 nodemcu_integer_0.9.6-dev_20150704.bin Connecting... Erasing flash... Took 2.25s to erase flash block Writing at 0x00008000... (7 %) A fatal error occurred: Invalid head of packet $
It was high time for me to start debugging esptool-ck, to find out why it was failing, and to see if there were problems I could solve.
I’d tried everything everything, and I was beginning to suspect my crappy, Chinese-knockoff FTDI converter is to blame. It was the one common denominator I thought could not possibly be the problem.
This gave me a good opportunity to mess around with JetBrains’ CLion, since importing esptool-ck into Xcode turned into a complete and utter clusterfuck. (Way to go, Apple. Why is Xcode such a pain in the ass to import existing files into and set dependencies? Seriously, way too hard.)
After a bit of debugging, I caught esptool-ck blocking on the tcdrain() call multiple times when running in multiple CLion lldb debugging sessions. This is interesting, and seems to be a good place to start.
I suspected that OS X might have some issues, so as a start, I modified espcomm.c to remove calls to tcdrain() under OS X. The fact that esptool-ck was unable to exit, when it got stuck in the tcdrain() pointed to a potential hardware issue with the counterfeit circuit. A genuine device should simply not exhibit problems here.
Another particularly gruesome issue with the crappy knock-off FTDI boards, was that they refused to execute the close() properly and would actually hang. So the only way to fix the situation was to unplug the counterfeit FTDI board and power cycle the esp8266.
In the end, the problem was with the counterfeit FTDI hardware. Don’t buy this adapter for esp8266 programming, you will totally regret it:
After testing out an adapter using a genuine FTDI chip, all of the programming methods worked just fine.
Ahh! This article is written after i bought mine. I was having exactly same problem while trying to flash the micropython to ESP-01.
“Writing 507904 @ 0x0… 504832 (99 %)
A fatal error occurred: Timed out waiting for packet header”
Thanks alot for the article. I wasn’t able to find where the problem is.
🙂 Same here. Driving me nuts.
Thanks for your investigative effort. Luckily for me I have not one but two supposedly identical shitty FT-232 knockoffs.
After several hours of 99% writes and “Timed out waiting for packet header” errors, I happened across your writeup and finally thought to try the other FT232 – since I had no other options – and it worked.
Strangely under Windows I found the NodeMCU boards based on some other serial chip (CH340 I think) to be less reliable than the FT232 (knockoff). On a different machine running Linux, they seem to work great.
Still, at a dollar apiece for the FT232s and less than twice that per ESP, I’m still not complaining. I think my PICKIT 3 cost something like $40…
Thank you! A thousand times this!!! I had the same serial adapter. Stupid thing gets to 99% and fails.
Wow Wow!. I just happened to encounter this similar error i.e. “A fatal error occurred: Timed out waiting for packet header”, while trying to flash the esp8266. I have been facing this issue provided my connections are right. And after an hour of research online to fix this problem, I encountered your blog. I am using the same FTDI hardware shown in your image. Guess, its time to try flashing the firmware using a different/genuine serial to usb converter. Thank you for posting this.
Very interesting. I also notice different behaviour on windows (xp) and linux (Ubuntu 14.04). A CH341 works just fine on windows with esplorer and nodemcu flasher wheras on linux the identical hardware does not communicate reliably with esplorer and not at all with esptool – need to investigate deeper.
Same for me, I was using an FT232RL and it doesn’t work. I have used a FT2232D and now works just fine. Thanks for your advice
Out of curiosity, two questions: Was the FT232RL one of the counterfeit type from eBay? Was the FT2232D from a more reliable supplier?
Man, you’re save my life. I Changed to a generic CP2102 and now is working!!!
Thank you for share this solution…
Glad to hear!
Hi,
Recently I had the same problem. I was using the USB to RS232 converter with FTDI232RL chip and I set baud rates max 115200. Only sometimes flash downloading finished successfully. I have done some testing on Windows and Linux with FTD232RL and CP2104 chips. For CP2104 flashing always worked with baud rates: from 56700 up to 921600. For FT232RL and baud rates 230400, 115200 and lowers flash downloading failed frequently. I have checked what will happen if I set bigger baud rates for FT232RL. Surprisingly, for baud rate 921600 for all tests I have done the firmware was uploaded successfully.
Test environment:
On windows for testing I have used latest version V3.4.2 – Flash Download Tools. On Linux I have tested with esptool.py (dev-1.2) and esptool-ck (v0.4.11 ). The firmware which I was downloading into flash was AT firmware based on NONOS SDK 1.5.4 in non-boot mode. The module which I have used was ESP-WROOM-02 with 32MBit flash. Flash options I had enabled during flash downloading: mode: QIO, speed 40MHz, size 32m-c1.
Thank you, I was running into the same bug. I have bought 2 of this FTD, the one your photo shows “YP-05” and the second with “FTDI232” label. Both do not work.
Now I am using a “cu.SLAB-USBtoUART” FTDI an it is working fine.
Regards Jupp
thank you was going through the same problem using a ttl worked s2
Great post. I was trying to program with one of these adapters bought of Aliexpress and it didn’t work.
Switched to one I’d bought off eBay and it worked flawlessly.
Cheers!
Same as Pete !
Thank you so much !
Thank you so much for this post. I had exactly the same experience for three long evenings and was at my wits end trying to make it work.
Following your advice I switched from:
https://www.amazon.co.uk/tinxi-FT232RL-Serial-Adapter-Arduino/dp/B01C2P9GD2/ref=sr_1_1?ie=UTF8&qid=1483813793&sr=8-1&keywords=ftdi+usb+to+serial+adapter
to:
https://www.amazon.co.uk/d/Cables/Adafruit-70-FTDI-Serial-TTL-232-USB/B00SK8LK1W/ref=sr_1_31?ie=UTF8&qid=1483813927&sr=8-31&keywords=usb+to+serial+adapter+ftdi
It worked first time…
Dave
Thank you same problem here. Please show me your new “genuine FTDI adapter”…
It’s an FTDI C232HD-DDHSP-0 cable.
What If I have arduino D1 board with built in wifi wemos module and there’s no FTDI chip but I’m getting the same problem. This board is from aliexpress it was working and stopped.
I think the Arduino D1 uses a CH340G chip for USB to serial, which in my testing only managed 115200 bits per second firmware uploads.
I’m not sure how the Arduino IDE does its uploads to the D1, but it might help to try either slower speeds or using esptool.py or esptool-ck directly from the command line to see how the ESP8266 chip responds.
The ESP8266 is pretty tough overall, at the very least, if the chip is still functional you should always be able to see the initial boot string, which should always appear before any jump to user code.
Hi,
Trying to flash my first Sonoff and as always the first one is the hardest! Using a “prolofic” USB to serial converter, yes a cheapo of ebay. And yes a knockoff prolific, and needed to do this to resolve my Windows 10 issue:
http://www.totalcardiagnostics.com/support/Knowledgebase/Article/View/92/20/prolific-usb-to-serial-fix-official-solution-to-code-10-error
It successfully flashes and completes to 100% but the Sonoff is dead ie no LED activity.
In this instance, could the knockoff USB to serial driver be at cause?
If the updater gets to 100% and completes the process, it might not be the USB to serial adapter. At least if everything in the updater seems to indicate success…