Python Datalogger - Using pySerial to Read Serial Data Output from Arduino

After I became proficient with Arduino I plant myself trapped in its evolution environment (IDE). I needed to escape from the simplicity of the series port and transform the platform into a usable technology tool. I tried saving to SD cards, only decided adding more than hardware was superfluous; I tried Bluetooth and WiFi, but again, barring specific Cyberspace of Things applications, I establish those to be roundabout ways of achieving what I wanted: a simple datalogging organization. I ultimately arrived at Python's pySerial library, which reads directly form the serial port and was a complete solution to my predicament.

Using pySerial is much easier than one might expect, equally the nigh fundamental implementation consists of but three lines of code:

                    import                    serial                    ser                    =                    serial.Serial('/dev/ttyACM0') ser_bytes                    =                    ser.readline()                  

These three simple lines read a single row of data from the series port. In the case of Raspberry Pi, the serial port (on my Arduino) is located at '/dev/ttyACM0'. Yous may too discover yours there, or at an integer increment (ttyACM1, ttyACM2, etc.), or perhaps a different address completely. Check your Arduino IDE serial port for the exact location. Subsequently successfully reading a line from your Arduino, verify that it is in the desired format. I chose to print and read a single line, merely you may adopt comma separated or similar formats. The code above isn't especially interesting, but it verifies that pySerial is working and that you are parsing data correctly from your serial port. Once the method to a higher place is understood, nosotros tin advance onto loops and recording information in real-time.

Note: I will be using a DHT11 temperature sensor to produce data on the Arduino end. Since this is a tutorial on reading data from the serial port using Python, not Arduino, I recommend visiting a DHT11 tutorial to larn how to print temperature data from the sensor to the serial port (see here, or here).

Serial Read Loop

I will be using a while loop and keyboard interrupt (CTL-C) to loop and halt the datalogging. The data from the serial port besides needs to be converted from unicode to float (or some other datatype) so that the data can be candy in Python. In my instance, I am using "utf-8" to decode, which is Arduino'due south default encoding and the most normally used graphic symbol encoding format. You may too notice the 'ser.flushInput()' command - this tells the serial port to clear the queue and so that data doesn't overlap and create erroneous data points. Sometimes the conversion via bladder() tin can create errors, merely this is due to overprinting from the Arduino's finish. If you receive such an error, restart the python script and try again.

                    import                    series                    ser                    =                    serial.Serial('/dev/ttyACM0') ser.flushInput()                    while                    Truthful:                    endeavour:         ser_bytes                    =                    ser.readline()         decoded_bytes                    =                    float(ser_bytes[0:len(ser_bytes)-2].decode("utf-viii"))                    print(decoded_bytes)                    except:                    print("Keyboard Interrupt")                    pause                  

Now we have a working real-time data printer in Python. Decidedly, this isn't peculiarly interesting because nosotros aren't saving or plotting the information, so we'll embrace how to practice both of those adjacent.

Saving Serial Data to CSV File

In the code below I accept implemented a manner to save the serial data in existent-time to a .csv file.

                    import                    series                    import                    fourth dimension                    import                    csv                    ser                    =                    serial.Serial('/dev/ttyACM0') ser.flushInput()                    while                    True:                    endeavor:         ser_bytes                    =                    ser.readline()         decoded_bytes                    =                    float(ser_bytes[0:len(ser_bytes)-2].decode("utf-8"))                    impress(decoded_bytes)                    with                    open("test_data.csv","a")                    every bit                    f:             writer                    =                    csv.author(f,delimiter=                    ",")             author.writerow([fourth dimension.time(),decoded_bytes])                    except:                    impress("Keyboard Interrupt")                    break                  

Now we have a working datalogger! This is as simple as it gets, and it's remarkably powerful. The iii lines that outset as: '' with open("test_data.csv","a") as f: '' await for a file called 'test_data.csv' and create it if it doesn't exist. The "a" in parentheses tells Python to append the serial port information and ensure that no information is erased in the existing file. This is a grand result because it not only takes intendance of saving the data to the .csv file, but it creates i for united states of america and prevents overwriting and (most times) corruption. How considerate!

Alive Plotting Series Information Using matplotlib

To take things a bit further, I decided to aggrandize the content here and include a real-time plot. I find real-time plotting a useful tool when acquiring data of whatsoever kind. It is much easier to find faults in visualizations than it is to find them in streaming values.

NOTES: while I was using Raspberry Pi, I came beyond an issue between reading the series port, saving to .csv, and updating the plots. I establish that updating the plot occupied a lot of processing time, which resulted in slower reading of the series port. Therefore, I advise anyone who is using the method below to assess whether you lot are reading all the bytes that are beingness outputted past the Arduino. I constitute that I was missing bytes or they were getting backed up in the queue in the buffer. Do some tests to verify the speed of your loop. This will prevent lost bytes and dropouts of data. I found that my loop took roughly half a second to complete, which means that my series port should not be outputting more than than ii points per second. I really used 0.8 seconds every bit the time between information records and it appeared to take hold of all data points. The ho-hum loop is a event of the plotting, so once you comment out all of the plot lawmaking, you will get a much higher data rate and .csv salvage rate (just as to a higher place).

                    import                    serial                    import                    time                    import                    csv                    import                    matplotlib                    matplotlib.utilize("tkAgg")                    import                    matplotlib.pyplot                    as                    plt                    import                    numpy                    as                    np                    ser                    =                    serial.Serial('/dev/ttyACM0') ser.flushInput()  plot_window                    =                    twenty                    y_var                    =                    np.assortment(np.zeros([plot_window]))  plt.ion() fig, ax                    =                    plt.subplots() line,                    =                    ax.plot(y_var)                    while                    Truthful:                    try:         ser_bytes                    =                    ser.readline()                    attempt:             decoded_bytes                    =                    float(ser_bytes[0:len(ser_bytes)-                    2].decode("utf-8"))                    print(decoded_bytes)                    except:                    proceed                    with                    open("test_data.csv","a")                    as                    f:             writer                    =                    csv.writer(f,delimiter=                    ",")             writer.writerow([time.time(),decoded_bytes])         y_var                    =                    np.suspend(y_var,decoded_bytes)         y_var                    =                    y_var[i:plot_window+                    1]         line.set_ydata(y_var)         ax.relim()         ax.autoscale_view()         fig.canvas.describe()         fig.canvass.flush_events()                    except:                    print("Keyboard Interrupt")                    break                  

dht11_serial_read.png

Figure one: DHT11 Live Plot from Series Port

Plot produced by matplotlib in Python showing temperature data read from the serial port. During this plot, the sensor was exposed to a estrus source, which can be seen here as an increase from 31 to 35 degrees C.

Conclusion

This tutorial was created to demonstrate that the Arduino is capable of acting equally an independent data logger, carve up from wireless methods and SD cards. I found Python'southward pySerial method a while ago, and I wanted to share its capabilities with makers and engineers that may be having the same problems that I was encountering. Printing data to Arduino's serial port and then reading it through Python gives the user the freedom to investigate the data further, and take advantage of the advanced processing tools of a reckoner, rather than a micro controller. This method also allows the user to bridge the gap between live data and laboratory measurements. With real-time datalogging via the series port, ane can mimic the laboratory setup of acquisition, analysis, and live observation. Often, with Arduino the user is trapped in the serial port, or is relegated to advice via protocols, which can take fourth dimension and energy. Withal, importing the data into Python frees the user of middle-men and allows the data to exist candy in any way preferred. I use pySerial often, whether for recording temperature data using thermocouples, or high-frequency hall sensor measurements to monitor moving parts. This method can be used to accept pressure level measurements in the laboratory, or fifty-fifty record calibration data to better your instrumentation accuracy; the possibilities are truly countless.

"As an Amazon Associates Program member, clicking on links may result in Maker Portal receiving a small committee that helps support future projects."

Encounter More in Arduino and Python:

Arduino, Data Assay, Programming, Python, Most Pop

pySerial, Python, programming, series, bytes, datalogger, data, Arduino, Analyze, Information Analysis, Serial Port, DHT11, matplotlib, loop, while loop, csv, tkAgg, interrupt, figure, 2018 #i, Most Popular 2018